Search-Based Generation of Undetected Quantum Circuit Mutants

Search-Based Generation of Undetected Quantum Circuit Mutants EÑAUT MENDILUZE USANDIZAGA, Simula Research Laboratory & Oslo Metropolitan University, Norway

arXiv:2608.09318v1 [cs.SE] 10 Aug 2026

THOMAS LAURENT, RI Lero & Trinity College Dublin, School of Computer Science and Statistics, Ireland PAOLO ARCAINI, National Institute of Informatics, Japan SHAUKAT ALI, Simula Research Laboratory & Oslo Metropolitan University, Norway Quantum mutation analysis is emerging as an essential technique for evaluating test suites due to the limited availability of real faulty quantum programs. However, existing quantum mutation analysis tools use fixed gate-based mutations, resulting in mutants that are easy to detect, which reduces their effectiveness in assessing the quality of test suites. We propose QUMUG, a search-based approach for generating challenging mutants by utilising parameterisable quantum gates. QUMUG employs search algorithms to optimise mutation parameters and find non-equivalent mutants passing a given test suite. In our evaluation over 30 quantum programs, QUMUG produced mutants that are three times more challenging than the mutants generated by existing tools. Among the four evaluated search algorithms, the genetic algorithm was the most effective, generating an average of 494 undetected mutants per program with a 99.67% success rate and 94.3% non-equivalent ratio. The generated mutants demonstrated their effectiveness by requiring the addition of five times more test cases to the test suite than the mutants generated by existing tools. We also analysed the behaviour of higher order mutants in quantum circuits, and showed that while first order mutations are more effective for enhancing the test suite, higher order mutants highlight the need for new unique test cases. CCS Concepts: • Computer systems organization → Quantum computing; • Software and its engineering → Software testing and debugging. Additional Key Words and Phrases: Mutation Analysis, Quantum Computing, Search-Based Software Engineering ACM Reference Format: Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali. 2026. Search-Based Generation of Undetected Quantum Circuit Mutants. ACM Trans. Softw. Eng. Methodol. X, X, Article X (August 2026), 36 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

Quantum computing is a developing field that holds the potential to address complex problems faster than traditional classical computing [40]. However, quantum computing also introduces new challenges, particularly in ensuring the reliability of quantum software [25, 34, 58]. For instance, quantum programs often exhibit non-deterministic behaviour, and their internal states cannot be observed without collapsing the quantum state. As a result, there is a pressing need for innovative Authors’ Contact Information: Eñaut Mendiluze Usandizaga, enaut@simula.no, Simula Research Laboratory & Oslo Metropolitan University, Oslo, Norway; Thomas Laurent, tlaurent@tcd.ie, RI Lero & Trinity College Dublin, School of Computer Science and Statistics, Dublin, Ireland; Paolo Arcaini, arcaini@nii.ac.jp, National Institute of Informatics, Tokyo, Japan; Shaukat Ali, shaukat@simula.no, Simula Research Laboratory & Oslo Metropolitan University, Oslo, Norway. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7392/2026/8-ARTX https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:2

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

software testing techniques that are tailored to the unique characteristics of quantum computing, such as superposition and entanglement [35, 56]. In recent years, various software testing techniques have been developed for quantum software [1, 8, 9, 25]. It is essential to evaluate the effectiveness of these techniques, which requires the establishment of testing benchmarks using faulty programs [6, 59]. Unfortunately, such benchmarks are scarce in the realm of quantum computing. To mitigate this scarcity, researchers have turned to mutation analysis to systematically generate benchmarks of faulty programs [13, 29]. For quantum software, mutation analysis involves generating “mutants” of quantum programs by applying gate-based operations, resulting in modified versions that introduce specific faults. This technique enables the creation of a larger set of comprehensive benchmarks for assessing quantum software testing techniques. In a previous study [30], we performed an analysis of the mutants generated by Muskit [29]. We found that Muskit’s mutation operators introduce changes that have too big of an effect on the quantum programs outputs. This leads to mutants that behave completely differently from the original program, making them easy-to-detect, with most being caught by even a single test case. Therefore, mutation scores are usually very high, not because test suites are particularly effective, but because the mutants themselves are trivial to detect. Moreover, in the context of mutation testing, since these easy-to-detect mutants are already covered by existing tests, they do not encourage the development of new or stronger test cases [55]. In this paper, we introduce Quantum Undetected MUtant Generation (QUMUG), an approach that directly addresses these limitations. Unlike quantum mutation tools available in the literature (i.e., Muskit and QMutPy), QUMUG focuses on generating mutants that remain undetected by a test suite. By doing so, we are able to generate mutants that represent faults not covered by the current test suite, revealing a deficiency in it. We propose novel mutation operators utilising parameterisable quantum gates, which allow us to control the magnitude of the applied effect. These gates offer a large range of parameters, providing us with a wide variety of mutants; however, they also create a very large search space to explore. To address this issue, we utilise search-based algorithms to fine-tune these parameters and to determine where in the circuit the mutations should be applied. Furthermore, we introduce the possibility to incorporate multiple mutations within the same circuit to generate higher order mutants, a concept widely used in classical mutation analysis and that has demonstrated potential in test suite assessment [37]. We evaluate QUMUG using a total of 30 quantum programs that vary in type (six different algorithms) and number of qubits (3 to 7). To perform a comprehensive evaluation of QUMUG and validate the generated mutants, we simulate a mutation testing scenario: we generate mutants based on a given test suite and evaluate whether they require the inclusion of new tests to be detected. We compare the performance of four different search algorithms: genetic algorithm (GA), hill climbing (HC), 1+1 evolutionary algorithm ((1+1)-EA), and random search (RS), in finding mutants requiring new test cases. The evaluation demonstrates that GA achieved the best results for generating mutants that are not detected by the test suite given as input and that are not equivalent: it generated an average of 494 mutants per program, with a 94% non-equivalent ratio. It achieved a 99.67% success rate across all programs, failing to generate non-equivalent mutants that pass the test suite only once out of 300 executions. Moreover, the results indicate that QUMUG, using any of the four search algorithms, generated more challenging mutants than the baselines, with mutants detected by three times fewer test cases than the ones generated by the baselines. GA showed the best results in terms of generating mutants that contribute to the improvement of the test suite, requiring a

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:3

median of five different test cases to detect all of them. In contrast, the mutants generated by the baseline tools could be detected by simply adding one new test case to the test suite. We also analysed the behaviour of higher order mutants for quantum programs, showing that while they contained, in proportion, fewer equivalent mutants than first order mutants, they were less challenging (i.e., they were detected by more test cases). The generated higher order mutants required fewer test cases to be detected, contributing less to the improvement of the test suite. Still, the higher order mutants generated by QUMUG were better than the mutants generated by the tools in the literature. We also studied interactions between the mutations introduced into the same circuit, which showed that most higher order mutants were generated by combining mutations in different circuit positions: affecting different qubits or gates. The analysis of test cases detecting higher order mutants showed that detection is not always directly related to the test cases detecting each individual mutation. The results show that interactions between mutations can either increase or decrease a mutation’s effect and even make it detectable in cases where they would not be on their own. This suggests that, as in classical mutation testing, there could be subtle higher order quantum mutants which represent unique faults that can not be represented with first order mutants, and thus need the inclusion of specific test cases to detect them. To summarise, the contributions of this paper are: (1) QUMUG, a novel approach that introduces new parametric mutation operators capable of generating more challenging mutants, which represent faults not detected by the given test suite, thereby contributing to its improvement. (2) The first implementation of higher order mutants for quantum programs, leading to interesting relationships among mutations and proving that they require different test cases to be detected compared to first order mutants. Through these contributions, we provide researchers and developers with a novel approach for enhancing their test suites by offering a wider, more challenging variety of mutants that represent a broader range of faults. The paper is structured as follows. Sections 2 and 3 present the background and related work, respectively. We present the methodology of the approach in Section 4, followed by the experiment design in Section 5. Finally, we introduce the experimental results in Section 6 and provide a more detailed discussion in Section 7. We conclude the paper in Section 8. 2

Background

This section introduces some essential concepts to understand the paper. First, Section 2.1 presents fundamental concepts of quantum computing, including quantum circuits and quantum software testing. Next, Section 2.2 covers the principles of mutation analysis, while Section 2.3 explains how mutation analysis adapts to the quantum realm. 2.1

Quantum Computing

Quantum computing leverages the principles of quantum mechanics to perform calculations significantly faster than classical computers, known as quantum speedup [40]. Quantum computers use quantum bits (or qubits) as their basic units for computations. Qubits exhibit unique properties, such as superposition and entanglement, which enable calculations to be performed in ways that classical bits cannot. Quantum entanglement refers to the ability of qubits to be interconnected regardless of the distance between them [56]. This implies that, by measuring the state of one qubit, the state of another one can be directly determined. Superposition refers to the ability of a qubit to exist in multiple states simultaneously [35]. Unlike a classical bit, which can only be either 0 or 1 at any given time, a qubit can remain in a superposition state of both 0 and 1. By leveraging these quantum principles, quantum programs can process information more efficiently. A commonly used notation represents the state of a qubit as |𝜓 ⟩ = 𝛼 |0⟩ + 𝛽 |1⟩, where 𝛼 and 𝛽 are complex probability amplitudes [35]. In order to be able to interpret the state of a qubit in a classical way, ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:4

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Fig. 1. Bloch sphere qubit representation

Fig. 2. Quantum circuit representation of a GHZ Entangled state of three qubits

the qubit must be measured, which collapses its state into one of the computational basis states |0⟩ or |1⟩. This collapse destroys the original superposition, meaning the qubit is no longer in a quantum state. Figure 1 shows a Bloch sphere, a common visual representation of the state of a qubit [35]. The Bloch sphere is defined with three axes, labelled 𝑋 , 𝑌 , and 𝑍 , each corresponding to a different possible measurement base. The measurement basis specifies the axis along which the state is projected during this process [35]. The default measurement base is the 𝑍 -basis, where the measurement distinguishes between |0⟩ and |1⟩. By applying appropriate single-qubit rotations before measurement, we can effectively change the measurement axis to align with the 𝑋 - or 𝑌 -basis. In this representation, the position of the qubit’s state, |𝜓 ⟩, is parametrised by three angular variables: 𝜃 , 𝜙, and 𝜆. The angle 𝜃 defines the position along the sphere’s vertical axis, determining the relative weighting between the computational basis states |0⟩ and |1⟩. The angle 𝜙 specifies the rotation around the sphere’s equator, thereby setting the relative phase between these basis states. The third parameter 𝜆 corresponds to an overall phase rotation. Each of these parameters controls a distinct type of rotation or “flip” on the Bloch sphere. 2.1.1 Quantum Circuits. Quantum programs are typically represented as quantum circuits, which describe the sequence of quantum operations applied to qubits. These operations are implemented through quantum gates. Quantum gates can be understood as rotations on the qubit’s Bloch sphere, allowing qubits to change between different states [35]. Figure 2 shows an example of a quantum circuit consisting of three qubits, 𝑞 0 , 𝑞 1 , and 𝑞 2 . As shown in the figure, quantum gates may operate on a single qubit (single-qubit gates) or on multiple qubits (multi-qubit gates). For instance, a Hadamard gate, represented as 𝐻 in the figure, is a single-qubit gate commonly used at the beginning ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:5

Table 1. Standard quantum gates and their representation as 𝑅-gates and a 𝑈 gate

Single-qubit gate 𝐻 𝑋 𝑌 𝑍 𝑇 𝑆 𝑆𝑑𝑔

Equivalent representation as 𝑅-gates

𝑈 gate

𝑅𝑥 (𝜋) · 𝑅 𝑦 ( 𝜋2 ) 𝑅𝑥 (𝜋)

𝑈 ( 𝜋2 , 0, 𝜋) 𝑈 (𝜋, − 𝜋2 , 𝜋2 ) 𝑈 (𝜋, 0, 0) 𝑈 (0, 0, 𝜋) 𝑈 (0, 0, 𝜋4 ) 𝑈 (0, 0, 𝜋2 ) 𝑈 (0, 0, − 𝜋2 )

𝑅 𝑦 (𝜋) 𝑅𝑧 (𝜋) 𝑅𝑧 ( 𝜋4 ) 𝑅𝑧 ( 𝜋2 ) 𝑅𝑧 (− 𝜋2 )

of quantum circuits to create superpositioned states. The controlled-NOT (𝐶𝑋 ) gate, indicated by a dot and a cross in the figure, is a two-qubit operation that acts as a conditional gate, where the state of one qubit defines whether the operation in the second qubit is performed. By combining these gates, the circuit in Figure 2 shows the representation of a Greenberger–Horne–Zeilinger (GHZ) entanglement state, a maximally entangled state, where the state of each individual qubit is entirely dependent on the states of the other qubits [16]. The circuit also includes a group of classical bits, referred to as 𝑐3 at the bottom part of the figure, known as classical registers. At the end of the circuit, the measurement operations are shown, represented by a square with a semicircle and an arrow that connects the qubits to the classical registers. These operations collapse the quantum state and translate it into the classical registers present in the circuit. This work uses specific parametric gates that allow us to control the desired effect in the circuit. To manipulate the orientation of a qubit’s state, quantum circuits employ single-qubit rotational gates (𝑅-gates): 𝑅𝑥 (𝜃 ), 𝑅 𝑦 (𝜃 ), and 𝑅𝑧 (𝜙), which rotate the qubit around the 𝑋 , 𝑌 , and 𝑍 axes of the Bloch sphere, respectively. By combining such rotations, any arbitrary single-qubit transformation can be achieved. In addition to the rotational gates, there is also the universal gate 𝑈 , which can represent any single-qubit gate. The 𝑈 gate consists of three parameters 𝜃 , 𝜙, and 𝜆; by setting specific values to the three parameters, it is capable of applying a specific rotation to the Bloch sphere. When constructing a circuit, we typically use predefined gates that represent specific versions of these parametric gates mentioned. Table 1 lists some of the most common single-qubit gates and their translation to a combination of 𝑅-gates, and to a 𝑈 gate [20]. 2.1.2 Quantum Software Testing. Software testing for classical systems is a well-established field. However, in the quantum realm, the need for effective testing approaches remains, as quantum software introduces challenges that classical testing techniques cannot directly address. Indeed, many methods in classical software testing rely on checking the internal state of a program or copying its state, which, in the case of quantum software, cannot be done due to the collapse of the quantum state [56]. To address these issues, new software testing techniques have been developed in recent years with the goal of testing quantum software [2, 3, 5, 8, 9, 17, 25–27, 32, 47, 49, 50, 52, 53, 57]. When testing quantum software, we refer to the section of the circuit where the main computations are performed as the circuit under test. Typically, this section is located right after the qubit initialisation and before measurements, representing the core of the quantum algorithm. Figure 2 highlights the circuit under test in our example circuit, which consists of the 𝐻 gate on 𝑞 0 and two 𝐶𝑋 gates connecting all three qubits. The input of a quantum circuit is considered to be the initialisation of the qubits, as presented in the literature [51]. A qubit can be initialised in either a classical or a quantum state. To initialise ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:6

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

a qubit in a classical state, it is sufficient to set its initial state to either 0 or 1 when creating the quantum circuit. The compiler will then translate this classical state using the appropriate quantum gates. In classical initialisation, we consider the possible initialisations based on the binary combinations of all qubits, totalling 2#Qubits . For quantum initialisation, a separate quantum circuit must be constructed to generate the desired quantum state. This circuit is then introduced before the circuit under test. By doing this, the desired quantum state is achieved prior to executing the circuit under test [57]. After all the gates in the circuit have been executed, each qubit is measured to determine the output of the circuit. The output of a quantum circuit is represented as a distribution of all possible outcomes obtained at the end of the circuit. This distribution shows the probabilities of each possible measurement outcome when the circuit is executed. It is generated by running the circuit multiple times (i.e., running multiple shots) with the same input and recording each of the outcomes, which indicates how often each possible output state occurs. The resulting output distribution is then compared with the oracle output distribution to assess the circuit’s correctness.

2.2

Mutation Analysis

Mutation analysis is a technique widely used in classical software testing to evaluate testing methods. Essentially, mutation analysis involves injecting artificial faults in the programs under test and determining whether tests can detect those faults [55]. Mutation analysis employs several operators to systematically modify the program under test, thus creating faulty versions known as mutants [55]. Each of the mutants can have a different number of mutations, i.e., modifications, introduced in the program. Mutants with only one modification in the program are referred to as first order, while higher order mutants have more than one [19]. These mutants serve as a benchmark of faulty programs that can be used to assess the effectiveness of the tests. The ability of a test suite to identify faults is typically measured by the mutation score, which indicates the proportion of detected mutants out of all mutants [55]. A higher mutation score reflects a better test suite, as it is capable of detecting more faults. It is important to note that the mutation score depends not only on the test suite itself but also on the mutants used to compute it and their quality. Therefore, different mutation tools implementing different mutation operators could return different mutation scores for the same test suite [4, 21, 24]. Mutants are typically classified based on whether the tests can detect them. Inside the category of undetected mutants, we can find equivalent mutants, which represent a significant challenge in mutation analysis. Even though these mutants have been syntactically altered, they exhibit identical behaviour to the original program for all inputs, making them undetectable. Equivalent mutants pose a problem as they misleadingly decrease the mutation score, making test suites appear worse than what they actually are, and require considerable developer time to analyse [19, 41]. The detected mutants can be classified into different groups. On one hand, we have clusters of duplicated mutants, which, although they may contain different changes, behave the same. As a result, detecting one of these mutants will also result in the detection of all others in the cluster. If a large group of duplicated mutants is detected, it can artificially increase the mutation score without reflecting a real increase in fault detection capability. Subsumed mutants do not behave exactly the same but are always detected when another mutant, i.e., a subsuming mutant, is detected. Subsumed mutants also inflate the mutation score artificially, as the faults they represent are already represented by the subsuming mutant.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:7

S 𝑞 0 |0⟩

1

H

𝑞 1 |0⟩

4 2

𝑞 2 |0⟩

5 3

6

𝑐3 T Fig. 3. Quantum Circuit Operators

2.3

Quantum Mutation Analysis

Similar to other software testing techniques, mutation analysis also needs adaptation for quantum software due to its unique characteristics. To facilitate this adaptation, two quantum mutation analysis tools have been developed in the literature, Muskit [29] and QMutPy [13]. These tools treat quantum circuits as the original programs and modify them by applying gatebased operations. Both tools perform changes in the circuit by either adding a new gate, removing an existing gate, or replacing an existing gate with another one. Figure 3 illustrates examples of the application of these mutation operators to the circuit shown in Figure 2. The numbers in the figure indicate the positions where a mutation can be applied in the circuit; for the addition operator, the position is given by the gate following the gap where the mutation will be introduced. The Add operator introduces a new quantum gate to the circuit at a specific location, in this case introducing a T gate in qubit 𝑞 2 , position 3. The Remove operator eliminates a gate already present in the circuit, as it is the case in the figure for the CX gate connecting qubits 𝑞 1 and 𝑞 2 in position 3. The Replace operator substitutes an existing gate with another; the figure shows the substitution of the H gate in qubit 𝑞 0 , position 1, with an S gate. 3

Related Work

In this section, we review related work. First, we discuss prior research on both classical and quantum mutation analysis in Section 3.1. Then, we explore other relevant studies that utilise quantum circuit mutation operators in Section 3.2. 3.1

Classical Mutation Analysis

Mutation analysis has been widely studied in classical software engineering as an effective technique for assessing test suite quality and improving fault detection [55]. Surveys by Jia and Harman [19] and by Papadakis et al. [36] have highlighted its relevance, emphasising both its strong fault-revealing capability and its role in evaluating test adequacy. These studies also identify key challenges, such as the high computational cost of executing mutants and the problem of equivalent mutants, which have driven continued research into more efficient mutation analysis techniques. A recent empirical study by Sánchez et al. [44] demonstrates the practical relevance of mutation analysis in real-world development environments. The authors conducted a qualitative study with open-source developers and reported high satisfaction with mutation analysis, highlighting its usefulness in improving test quality. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:8

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

As shown in the literature review by Silva et al. [43], several approaches have already been developed for classical software to generate mutants using search-based techniques. Their analysis shows that while the genetic algorithm is the most popular and widely evaluated method, specialised algorithms often demonstrate better stability and convergence rates in specific contexts. We decided to use a genetic algorithm in this work because it serves as a well-established and widely accepted baseline for search-based optimisation, allowing us to focus on evaluating the proposed mutation operators for quantum programs. For instance, the work by Schwarz et al. [42] tries to address the limitations of traditional mutation testing by proposing a genetic algorithm to generate high-impact, undetected mutants. Similarly to QUMUG, Schwarz et al. prioritise maximising the “unique impact” of undetected mutations to identify deficiencies in test suites. Their evaluation demonstrated that an evolutionary approach can increase the percentage of undetected mutants while simultaneously increasing their average impact by a factor of five. In our case, we focus on finding undetected mutants for a given test suite by applying a novel parametric mutation operator for quantum circuit mutants and using search algorithms to tune the mutations. When referring to higher order mutants, Papadakis and Malevris [37] empirically compared first and second order mutation strategies to address the high computational and manual costs of mutation analysis. Their study of industrial C programs revealed that while using all first order mutants remains the most effective choice for fault detection, higher order mutation strategies achieve a superior cost-benefit ratio. The authors found that higher order mutation reduces the number of equivalent mutants by 80% and requires 30% fewer test cases with a moderate drop in fault detection capabilities compared to first order mutation. Recognising the potential of higher order mutants demonstrated in classical software for test suite assessment, we aimed to explore how these mutants would behave in the quantum domain. When evaluating the difficulty of detecting a mutant, Visser [46] considers various factors affecting mutant detectability through an exhaustive analysis of the input space. Unlike previous studies that relied on fixed test suites and manual analysis, Visser’s approach eliminates test-suite bias, allowing for a more accurate calculation of the “kill ratio” of mutants. In our case, as explained better in Section 5.3, we use a comprehensive test suite that includes all possible classical inputs to quantum circuits, as well as an equal number of quantum inputs. We also consider the three output measurement basis, resulting in what we believe to be a thorough test suite for assessing the difficulty of mutant detection. 3.2

Quantum Mutation Analysis

Current research in quantum mutation analysis has focused on generating mutants by introducing gate-based modifications to quantum circuits. Mendiluze Usandizaga et al. [29] introduced Muskit, a mutant generator for quantum circuits that creates mutants by adding, removing, or replacing gates within the circuit, regardless of the gate type. On the other hand, QMutPy, developed by Fortunato et al. [11–13], is a quantum circuit mutation tool built upon the classic Python mutation tool, MutPy. In addition to the mutation operators introduced by Muskit, QMutPy also offers two new mutation operators related to measurement gates. QMutPy generates mutants based on the syntactic equivalence of gates, which, as the authors state, helps reduce the overall number of generated mutants. Both tools aim to generate mutants to evaluate quantum software testing techniques and to apply mutations to Qiskit code. Mutants generated by these tools are often redundant and easy to detect, providing little help when assessing test suites. QUMUG, on the contrary, aims to generate mutations that are undetected by a given test suite, thereby forcing improvements to the test suite and making the resulting mutations more valuable for testing. To that end, we present new parametric mutation operators based on those provided by previous tools. To mutate a circuit, QUMUG adds or replaces parametric ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:9

gates and tunes them by using a search algorithm to find the optimal mutants for our goal. Also, compared to current tools, QUMUG can generate higher order mutants, which have not yet been considered for quantum circuits. Mendiluze Usandizaga et al. [30] conducted an empirical evaluation of current literature on mutants, analysing the behaviour of over 700,000 mutants across 385 quantum circuits. The authors examined how various characteristics of the circuits, algorithms, and mutations influenced the detection of these mutants, as well as the interactions between different characteristics. This study provides valuable insights into mutant behaviour in quantum circuits. In our case, instead of assessing the mutants themselves and analysing the effects of individual mutation characteristics, as Mendiluze Usandizaga et al. do, we propose QUMUG, an approach to generate mutants that are undetected by a test suite. Also, as their study is based on the mutants generated by Muskit, there is no analysis on higher order mutants, which we provide as part of our study. Some works focus on other aspects of mutation analysis. For instance, Wang et al. [53] present MutTG, a mutation-based test generation approach for finding the minimum number of test cases that maximise the number of detected mutants. The authors define mutant difficulty based on the number of test cases that detect a mutant. While this concept is related to mutant detectability, our objective focuses on generating surviving mutants that are not detected by the existing test suite, whereas Wang et al. focus on generating test cases that maximise mutant detection. We believe that QUMUG could be complementary by offering more challenging mutants to assess the effectiveness of generated test suites. Kölle et al. [23] propose a quantum circuit synthesis approach that utilises a genetic algorithm and mutates quantum circuits using gate-based operations. In this case, the authors utilise the same addition, removal, and replacement operations as Muskit and QMutPy, and introduce a new type: the swap operation. The mutation operations are not used directly to generate mutants for test assessment, but rather to help the search algorithm find optimised circuits. Authors conclude that the “swap, addition, deletion” strategy optimises quantum circuits most efficiently, producing near-optimal results and reducing circuit depth. Ishimoto et al. [18] applied quantum circuit mutations as a fault localisation strategy for quantum circuits. The authors utilised QMutPy as the mutation tool to generate mutants and assigned a suspiciousness score to each statement in the code based on how test case detection changes when that statement is mutated. The authors demonstrate that quantum-specific mutation operators are more effective at localising faults in quantum circuits than classic mutation operators. Mendiluze Usandizaga et al. [45] presented QRep, an automated quantum circuit repair approach by gate prioritisation. The authors implemented gate-based modifications, similar to mutations described in the literature, to faulty circuits using an iterative approach that prioritises quantum gates. The authors define a suspiciousness score based on how each gate’s modification affects the circuit, and rank the gates by their likelihood of being faulty. The results show that QRep repairs 70% of faults in circuits with up to 13 qubits, offering greater scalability than other methods in the literature. This study highlights that gate-based modifications can be applied to both quantum circuit fault localisation and repair. The authors also mention that future work may involve multi-gate modifications to the circuit to enable repair of additional faults. QUMUG focuses solely on mutation analysis, with the main objective of generating mutants that contribute to the improvement of the test suite. However, as we present five new operators for generating mutations, we believe these operators could also be useful in other quantum software applications. Also, as mentioned in the work by Ishimoto et al. [18], the inclusion of higher order mutations could help in several domains apart from mutation analysis and testing.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:10

4

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

QUMUG: Proposed search-based quantum mutants generator

QUMUG is motivated by the fact that the quantum circuit mutations proposed in the literature drastically affect the circuit’s behaviour. These alterations lead to substantial changes in the circuit’s output, making any differences easy to observe, regardless of the tests employed. The majority of the tests trigger the fault, making the mutants trivial or easy-to-detect. Based on this observation, we aim to create mutants that would produce minor effects on the circuit, so that they would require a stronger test suite to be detected. In this section, we will discuss the methodology and implementation of QUMUG. 4.1

Overview

We propose a new type of parameterisable mutation operators. Similar to existing quantum mutation tools Muskit and QMutPy, these operators generate mutants by applying gate-based modifications; however, rather than using predefined or fixed gates, QUMUG inserts parameterisable gates into the circuit. These gates allow us to control the magnitude of the quantum phase applied to a state, providing control over the strength of the mutation. Another positive aspect of this type of mutation is the diversity it provides. The tools available in the literature are restricted by fixed angle values, resulting in a limited number of mutants. In our case, since the parameters adjusted in the gates are floating-point values, this presents infinite possibilities for creating new mutants that have not been previously explored, each with a specific phase shift. However, this also leads to an infinite number of mutants, which makes their execution infeasible. Therefore, we introduce a search-based algorithm to find the optimal parameters for each mutation. This strategy reduces the number of mutants and discards those that introduce drastic changes to the circuit’s behaviour. In addition, we explore the possibility of introducing multiple mutations within the same quantum circuit, thereby generating higher order quantum mutants. To the best of our knowledge, there has not been a method for applying this concept in quantum circuits, even though it is a widely used strategy in classical software testing [19]. By generating higher order mutants, we also expand our search space, as we could explore the interactions of different mutations in the same or different qubits. 4.2

Formal Definitions

We now introduce the notation and fundamental concepts used in the remainder of the section. Definition 1 (Circuit Under Test). As introduced in Section 2.1.2, the circuit under test represents the main computational component consisting of a sequence of quantum gates that capture the internal logic of the quantum computation. In our study, mutations are applied exclusively to the circuit under test. In the following, for the sake of brevity, we will call the circuit under test simply as circuit. We define our original circuit as C, and the mutated circuit as C ′ , which is the circuit obtained by applying one or more mutation operations to C, resulting in a modified circuit structure. The initialisation and measurement components that define the execution context of C and C ′ are handled separately through test cases (see Definition 2). Definition 2 (Test Case). A test case, denoted by tc, is an element that defines the execution context for C. Formally, a tc is defined as a pair tc = (i, m), where i denotes the circuit initialisation applied prior to executing C, and m denotes the measurement basis used after execution to obtain the resulting output (e.g., 𝑋 , 𝑌 , or 𝑍 basis). ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:11

Definition 3 (Test Suite). We define a test suite TS = {tc 1, . . . , tc𝑘 } as a set of test cases used to evaluate a circuit. In our context, TS is used to determine whether a mutated circuit C ′ passes or fails the evaluation criteria by executing both C and C ′ under the same TS. Definition 4 (Output Distribution). The output distribution of a circuit C for a test case tc is the probability distribution over measurement outcomes obtained by executing the circuit in the context specified by tc. Formally, it represents the probabilities of observing each possible measurement outcome o among all possible measurement outcomes O: ∑︁ DC (tc) = {Pr(o | C, tc) | o ∈ O} where Pr(o | C, tc) = 1. o∈O

The original output distribution, denoted by DC (tc), is obtained by executing the original circuit C multiple times (shots) in the context defined by a test case tc. The mutant output distribution, denoted by DC ′ (tc), is obtained by executing a mutant circuit C ′ in the same context defined by tc. Definition 5 (Mutant Detection). Mutant detection refers to the process of determining whether C ′ can be distinguished from C under a given tc. For each test case tc ∈ TS, a statistical test is applied to compare the original output distribution DC (tc) with the mutant output distribution DC ′ (tc). A mutant is considered to be detected by a test case tc if the resulting p-value of the statistical test is below a predefined significance threshold 𝛼, indicating a statistically significant difference between the two distributions. ( 1, if StatisticalTest (DC (tc), DC ′ (tc)) < 𝛼, ′ Detect (C, C , tc) = 0, otherwise. While mutant detection determines whether a statistically significant difference exists between the output distributions of the original and mutated circuits, it provides only a binary indication of their difference. To better assess these differences, we use a distance metric to measure the distance between the two distributions. This continuous measure helps detect subtle behavioural changes in mutants. Definition 6 (Distance Metric). In our context, a distance metric quantifies the similarity between two probability distributions, considering both the differences in their probabilities and the shapes of their distributions. We denote the distance for circuits C and C ′ in the context specified by tc as Distance(C, C ′, tc). It is used to measure the distance between the original output distribution DC (tc) and the mutant output distribution DC ′ (tc). It ranges from 0 to 1, where 0 indicates matching distributions and 1 indicates completely different distributions. 4.3

Mutation Operators

In QUMUG, we use the three basic rotational gates (𝑅𝑥 , 𝑅 𝑦 , and 𝑅𝑧 ) along with the universal gate (𝑈 ). We chose these gates as they allow us to generate mutants with minimal impact on the circuit. Rotational gates enable us to specify the desired phase shift of a qubit in a specific direction without altering other directions. This capability allows us to specify the magnitude of the change applied to the qubit. The universal gate, on the other hand, can represent any single-qubit gate by applying specific angle shifts to all three phases simultaneously. We present five parameterisable mutation operators, listed below: • 𝐴𝑑𝑑_𝑅𝑥 : it inserts an 𝑅𝑥 gate parameterised by 𝜃 , introducing a rotation around the 𝑋 -axis. • 𝐴𝑑𝑑_𝑅 𝑦 : it inserts an 𝑅 𝑦 gate parameterised by 𝜃 , introducing a rotation around the 𝑌 -axis. • 𝐴𝑑𝑑_𝑅𝑧 : it inserts an 𝑅𝑧 gate parameterised by 𝜙, introducing a rotation around the 𝑍 -axis. • 𝐴𝑑𝑑_𝑈 : it inserts a 𝑈 gate parameterised by 𝜃, 𝜙, 𝜆, allowing for single-qubit state transformations. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:12

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

• 𝑅𝑒𝑝𝑙𝑎𝑐𝑒_𝑈 : it replaces an existing single-qubit gate with a 𝑈 gate parameterised by 𝜃, 𝜙, 𝜆. We chose to focus on adding a new gate rather than removing or replacing an existing one, except for the 𝑈 gate. This is because removing or replacing an existing gate can significantly affect the original circuit, leading to major alterations. Replacing a gate with a parameterised 𝑈 gate is included as an option since the 𝑈 gate can mimic the behaviour of any single-qubit gate. This option allows a close approximation of the replaced gate, which could lead to subtle deviations if tuned correctly. 4.4

Individual Representation

In QUMUG, an individual I represents a mutant C ′ derived from C, encoded as an array of mutation parameters that are applied to C to generate the resulting mutated circuit C ′ . In the case of a single mutation (i.e., a first order mutant), the individual will consist of a single array that represents the five parameters needed to generate a mutation: 𝐼 firstOrder = [P, M, 𝜃, 𝜑, 𝜆] These parameters include Position P ∈ {1, . . . , |C|}, which indicates the location in C where the mutation is applied; mutation operator M ∈ {𝐴𝑑𝑑_𝑅𝑥 , 𝐴𝑑𝑑_𝑅 𝑦 , 𝐴𝑑𝑑_𝑅𝑧 , 𝐴𝑑𝑑_𝑈 , 𝑅𝑒𝑝𝑙𝑎𝑐𝑒_𝑈 }, where M represents a possible mutation selected from the set of available mutation operators; and three phase values 𝜃, 𝜑, 𝜆 ∈ [−180, 180], which represent the floating-point parameters controlling the phase settings. In the case of generation of higher order mutants, the individual will consist of several arrays, depending on the desired number of mutations, each containing the same five variables: 𝐼 highOrder = [[P1, M1, 𝜃 1, 𝜑 1, 𝜆1 ] . . . [P𝑛 , M𝑛 , 𝜃 𝑛 , 𝜑𝑛 , 𝜆𝑛 ]] In the following, with an abuse of notation, we will indicate an individual directly with the obtained mutant C ′ . 4.5

Fitness Function

The fitness function measures the impact of mutations on the original circuit. The goal is to identify mutants that pass the test suite given in input (TS) while still causing a behavioural change in the circuit. We first define tcd as the number of test cases that detect the mutant C ′ : ∑︁ tcd (C ′, TS) = Detect (C, C ′, tc) (1) tc∈TS

For each mutant, the sum of the distances of all test cases is also calculated, providing insight into how close the mutant is to passing the test cases. This is defined as EffectMeasure and is used in the fitness function to help guide the search towards the desired direction. ∑︁ EffectMeasure(C ′, TS) = Distance(C, C ′, tc) tc∈TS

Therefore, the fitness function to be minimised is defined as follows: ( tcd (C ′, TS) × #tc + EffectMeasure(C ′, TS) if tcd (C ′, TS) > 0 ′ 𝑓 (C ) = #tc − EffectMeasure(C ′, TS) if tcd (C ′, TS) = 0

(2)

The fitness function is defined utilising two cases, each consisting of a method with two terms. This first case is applied when at least one test case is able to detect the mutant (i.e., tcd (C ′, TS) > 0) and starts by multiplying the number of tests that detect the mutant by the total number of tests in the given test suite (i.e., tcd (C ′, TS) × #tc). By doing so, the function assigns more relevance to each test case that passes, increasing its impact on the fitness function. To guide the search towards ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:13

the passing of the test cases, EffectMeasure(C ′, TS) is added, providing some more insight into the effect of the introduced mutations. The second case in the fitness function is utilised if all the test cases in the provided test suite pass, (i.e., tcd (C ′, TS) = 0). In this case, the objective is to maximise the effect of the mutation, ensuring that the mutant is not equivalent, even after successfully passing all the tests in the given suite. To achieve this, EffectMeasure(C ′, TS) is subtracted from the total number of given test cases #tc, that way ensuring that the fitness function remains a minimisation function and will always produce a smaller value than the best outcome of the first case. 5

Experiment Design

In this section, we explain how we plan to evaluate QUMUG. First, in Section 5.1, we define the research questions we aim to answer and outline the methods we will use to address them. Then, in Sections 5.2 and 5.3, we present the subject systems and the test suite TS that will be used by QUMUG. In Section 5.4, we define how we assess the equivalence of mutants. In Section 5.5, we present the approaches compared in our experiments: Section 5.5.1 describes the search algorithms used by QUMUG, and Section 5.5.2 the mutation approaches from the literature used as baselines. Then, in Section 5.6, we define the experimental setup used in our experiment. In Sections 5.7 and 5.8, we present the metrics and statistical tests used to evaluate the approach, respectively. Finally, in Section 5.9, we describe the execution environment utilised in the experiment. All detailed experiment results, code, and data are available in the online repository [31]. 5.1

Research Questions (RQs)

The objective of QUMUG is to generate mutants that represent faults not detected by the provided test suite, to help improve it. To assess both the effectiveness of the proposed generation strategy and the characteristics that influence the detectability of these mutants, we define the following research questions: RQ1: How good is QUMUG in generating non-equivalent, undetected quantum circuit mutants that contribute to the improvement of the test suite? RQ1.1: How many non-equivalent mutants that pass the user test suite are generated? RQ1.2: How challenging to be detected are the generated mutants? RQ1.3: Do the generated mutants contribute to the inclusion of more test cases in the test suite? In this RQ, we assess whether QUMUG can generate mutants that are not detected by the test suite given as input and are not equivalent, and whether these mutants are more challenging and contribute more to improving the test suite than mutants generated with existing tools. This allows us to evaluate the practical value of the mutation generation strategy. For that, in RQ1.1, we compare the ability of each search approach to generate undetected, non-equivalent mutants and compare them with those generated by tools in the literature. Then, in RQ1.2, we further analyse the generated mutants in terms of how challenging they are to detect compared to those generated by the tools in the literature. Finally, in RQ1.3, we assess how much the generated mutants contribute to improving the test suite, evaluating the need to add new test cases to the test suite given in input. RQ2: What is the impact of higher order mutation on the generation and detection of quantum circuit mutants? RQ2.1: How do higher order mutants compare to first order mutants? RQ2.2: How do mutations in the same circuit interact with each other in higher order mutants? RQ2.3: How do interactions between mutations affect test case detection of higher order mutants? ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:14

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Table 2. Characteristics of the original benchmarks (i.e., quantum algorithms).

Algorithm

Output type

# qubits

# gates

# single-qubit gates

# multi-qubit gates

Depth

ae ghz qft qpe vqe w-state

Dominant Diverse Diverse Dominant Dominant Diverse

3-7 3-7 3-7 3-7 3-7 3-7

15-53 4-8 8-32 9-36 14-34 10-26

9-25 1 3-7 5-13 9-21 5-13

6-28 3-7 5-25 4-23 5-13 5-13

12-36 4-8 7-15 7-16 8-12 7-15

Amplitude Estimation (ae); Greenberger-Horne-Zeilinger state (ghz); Quantum Fourier Transform (qft); Quantum Phase Estimation (qpe); Variational Quantum Eigensolver ( vqe); W-State (w-state).

In RQ2, we examine the role of higher order mutation in quantum mutation analysis. Here, we aim to determine whether combining multiple mutations results in mutants that are more or less challenging, and whether interactions among mutations lead to unique behavioural changes in quantum circuits. To answer this question, we utilise the mutants generated by the best search algorithm in RQ1 and, first, in RQ2.1, we compare first order mutants with higher order mutants using the metrics from RQ1 to assess their effectiveness. Then, we further analyse the interactions of the mutations in higher order mutants in terms of mutation characteristics and their combination in RQ2.2, as well as the test case detection for each mutation in RQ2.3. 5.2

Subject Systems

For our evaluation, we selected six quantum algorithms, each having five quantum circuit implementations that range from three to seven qubits, totalling 30 quantum circuits. We decided to use a minimum of three qubits in our circuits because one of the algorithms (VQE) cannot be represented with fewer qubits. We imposed an upper limit due to resource constraints. These circuits were sourced from MQT Bench [39], a widely recognised benchmark suite for quantum circuits. The selection of these circuits was guided by the different output types outlined in [30], which categorise quantum algorithms into two groups: dominant output and diverse output. Dominant output algorithms aim to identify the quantum state with the highest probability, while diverse output algorithms focus on the distribution and relationships between the probabilities of multiple states. We chose three quantum algorithms from each category, ensuring a diverse set of circuits that represent various types of quantum circuits. The specific characteristics of the selected quantum circuits are summarised in Table 2. 5.3

Test Suite

The test suite TS used in this study is defined by combining different quantum circuit initialisations and output measurement bases (see Definition 2). First, we randomly generate 20% of all possible classical input initialisations for each qubit number. This number was chosen based on preliminary experiments showing that a smaller number of inputs resulted in test suites being too weak. Moreover, this percentage was chosen to avoid excessive time consumption during the experiments, as the test suite needs to be executed several times during the search. Then, as the number of quantum initialisations can be infinite, we choose to generate a number of quantum initialisations equivalent to the number of classical initialisations generated. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:15

To generate these quantum states, we utilised a method proposed by Ye et al. [57]. Among the various methods suggested in [57] for creating quantum states with template-based quantum circuits, we opted for the UCNOT approach, as, according to their evaluation, it is the most effective for detecting mutants. This method uses a template of 𝑈 gates and 𝐶𝑋 gates to connect all qubits and generate random quantum states. By combining both classical and quantum inputs, we achieve a number of possible initialisations proportional to the number of qubits in the circuit: 2#𝑄𝑢𝑏𝑖𝑡𝑠 × 0.4. These inputs are not specific to any particular quantum algorithm but depend on the number of qubits in the circuit. Algorithms that use the same number of qubits will share identical inputs for test cases. Regarding the circuit’s output, we decided to increase the variability of our measurement approach. As each qubit can be measured in three different bases (𝑋 , 𝑌 , and 𝑍 ), we decided to include all three bases in our test suite. This adds another layer of variability to our test cases. Consequently, each of the inputs mentioned earlier will produce three possible outputs, one for each base. As a result, the total number of test cases is 2#Qubits × 0.4 × 3. To address the inherent uncertainty in quantum computing, each test was executed several times for a given number of shots. We performed a number of shots proportional to the number of qubits, calculated as 2#qubits × 2. This approach enabled us to obtain a good estimate of the output distribution. 5.4

Non-equivalent Mutant Validation

The objective of QUMUG is to generate mutants that represent faults not covered by TS. Therefore, in the evaluation of QUMUG, it is necessary to distinguish between mutants that pass the test suite because they represent uncovered faults and mutants that are equivalent to the original circuit. Equivalent mutants do not provide useful information about the test suite, as their behaviour cannot be distinguished from the original circuit. To assess mutant equivalence, we extend TS to create a more comprehensive test suite, denoted as CTS. Since CTS is exclusively utilised for the external evaluation of the generated mutants and is not included in the search process, its execution time does not limit the approach. Therefore, we include all possible classical input initialisations left and the same number of quantum initialisations, also measured across all three bases. As a result, CTS is five times larger than TS. A generated mutant is considered non-equivalent if it is detected by at least one test case from CTS. In other words, if the comprehensive test suite is able to distinguish the mutant from the original circuit, the mutant is considered to exhibit an observable behavioural difference and, therefore, represents a fault not covered by the original test suite TS. This validation process allows us to identify mutants that pass TS while still representing behaviours that could be detected by additional testing. 5.5

Compared Approaches

In this section, we describe the approaches that will be compared to assess the effectiveness of QUMUG. First, in Section 5.5.1, we outline the four search algorithms that will be utilised by QUMUG. Then, in Section 5.5.2, we discuss the state-of-the-art mutant generators that we will use as baselines for QUMUG. 5.5.1 Search Algorithms used in QUMUG. In this work, we evaluate the performance of QUMUG using four different search algorithms: Genetic Algorithm (GA), which is included as it is one of the most widely used evolutionary algorithms in the literature; Hill Climbing (HC), which we include as a local search strategy; 1+1 Evolutionary Algorithm ((1+1)-EA), which serves as a simple evolutionary baseline; and Random Search (RS) as a baseline without any learning or guided ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:16

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Table 3. Total number of mutants produced by each baseline tool

Algorithm

Muskit

QMutPy

Total

ae ghz qft qpe vqe w-state

4150 665 2407 2691 2955 2205

2120 580 1754 1998 1560 1380

2535 1245 4161 4689 4515 3585

Total

15073

9392

24465

search process, providing a reference point for comparison. Each of these algorithms represents a distinct search strategy, allowing us to assess the behaviour of QUMUG under diverse optimisation dynamics. Regarding the search operators, we needed to adjust some commonly used ones to fit our encoding and search. In the crossover step for GA, the parents are combined using a personalised crossover operator designed specifically for our solution representation, a variation of the commonly used single-point crossover. Since individuals have different settings based on the number of mutations desired, we decided to implement a specific version of single-point crossover for each type. In cases where multiple mutations are desired, i.e., higher order mutants, the individuals are split in a way that each mutation remains intact. Practically, this means that the points selected during the crossover operation can only occur between arrays, without splitting any specific sub-array and modifying individual mutations. For first order mutations, since they consist solely of one mutation, the crossover point is chosen between the variables of that single mutation and combined with another to create a new mutation. Regarding the mutation operator (for the three search algorithms that utilise it, i.e., GA, HC and (1+1)-EA), we utilised a normalised mutation operator that modifies the individual by selecting a random value from a normalised distribution. It is important to note that all external components remained consistent in all search algorithms, and the generated mutants were evaluated in the same manner. 5.5.2 State-of-the-art Mutant Generators. We compare QUMUG with two existing approaches discussed in the literature Muskit [29] and QMutPy [13]. Both approaches systematically generate quantum circuit mutants by applying gate-based operations, with the sole objective of generating all possible mutants. Our primary objective is to assess how easy-to-detect their mutants are, and to determine whether QUMUG can generate more challenging mutants. To achieve this, we generated mutants using both state-of-the-art approaches. Table 3 displays the total number of mutants obtained from each mutation generator when applied to the same selected quantum circuits. For our evaluation, we select only the mutants that pass the test suite TS given in input, as these are the mutants the user would be interested in. Then, we evaluate whether the remaining mutants are detected by CTS to determine if these mutants are equivalent or represent faults that were not detected by TS. Then, both tools are compared against the search-based approaches utilising QUMUG. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

5.6

X:17

Experimental Setup

To conduct our experiments, we need to establish several key parameters. Initially, we conducted a preliminary experiment using a small set of circuits with up to five qubits 1 . This preliminary experiment allowed us to observe the behaviour of GA and to fine-tune its parameters accordingly. We employed a grid search algorithm to identify the optimal parameters for GA operators, which indicated that the optimal parameters for both first order and higher order mutants include a population size of 100, with a mutation probability (Mutpb) of 0.5 and a crossover probability (Cxpb) of 0.5. To maintain consistency in our experimental design, we allocated a budget of 100 generations for each execution. For GA, this means 100 generations with a population of 100. To ensure a fair comparison with the other algorithms, we based the experiments on the number of evaluations, allocating a maximum of 10,000 evaluations (100 generations multiplied by 100 individuals in a population) for each search algorithm. Also, a preliminary assessment of the mutation operators revealed that the vast majority of the selected angles for the non-detected mutants found in the search fell within the range of -0.5 to 0.5 for all the gate parameters. This observation allowed us to reduce the search space from the full 360 degrees to a range of 1 degree, specifically from -0.5 to 0.5, as the remaining values seemed to have too much impact and were therefore irrelevant to our case. In this study, we limited the number of mutations included in our higher order mutants to two, thereby creating second order mutants. We made this decision due to time and resource constraints, and we believe that incorporating two mutations into the circuit adequately captures the potential for interactions among multiple mutations. As part of the fitness evaluation (see Section 4.5), we need to select a statistical test to identify mutant detection (see Definition 5) and a distance metric (see Definition 6) to quantify the difference between the original and mutated circuits for each test case. For the statistical test, we have chosen the chi-square test with a significance level 𝛼 of 0.05, which is commonly used in the literature [29, 30, 47, 51]. For the distance metric, we opted for the Hellinger distance, as it is one of the most frequently employed methods in the literature for measuring differences between output distributions [14, 32, 33, 38]. The Hellinger distance ranges from 0 to 1, where 0 indicates matching distributions and 1 indicates completely different distributions and is defined as follows: √︄ 2 √︁ 1 ∑︁ √︁ ′ Pr(o | C, tc) − Pr(o | C ′, tc) Hellinger (C, C , tc) = √ 2 o∈O To account for the stochasticity of QUMUG and the resource constraints inherent in executing quantum circuits, each experiment was repeated 10 times under identical experimental conditions. Note that the mutant generators in the literature used as baselines (see Section 5.5.2), are deterministic in terms of the number of generated mutants but still contain a stochastic factor related to the gate parameters; thus, they were also executed 10 times for each circuit. Also, as the generation of TS has a stochastic factor in selecting the inputs, we generated and stored 10 different TS to ensure that all the approaches are compared using the same TS in each run. 5.7 Evaluation Metrics To answer the research questions presented in Section 5.1, this section describes how the effectiveness of QUMUG is evaluated. 5.7.1 Non-equivalent mutants passing TS. To evaluate the effectiveness of the approaches in generating mutants that represent faults not covered by TS, we measure the number of generated 1 The results of this preliminary experiment can be found in the external repository containing all the results [31].

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:18

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

mutants that pass the test suite TS and are classified as non-equivalent according to the validation process described in Section 5.4. We report both the total number of mutants passing TS and the total number of non-equivalent mutants. We also calculate the proportion of non-equivalent mutants among all mutants that pass TS. The total number of non-equivalent mutants obtained provides information about the number of useful mutants generated by each approach, while the proportion of non-equivalent mutants among all mutants that pass TS accounts for the possibility of obtaining equivalent mutants. These metrics are used to answer RQ1.1 and RQ2.1. 5.7.2 Test case detection ratio. To evaluate how challenging the generated mutants are to detect, we measure the proportion of test cases from CTS that are able to detect each mutant. For a mutant C ′ , its detection ratio is defined as: DetectionRatio(C ′, CTS) =

tcd (C ′, CTS) × 100 |CTS|

(3)

where tcd is defined as in Equation 1. Lower detection ratios indicate more challenging mutants, as fewer test cases are able to expose the behavioural differences between the mutant and the original circuit. For each approach, we report the median detection ratio across all generated mutants. This metric is used to answer RQ1.2 and RQ2.1. 5.7.3 Test suite contribution. To assess whether the generated mutants contribute to the improvement of TS, we measure the number of additional test cases required to detect the generated mutants per program. Starting from TS, we iteratively select new test cases from CTS based on the number of mutants they detect. At each step, the test case detecting the largest number of remaining undetected mutants is selected and added to the test suite until all generated mutants are detected. The final count of added test cases represents the number required to detect all generated mutants. A larger number of required test cases indicates that the generated mutants represent a more diverse set of faults and provide greater potential for test suite improvement. This metric is used to answer RQ1.3 and RQ2.1. 5.7.4 Second order mutants characteristics. To analyse the characteristics associated with successful higher order mutants, we measure the frequency of each mutation characteristic among the generated mutants. This analysis allows us to identify which mutation characteristics and combinations are more frequently associated with higher order mutants that satisfy the main objective of QUMUG, i.e., generating undetected mutants that contribute to the improvement of the test suite. This metric is used to answer RQ2.2. 5.7.5 Second order mutants test case detection. To analyse how combining mutations affects mutant detection, we compare the test cases that detect individual mutations with those that detect the corresponding higher order mutant. Let tcd fom denote the set of unique test cases that detect the constituent first order mutants independently, and let tcd hom denote the set of test cases that detect the corresponding higher order mutant. By comparing these two sets, we analyse whether the combination of mutations keeps the detection behaviour observed for the individual mutations or whether interactions between mutations lead to different detection sets. This metric is used to answer RQ2.3. 5.8

Statistical Analysis

To account for the stochastic nature of the search-based approaches, we performed statistical analyses to compare the different mutation generation approaches across the metrics mentioned ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:19

in Section 5.7. Each approach was executed using 10 independently generated test suites for each subject system. Since these executions represent repeated measurements of the same quantum circuit rather than independent observations, we considered the circuits as the experimental unit. Therefore, for each metric and approach, the results obtained from the 10 runs were aggregated using the median, producing a single representative value per circuit while preserving robustness against variability introduced by the stochastic approaches. We used the Friedman test [10] to assess whether there were statistically significant differences among the compared mutation generation approaches. The Friedman test is a non-parametric alternative to repeated-measures ANOVA and is appropriate for comparing multiple related samples. Since all approaches were evaluated on the same set of circuits, the results were treated as paired observations. When significant differences were found, pairwise comparisons were performed using the Wilcoxon signed-rank test [54] with Holm correction [15]. For measuring the effect size of the comparison, we utilise the Cliff’s Delta (𝛿) [7], which measures how large and in what direction differences occur using probabilistic dominance. This effect size allows us to evaluate how substantial the observed differences are; we define the strength of the effect size as follows [28]: (i) Negligible: |𝛿 | < 0.15 (ii) Small: 0.15 ≤ |𝛿 | < 0.33 (iii) Medium: 0.33 ≤ |𝛿 | < 0.47 (iv) Large: |𝛿 | ≥ 0.47. By using this combination of tests, we can assess not only whether differences exist but also how substantial those differences are across the evaluated approaches. For count-based metrics, missing values caused by the absence of generated mutants were replaced by zero because they represent the absence of generated mutants. For test case detection ratio, undefined values caused by zero generated mutants were excluded because they do not represent failed detection but rather the absence of mutants to evaluate. This ensures that comparisons are performed only on valid paired observations. 5.9

Execution Environment

All original and mutated circuits were executed under the same conditions, utilising the same computational resources. The experiments were conducted on a national high-performance cluster of servers, which included 2 AMD Epyc 7601 processors, 2TB of RAM, an AMD Vega20 GPU, and a high-speed 4TB NVMe drive. All quantum circuits were written in QASM, and we used Qiskit version 1.1.0 to create and modify the circuits as needed. To enhance consistency and reproducibility, a fixed random seed was employed during the simulations. 6

Results and Analysis

In this section, we present the results from the experiments described in Section 5. We implement QUMUG, using four search algorithms, and compare it against two baseline quantum circuit mutation generators. We also examine the effects of introducing either single or multiple mutations in a quantum circuit. 6.1

Results for RQ1 – QUMUG effectiveness

In this first research question, we evaluated QUMUG for finding the desired mutants. In RQ1.1 (Section 6.1.1), we assessed QUMUG’s ability to identify mutants that pass all tests in TS and verify that the mutants found are actually non-equivalent (i.e., detected by CTS). In RQ1.2 (Section 6.1.2), we analysed whether the obtained mutants are detected by fewer test cases than those generated by the baseline approaches, indicating that they are more challenging mutants. Finally, in RQ1.3 (Section 6.1.3), we examined whether these mutants improve the given test suite, analysing how many new test cases must be added to TS to detect the generated mutants. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:20

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Fig. 4. RQ1.1 – Median number of mutants passing and non-equivalent by search across all circuits per run.

6.1.1 Results for RQ1.1 – Non-equivalent mutants that pass TS. We first report the success rates of different approaches at finding at least one mutant that is not detected by TS and is not equivalent for each quantum circuit. The results showed that Muskit and GA achieved the highest success rates, at 100% (300/300) and 99.67% (299/300), respectively; GA failed in only one run for the 7-qubit QFT. The other approaches show a success rate of 86.33% (259/300) for RS, 83.67% (251/300) for QMutPy, 82% (246/300) for (1+1)-EA and 76.33% (229/300) for HC with the lowest success rate. This success rate reflects the ability of different approaches to find at least one mutant for a quantum circuit, but it does not indicate the number of mutants obtained or their quality. To assess these aspects, we perform more detailed analyses, as explained in the following. Figure 4 shows the median number of mutants obtained from running the approaches for all 30 quantum circuits. The left bar (lighter colour) represents the number of mutants passing TS, and the right bar (darker colour) shows the number of mutants that are not equivalent based on CTS. The whiskers on the top of the bars represent the standard deviation of the values across the runs. The figure shows that the approaches that obtain the majority of mutants that pass TS are (1+1)-EA and GA, while the literature tools Muskit and QMutPy are those that generate the fewest. The value highlighted in the bar shows the percentage of mutants not detected by TS that were non-equivalent; results indicate that GA is better at optimising the effect of the mutants to make them pass TS but still be detectable. The literature tools show the worst performance by having only about half of the mutants that pass to be non-equivalent. To further support these results, we employed a statistical test to determine whether the observed differences are significant. First, the Friedman test revealed statistically significant differences among the compared approaches across all evaluated metrics (𝑝 < 0.05), indicating that at least one approach differs from the others. To further identify where these differences occur and quantify their magnitude, we conducted post hoc pairwise comparisons using the Wilcoxon signed-rank test with Holm correction, along with effect size analysis using Cliff’s delta. Table 4 reports the pairwise comparison results for all the approaches. The statistical tests confirm that GA significantly outperforms all other approaches, except (1+1)EA, in obtaining a higher number of mutants that pass TS and a higher number of non-equivalent mutants. It shows a medium effect against the other search algorithms, HC and RS, and a large effect size against the literature tools Muskit and QMutPy. Regarding the non-equivalent ratios ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:21

Table 4. RQ1.1 – Pairwise statistical comparison of the approaches for generated mutants, non-equivalent mutants and non-equivalent ratio. Effect sizes are reported using Cliff’s 𝛿 with qualitative interpretation.

#Passed TS

#Non-equivalent

Non-equivalent ratio

Comparison

𝑝-value Effect size (𝛿) 𝑝-value Effect size (𝛿) 𝑝-value Effect size (𝛿)

GA vs (1+1)-EA GA vs HC GA vs RS GA vs Muskit GA vs QMutPy (1+1)-EA vs HC (1+1)-EA vs RS (1+1)-EA vs Muskit (1+1)-EA vs QMutPy HC vs RS HC vs Muskit HC vs QMutPy RS vs Muskit RS vs QMutPy Muskit vs QMutPy

0.31 – 0.17 – < 0.05 Medium (0.34) < 0.05 Medium (0.37) < 0.05 Medium (0.44) < 0.05 Medium (0.46) < 0.05 Large (0.52) < 0.05 Large (0.77) < 0.05 Large (0.94) < 0.05 Large (0.97) 0.06 – < 0.05 Small (0.19) < 0.05 Small (0.30) < 0.05 Small (0.31) 0.07 – < 0.05 Medium (0.44) < 0.05 Large (0.65) < 0.05 Large (0.75) 0.20 – < 0.05 Small (0.13) 0.31 – 0.07 – < 0.05 Medium (0.36) < 0.05 Large (0.55) 0.38 – 0.17 – 0.21 – < 0.05 Small (0.30) < 0.05 Large (0.90) < 0.05 Large (0.83)

0.41 – 0.45 – < 0.05 Medium (0.44) < 0.05 Large (1.00) < 0.05 Large (0.97) 0.91 – 0.45 – < 0.05 Large (0.91) < 0.05 Large (0.81) 0.45 – < 0.05 Large (0.78) < 0.05 Large (0.68) < 0.05 Large (0.73) < 0.05 Large (0.68) 0.45 –

among the mutants that pass, GA still achieves a significant difference from RS and the literature tools, with medium and large effect sizes, respectively. The comparison with (1+1)-EA and HC shows that the differences are not significant, highlighting that the three approaches might have similar non-equivalent ratios, as shown in Figure 4. Among the rest of the search-based approaches (i.e., apart from GA), the observed differences are either non-significant or with only small effect sizes, suggesting that these approaches achieve comparable performance. When compared to the literature tools in terms of the number of mutants passing TS, Muskit exhibits performance comparable to the search-based approaches, as no significant differences were observed. In contrast, QMutPy differs significantly from (1+1)-EA and HC, with large and medium effect sizes, respectively. A similar trend is observed for the number of non-equivalent mutants: Muskit remains comparable to the search-based approaches, differing significantly only from (1+1)-EA, with a medium effect size; QMutPy again shows significant differences against all search-based approaches, with large effect sizes for (1+1)-EA and HC, and a small effect size for RS. Regarding the ratio of non-equivalent mutants, all search-based approaches significantly outperform both literature-based tools, consistently exhibiting large effect sizes. Overall, QMutPy demonstrates the weakest performance, as Muskit also significantly outperforms QMutPy in terms of both the number of mutants passing TS and the number of non-equivalent mutants, with large effect sizes. Concluding remarks for RQ1.1: GA and (1+1)-EA demonstrated the highest effectiveness in generating both the highest number of mutants passing the test suite and the number of them that are not equivalent. All search approaches showed superior performance in terms of the non-equivalent ratio compared to Muskit and QMutPy.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:22

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Fig. 5. RQ1.2 – DetectionRatio for the mutants generated by each approach. Table 5. RQ1.2 – Pairwise statistical comparison of the approaches for the DetectionRatio.

Comparison

𝑝-value

Effect size (𝛿)

Comparison

𝑝-value

Effect size (𝛿)

(1+1)-EA vs Muskit HC vs Muskit (1+1)-EA vs QMutPy GA vs QMutPy GA vs RS (1+1)-EA vs RS GA vs HC (1+1)-EA vs HC

< 0.05 < 0.05 < 0.05 < 0.05 < 0.05 < 0.05 0.79 0.79

Large (-1.00) Large (-1.00) Large (-0.84) Large (-0.83) Small (0.21) Negligible (0.13) – –

GA vs Muskit RS vs Muskit RS vs QMutPy HC vs QMutPy HC vs RS Muskit vs QMutPy GA vs (1+1)-EA –

< 0.05 < 0.05 < 0.05 < 0.05 < 0.05 0.79 0.79 –

Large (-1.00) Large (-1.00) Large (-0.84) Large (-0.83) Small (0.18) – –

6.1.2 Results for RQ1.2 – Test case detection. In this RQ, we want to evaluate how challenging the mutants generated by each approach are. For that, we consider only the mutants that pass TS and are not equivalent, as these are the mutants relevant for our study. Figure 5 shows, for each approach, the distribution of DetectionRatio across the mutants from CTS as defined in Section 5.7.2. In this RQ, we treat each mutant generated across all runs and quantum circuits as a data point. The value highlighted shows the median DetectionRatio obtained. The figure clearly shows that the four search-based approaches achieve mutants with lower DetectionRatio compared to Muskit and QMutPy, meaning the obtained mutants are more challenging to detect. Among the search-based approaches, there appears to be some similarity in detection distributions, with all showing triangular shapes that mostly focus on lower #tc detections. To verify the findings, we also conducted the same statistical tests as in the previous experiment; the Friedman test again revealed that at least one approach differs from the others significantly (𝑝 < 0.05). The results of the pairwise comparison are shown in Table 5. The statistical tests confirmed our conclusion from the figure, showing that all search algorithms have significantly lower detection rates for the generated mutants than the literature tools, with a large effect size. Among them, as noted earlier, the statistical tests confirm that there is no big difference in DetectionRatio, with only RS having a small or negligible effect size relative to the other search approaches. This ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:23

Fig. 6. RQ1.3 – New tc needed to detect mutants generated.

indicates that the mutants that pass TS and are not equivalent across approaches have similar DetectionRatio. Concluding remarks for RQ1.2: Even though no big differences were observed among the search-based approaches in the difficulty of detecting the mutants, all of them significantly outperformed Muskit and QMutPy by generating mutants that are detected by fewer test cases. 6.1.3 Results for RQ1.3 – Test suite contribution. In this RQ, we analysed how much the set of mutants generated by each approach contributes to improving TS as defined in Section 5.7. Figure 6 shows the distribution of the median number of test cases needed to be added to TS in order to detect all the generated mutants for each quantum circuit. The results show that GA achieves the highest score, requiring the median number of five additional tests to be included in the test suite, followed by RS, which requires four test cases but shows a wider vertical spread, indicating greater variability. The literature tools seem to perform the worst, as the median indicates that including a single test case in the test suite detects all the mutants generated. These findings are corroborated by the statistical test results shown in Table 6. The table shows that GA performs significantly better than all other approaches except for RS, with which the differences seem not to be significant. Again, all search approaches show significantly better results than Muskit and QMutPy, all of them with large effect sizes. This demonstrates the superiority of the search algorithms in creating mutants that yield more test cases. Concluding remarks for RQ1.3: GA provides the mutants that contribute more to improving the test suite, as they require a median of five new tests to be added to TS to detect them. All search approaches significantly outperform Muskit and QMutPy, showing the need to improve the test suite.

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:24

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Table 6. RQ1.3 – Pairwise statistical comparison of the approaches in terms of number of new tc needed to detect the generated mutants.

Comparison

𝑝-value

Effect size (𝛿)

Comparison

𝑝-value

Effect size (𝛿)

GA vs (1+1)-EA GA vs Muskit (1+1)-EA vs QMutPy RS vs QMutPy RS vs HC HC vs Muskit GA vs RS Muskit vs QMutPy

< 0.05 < 0.05 < 0.05 < 0.05 < 0.05 < 0.05 0.12 0.20

Large (0.98) Large (1.00) Large (0.74) Large (0.65) Large (0.47) Large (0.56) -

GA vs HC GA vs QMutPy (1+1)-EA vs Muskit RS vs Muskit HC vs QMutPy (1+1)-EA vs RS (1+1)-EA vs HC –

< 0.05 < 0.05 < 0.05 < 0.05 < 0.05 < 0.05 0.16 –

Large (0.98) Large (0.99) Large (0.73) Large (0.62) Large (0.61) Medium (-0.42) – –

(a) Number of mutants generated per run.

(b) DetectionRatio for the mutants generated.

(c) Number of new tc needed to detect mutants.

Fig. 7. RQ2.1 – Comparison of first order and second order mutants generated by GA.

6.2

Results for RQ2 – High order mutants

In this RQ, we analyse how different the obtained mutants are when we introduce a single or multiple mutations to the circuit. For this analysis, although all search algorithms generated both first order and second order mutants, we chose to focus only on the mutants produced by GA, as it achieved the best overall performance in RQ1. First, in RQ2.1 (Section 6.2.1), we compare first order and second order mutants by utilising the same metrics used in RQ1. Then, in RQ2.2 (Section 6.2.2), we focus on analysing the characteristics of second order mutants and how the characteristics of mutations affect the generation. Finally, in RQ2.3 (Section 6.2.3), we examine the interactions among mutations introduced into the same circuit and how these interactions affect mutant detection. 6.2.1 Results for RQ2.1 – First order vs second order mutants. Figure 7 shows the results obtained from comparing first order and second order mutants generated by GA. Figure 7a shows the total number of mutants obtained across all circuits per run per order. Figure 7b shows DetectionRatio across the mutants per order. Figure 7c shows the number of new tc needed to add to TS to detect the generated mutants per order. Figure 7a shows that GA is able to achieve a slightly higher number of first order mutants than of second order mutants. Second order mutants have a better ratio of non-equivalence, which results ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:25

Table 7. RQ2.1 – Statistical comparison between first order and second order mutations using the Wilcoxon signed-rank test and Cliff’s delta effect size. The sign of the effect size indicates the direction of the effect, with positive values leaning towards first order mutants and negative values towards second order mutants.

Metric

𝑝-value

Effect size (𝛿)

#Passed TS #Non-equivalent Non-equivalent ratio tc detection % Number of new tc needed

< 0.05 0.07 0.4 < 0.05 < 0.05

Small (0.18) – – Small (-0.22) Large (0.51)

in both orders having a similar number of non-equivalent mutants. In Figure 7b, DetectionRatio results show that, overall, first order mutants achieve a lower detection rate than second order mutants, which achieve a median detection rate that is more than double of that of first order: 5.2% vs 11.5%. Thus, we conclude that first order mutants are easier to find during the search, but they are more likely to be equivalent as they have a lower non-equivalent ratio than second order mutants. The higher non-equivalent ratio of second order mutants is also related to the bigger effect they produce in the circuit as shown in the DetectionRatio in Figure 7b. In Figure 7c, the results show that second order mutants require fewer tc to be detected, with a median of 3 compared to 4 for first order. This indicates that the generated second order mutants contribute less to the improvement of the test suite, as they can be detected using fewer tc, which is also related to having a larger effect on the circuits. Table 7 presents the statistical test results comparing the metrics for first order and second order mutants. The statistical tests confirm the previous findings and show that both orders generate similar numbers of mutants passing TS and being non-equivalent, with a small difference in terms of the number of mutants passing the test suite and no significant difference in terms of the number of non-equivalent mutants or the non-equivalent ratio. Also, in terms of tc detection, there is again a small effect size leaning towards second order mutants, shown by a negative Cliff’s 𝛿, indicating second order mutants have slightly more effect on the circuit and thus they are detected by more tc. Finally, the biggest difference between the two orders comes from the number of new tc needed to detect them, which shows a significant difference with a large effect size in favour of first order mutants. This means that while both orders show similar behaviour, first order mutants are more valuable for the study’s main objective, as they are the ones that contribute the most to the improvement of the test suite. Concluding remarks for RQ2.1: Overall, first order and second order mutants show comparable behaviour. While first order mutants have a slightly worse non-equivalent ratio, the non-equivalent mutants obtained are detected by fewer test cases and contribute more to the improvement of the test suite by needing to add more test cases to detect them. Thus, we conclude that first order mutants align better with our objective of finding non-equivalent mutants that pass TS and contribute to its improvement. 6.2.2 Results for RQ2.2 – Second order mutants characteristics. Following the analysis of second order mutants, we wanted to explore more the characteristics of each mutation and the interactions among them that generate second order mutants. Figure 8 shows the proportions of qubit-position combinations utilised in the generated second order mutants. The labels on the top divide the ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:26

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

Fig. 8. RQ2.2 – Qubits and positions utilised for generating second order mutants.

possible combinations into three categories: (1) Different position in the circuit but same qubit: 𝑃1! = 𝑃2 & 𝑄1 = 𝑄2 (Dark Blue). (2) Same position in the circuit and thus the same qubit: 𝑃1 = 𝑃2 & 𝑄1 = 𝑄2 (Blue). (3) Different position in the circuit and different qubit: 𝑃1! = 𝑃2 & 𝑄1! = 𝑄2 (Light Blue). Each of them is further divided into subsections showing the relative position of the circuit where the mutation is introduced: (1) Beginning of the circuit, referring to the first third of the circuit (Beg). (2) Middle of the circuit referring to the second third of the circuit (Mid). (3) End of the circuit referring to the last third of the circuit (End). Figure 8 shows that the majority of the generated second order mutants are composed of mutations in different positions and qubits (80.1%), suggesting that, even when not close to each other, mutations can interact. Figure 8 also shows that the least frequent combination is to have mutations in different positions but the same qubit, with only 4.5% of second order mutants falling into such category; instead, the mutations in the same position (i.e. one right after the other) are slightly more frequent with 15.3% of the cases. When referring to the relative position in the circuit, the figure shows that the most frequent combinations are mutations at distinct positions, with those combining mutations in the beginning and end of the circuit being the most frequent. Figure 8 suggests that mutations introduced far apart (both in terms of relative position and qubits) lead to more subtle changes, as QUMUG was able to generate more mutants passing TS and being non-equivalent with such characteristics. Figure 9 shows the proportions of combinations of gates used in each mutation in the generated second order mutants. The figure is first divided into two categories, depending on whether the gates 𝐺 used in each mutation are the same type or different, 𝐺1 = 𝐺2 and 𝐺1! = 𝐺2 respectively. Then, each of the main categories is further divided into the possible specific combinations that can be obtained utilising our operators, for instance “𝑅𝑧 - 𝑅𝑧 ” or “𝑈 - 𝑅𝑥 ”. The figure shows that the majority of the generated second order mutants use different gates for each mutation (68.8%), ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:27

Fig. 9. RQ2.2 – Gate combinations utilised for generating second order mutants.

compared to those using the same gate (31.2%). This suggests that mutations utilising gates that modify different bases in the circuit lead to more subtle changes, making them more relevant to our objective of finding mutants that pass TS and are non-equivalent. Regarding specific gates, the 𝑅𝑧 gate seems to be the most effective one, as it contributes to the most successful combinations, and when talking about an individual combination, the double 𝑅𝑧 seems to be the most efficient, as it individually achieves a higher percentage than the rest of the combinations, 15.1%. Concluding remarks for RQ2.2: Mutations applied to different parts of the circuits and affecting different bases represent a larger proportion of the generated second order mutants. Specifically, utilising the 𝑅𝑧 gate when generating second order mutations and applying it at the beginning or end of the circuits yields mutants that align more closely with our objective. 6.2.3 Results for RQ2.3 – Second order mutants test case detection. In this research question, we evaluate how interactions between mutations in second order mutants affect their detectability. We compare the set of unique test cases that detect each first order mutant independently tcd fom , with the set of test cases that detect the corresponding combined second order mutants tcd hom . We group them based on their overlap to analyse their interaction effects: (1) first order mutations are not detected but the second order mutant is detected. (2) There is no overlap between the two detection sets. (3) There is some overlap between the two detection sets. (4) The two detection sets are equal. (5) first order mutations are detected and tcd fom includes tcd hom . (6) first order mutations are detected and tcd fom is included in tcd hom . Figure 10 shows the distribution of the different groups of tcd that we obtained and their proportions among the generated second order mutants. The figure shows that most second order generated mutants share some extent of tcd hom with tcd fom . In only 0.66% of cases tcd hom is ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:28

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

(1) tcd hom ≠ ∅, tcd fom = ∅

(2) tcd hom ∩ tcd fom = ∅

(3) tcd hom ∩ tcd fom ≠ ∅

0.36%

0.30%

26.22%

(4) tcd hom = tcd fom

(5) tcd hom ⊂ tcd fom

(6) tcd fom ⊂ tcd hom

28.75%

26.25%

18.12%

Blue Dashed Circle = higher order detection test set

White Circle = first order detection test set

Fig. 10. RQ2.3 – Distribution of second order mutants by the relationship between tcd fom and tcd hom .

completely independent: in 0.36%, first order mutations are not detected independently (case (1)), but the second order mutant is; and in 0.3%, there is no overlap between the two sets (case (2)). Among the other cases, 28.75% show the exact same set for both tcd hom and tcd fom (case (4)), while the rest show some interaction between the two mutations that either increases or decreases their detection. In case of the 18.12% of mutants where tcd fom is included in tcd hom (case (6)), we can conclude that by combining both independent mutations in the same circuit, the effect of the fault was increased and thus the second order mutant was detected by more tc. On the contrary, in 26.25% of the cases tcd hom is included in tcd fom (case (5)), meaning that by combining both mutations in the same circuit, the effect of the mutation was reduced, reducing the tc detecting them. Lastly, we also have 26.22% of cases where tc detection sets differ but share some tests (case (3)), indicating that combining mutations in the same circuits can cause some tc to stop detecting them while others appear. Concluding remarks for RQ2.3: In 71.25% of second order mutants, the set of test cases that detect the second order mutant is different from the set of test cases that detect each of the first order mutants. This shows that in most cases, when first order mutants are introduced in the same circuit, they interact with each other and change the behaviour of the mutant. The effect of the mutations was either reduced or increased when another mutation was introduced into the circuit, and in some cases, even mutations that were not detected independently were detected together, highlighting the need for new test cases. 7

Discussion and Recommendations

In this section, we present the findings of our study and reflect on their implications for both researchers and practitioners. We also address the threats to the validity of our study, emphasising its internal, external, and construct validity. 7.1

Findings

The findings of this study demonstrate that search-based optimisation is an effective approach for generating challenging quantum circuit mutants that are currently undetected by existing test suites, yet remain not equivalent. Unlike mutation analysis tools in the literature, which often ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:29

create easily detectable mutants, QUMUG focuses the search on areas of the mutation space that highlight weaknesses in the current test suite. The experiments demonstrated the importance of explicitly considering the effect of the mutation during optimisation. Random search, which lacks guidance, produced a higher proportion of equivalent mutants than the guided search algorithms. This supports the inclusion of the second term in the fitness function (see Equation 2), which rewards mutants that remain challenging to detect while producing measurable variations from the original circuit. Without such guidance, the search algorithms naturally tend towards mutations that pass testing because they have little or no impact on the circuit, rather than because they represent genuinely challenging faulty scenarios, resulting in a higher number of equivalent mutants. The experiments also highlight the benefit of the newly introduced parametric mutation operators, as even random search, despite its simplicity, consistently generated more challenging mutants than existing quantum mutation tools. Another important finding is that more challenging mutants alone are insufficient for improving test suites. While more challenging mutants expose weaknesses in existing tests, repeatedly generating similar mutants yields weak results, as they often require the same additional test case to detect them. Our results therefore suggest that diversity should also be considered as an objective for search-based mutant generation. This observation explains the performance of the genetic algorithm, whose exploration maintains greater diversity than the local search approaches. By discovering a broader range of challenging mutants, the genetic algorithm exposes a wider variety of testing deficiencies and, consequently, provides greater opportunities to improve the overall quality of the test suite. Finally, the analysis of higher order mutants shows that interactions between first order mutations play an important role in quantum mutation testing. In the experiments, although the generation of first order and second order mutants exhibits comparable performance, the behaviour of second order mutants cannot always be predicted based on their individual first order mutations. We observed instances in which combining mutations increased, decreased, or altered their detectability compared with evaluating the same mutations independently as first order mutants. This indicates that higher order mutants are particularly valuable for quantum mutation analysis, as they often necessitate new test cases that would not arise from considering mutations independently as first order mutants. 7.2 Implications for Researchers Given the current state of quantum circuit mutation analysis and the limited availability of benchmarks, the introduction of new mutation operators and the representation of undetected faults provide valuable contributions to the research community. Existing quantum mutation approaches are often constrained by predefined mutation operators, potentially leading to an incomplete assessment of test suite effectiveness. By targeting the generation of mutants that remain undetected by existing test suites, QUMUG enables researchers to explore more challenging mutants and evaluate whether testing techniques can detect subtle faults. The generated mutants can serve as complementary evaluation benchmarks for quantum software testing approaches. Researchers developing new testing techniques can utilise these mutants to assess the effectiveness of their methods beyond traditional mutation operators. In particular, the ability to generate mutants specifically tailored to the weaknesses of a given test suite provides opportunities to study the limitations of current quantum testing practices and to identify areas where additional testing strategies are required. Furthermore, the introduction of higher order mutants into quantum circuits creates new research opportunities for understanding possible fault interactions in quantum programs. Unlike first order mutants, where individual changes may only represent isolated faults, higher order mutants allow ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:30

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

researchers to investigate the combined effects of multiple faults and their potential interactions within quantum circuits. These mutants can be used to analyse complex behaviours, like qubit dependencies or error propagation across entangled qubits. Such analyses may provide deeper insights into the relationship between circuit structure, fault characteristics, and test effectiveness. Finally, this work highlights the potential of search-based approaches for exploring the quantum mutation space. Rather than relying solely on manually designed mutation operators, future research can investigate adaptive or learning-based approaches for discovering new quantum faults. This opens opportunities for combining mutation testing with optimisation approaches, automated test generation, and fault modelling to improve the robustness and reliability of quantum software testing methodologies. 7.3

Implications for Practitioners

Mutation analysis can be costly and time-consuming, requiring significant computational resources. This challenge is amplified in quantum software, where existing mutation tools often produce trivial mutants that add little value. QUMUG allows practitioners to focus on generating targeted, challenging mutants that highlight weaknesses in their testing techniques. By creating mutants that evade detection by the current test suite, valuable insights can be gained about gaps in the test suite. This targeted methodology shifts mutation analysis from a broad evaluation of test adequacy to a more focused approach, prioritising efforts on faults that expose limitations in the existing test suite. The incorporation of higher order mutants further extends the practical applicability of mutation analysis by enabling the evaluation of multiple mutations in the same quantum circuit. In real-world quantum software, faults may not always occur independently, and interactions between multiple faulty components may lead to unexpected behaviours. By considering combinations of mutations, practitioners can evaluate whether their test suites can detect more complex faulty scenarios and improve their coverage of diverse faults. Furthermore, as QUMUG iteratively generates mutants based on the current test suite, it can be integrated into existing quantum software testing pipelines. For example, during continuous integration and testing processes, QUMUG could periodically search for new mutants and use them as indicators of potential weaknesses in the testing technique used. When no additional non-equivalent mutants can be discovered, this provides practitioners with greater confidence that the test suite has been evaluated against a broader and more challenging set of faulty scenarios. Overall, QUMUG provides practitioners with a mechanism to complement existing mutation testing practices by prioritising mutant quality over quantity. Rather than generating potentially redundant mutants, practitioners can focus on a set of higher quality mutants that provide deeper insights into test suite effectiveness. 7.4

Threats to Validity

In this section, we will discuss the key factors that may affect the validity of our conclusions and outline the measures taken to minimise their impact. We will address concerns related to the validity of our measurements and evaluation criteria, the construct validity, the experimental methodology, internal validity, and the generalisability of our findings beyond the evaluated settings, external validity. 7.4.1 Construct Validity. Construct validity assesses whether the metrics and evaluation methods used accurately measure the concepts examined in this study. In our experiments, we assess mutant detection using the chi-square test with a fixed significance threshold, comparing the output distributions of the original and mutated circuits, as in several earlier studies [29, 30, 47, 51]. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:31

While this offers an objective and reproducible criterion for identifying mutant detection, QUMUG offers the opportunity to utilise different statistical tests or significance thresholds, which might categorise some mutants differently. Since we developed the fitness function in a generic manner, we also needed to choose a specific distance metric to quantify the effect of the mutations in our experiments: the Hellinger distance. Although the Hellinger distance is well suited for comparing probability distributions [14, 32, 33, 38], other distance metrics, such as Jensen-Shannon divergence, could alter the optimisation process and potentially lead to different search behaviour. Another threat concerns the identification of equivalent mutants. Since proving semantic equivalence between two quantum circuits is generally infeasible, QUMUG assumes that mutants that cannot be distinguished by our comprehensive test suite are equivalent for the purposes of this study. Consequently, some mutants that pass the test suite may still be behaviourally different under inputs or observations not exercised by the test suite. To reduce this threat, we employ an extensive input-output testing strategy using both classical and quantum input states, with measurements performed in the three bases (𝑋 , 𝑌 , and 𝑍 ). Another aspect related to construct validity concerns the representation of circuit behaviour used for mutant detection. In this study, we utilise output probability distributions as the primary oracle, as they provide an observable representation of quantum circuit execution and are directly applicable when evaluating measurement outcomes. However, output distributions are not the only possible approach for characterising quantum circuit behaviour. Alternative representations, such as quantum state vectors or expectation values, may capture different aspects of circuit execution and could potentially identify behavioural differences that are not reflected in the measured output distributions. Consequently, mutants classified as equivalent under our oracle definition may exhibit differences when evaluated using alternative behavioural representations. Nevertheless, output distributions represent one of the most practical and widely applicable approaches for quantum software testing [22, 30, 32, 33, 38, 48], particularly when considering execution on current quantum hardware. Another potential threat concerns the realism of the generated mutants. Although the proposed mutation operators are designed to introduce challenging and behaviourally meaningful faults, they are not derived from existing faults. Therefore, the generated mutants should be interpreted as faultrepresentative testing artefacts rather than exact replications of real-world quantum programming faults. Nevertheless, the objective of mutation analysis is not necessarily to reproduce every possible real fault, but rather to provide challenging and informative artefacts for evaluating and improving the effectiveness of testing approaches. 7.4.2 Internal Validity. Internal validity concerns whether the observed results are influenced by aspects of the experimental methodology rather than the proposed approach itself. Since all evaluated search approaches are stochastic, their performance may vary between executions due to different random initialisations. To reduce the influence of randomness, every experiment was repeated ten times. We chose ten repetitions to balance obtaining representative results while managing computational costs, given the number of circuits, test cases, and evaluations required. The performance of search-based optimisation approaches is also influenced by their parameter settings, including the population size, mutation probability, and crossover probability. The parameters adopted in this work were selected through preliminary experimentation to provide stable search behaviour across all benchmark circuits. Nevertheless, different parameter configurations may influence convergence speed or the characteristics of the generated mutants. To assess whether the effectiveness of the proposed mutation framework depends on a particular optimisation strategy, we evaluated four search algorithms representing different search paradigms, including local search and population-based approaches. The algorithms’ consistent ability to ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:32

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

generate challenging mutants enhances our confidence that the improvements observed are due to the proposed mutation framework, rather than being a result of a single search algorithm.

7.4.3 External Validity. External validity concerns the extent to which the findings generalise beyond the experimental setting. Our evaluation was conducted on six quantum programs comprising a total of thirty quantum circuits spanning different categories and circuit sizes. Although these benchmarks represent a diverse collection of quantum algorithms commonly adopted in the literature, they do not cover the full spectrum of emerging quantum applications. The experiments were performed using quantum circuit simulation and circuits containing up to seven qubits. This limitation was primarily imposed by the computational cost of repeatedly executing thousands of test cases across multiple generations and independent search runs. Consequently, the reported results may differ for substantially larger quantum circuits or realistic environments. We acknowledge this as a possible limitation of the study; however, given the current state of quantum computers, we believe that the inherent noise in these devices makes it infeasible to evaluate our approaches, as the effect of the mutations can be hidden by noise or even aggravated [14].

8

Conclusions and Future Work

In this paper, we presented QUMUG, a search-based approach for generating quantum circuit mutants that are not detected by a given test suite yet not equivalent. By introducing novel parametric mutation operators and utilising search-based algorithms to explore the resulting mutation space, QUMUG systematically generates mutants that reveal previously undetected faults and help improve the test suite. Our evaluation of 30 quantum circuits spanning six quantum algorithms demonstrated that QUMUG consistently generates more challenging mutants than existing quantum circuit mutation tools. Among the evaluated search algorithms, the genetic algorithm achieved the best overall performance by producing a higher proportion of non-equivalent mutants that required new test cases for detection. Furthermore, we introduced higher order mutation testing for quantum circuits for the first time, and showed that interactions between multiple mutations produce behaviours that cannot always be inferred from their constituent first order mutants, highlighting the potential of higher order mutation analysis for understanding fault interactions in quantum software. Future work will explore several directions to further advance search-based quantum mutation testing. First, we plan to investigate multi-objective search approaches that optimise different mutation objectives simultaneously, rather than combining them into a single fitness function. For example, the objectives of generating undetected mutants and maximising behavioural impact could be optimised independently, allowing the search to identify a broader range of solutions. Mutant diversity could also be incorporated as a separate optimisation objective to encourage the generation of distinct mutants that expose different weaknesses in a test suite. Another promising direction is the design of additional mutation operators capable of representing a wider variety of quantum faults. In particular, we intend to investigate parameterisable mutations involving multi-qubit gates. Finally, we would also like to investigate alternative mechanisms for mutant detection, including different statistical tests and similarity measures, together with other quantum software testing techniques and oracle definitions, to assess how the effectiveness of search-based mutant generation varies under different testing settings. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:33

Acknowledgments E. Mendiluze Usandizaga is supported by Simula’s internal strategic project on quantum software engineering. S. Ali is supported by the Norwegian Quantum Software Center (Project #361350) funded by the Research Council of Norway and Oslo Metropolitan University’s Quantum Hub. P. Arcaini is supported by the ASPIRE grant No. JPMJAP2301, JST. The research presented in this paper has benefited from the Experimental Infrastructure for Exploration of Exascale Computing (eX3), which is financially supported by the Research Council of Norway under contract 270053. This work was supported, in part, by Research Ireland grants 20/FFP-P/8818, and 13/RC/2094_P2 and co-funded under the European Regional Development Fund through the Southern & Eastern Regional Operational Programme to Lero - the Research Ireland Research Centre for Software (www.lero.ie). AI-based tools, including ChatGPT, Gemini, and Grammarly, were used to assist with refining and polishing the manuscript text, including grammar and language checks. These tools were also used to support the visual presentation and refinement of plots, figures, and tables. The authors reviewed and validated all AI-assisted content and remain fully responsible for the final content of the publication. References [1] Rui Abreu, Shaukat Ali, Paolo Arcaini, José Campos, Michael Felderer, Claude Gravel, Fuyuki Ishikawa, Stefan Klikovits, Andriy Miranskyy, Anila Mjeda, Mohammad Reza Mousavi, Masaomi Yamaguchi, Lei Zhang, and Jianjun Zhao. 2026. Software Testing in the Quantum World. Computer 59, 4 (2026), 135–138. doi:10.1109/MC.2026.3655854 [2] Rui Abreu, João Paulo Fernandes, Luis Llana, and Guilherme Tavares. 2022. Metamorphic Testing of Oracle Quantum Programs. In 2022 IEEE/ACM 3rd International Workshop on Quantum Software Engineering (Q-SE). 16–23. doi:10.1145/ 3528230.3529189 [3] Shaukat Ali, Paolo Arcaini, Xinyi Wang, and Tao Yue. 2021. Assessing the Effectiveness of Input and Output Coverage Criteria for Testing Quantum Programs. In 2021 IEEE 14th International Conference on Software Testing, Validation and Verification (ICST). 13–23. doi:10.1109/ICST49551.2021.00014 [4] Paul Ammann, Marcio Eduardo Delamaro, and Jeff Offutt. 2014. Establishing Theoretical Minimal Sets of Mutants. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 21–30. doi:10.1109/ICST. 2014.13 [5] Thomas Back. 1996. Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford university press. [6] José Campos and André Souto. 2021. QBugs: A Collection of Reproducible Bugs in Quantum Algorithms and a Supporting Infrastructure to Enable Controlled Quantum Software Testing and Debugging Experiments. In 2021 IEEE/ACM 2nd International Workshop on Quantum Software Engineering (Q-SE). IEEE Computer Society, Los Alamitos, CA, USA, 28–32. doi:10.1109/Q-SE52541.2021.00013 [7] Norman Cliff. 1993. Dominance statistics: Ordinal analyses to answer ordinal questions. Psychological bulletin 114, 3 (1993), 494. doi:10.1037/0033-2909.114.3.494 [8] Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and José A. Cruz-Lemus. 2022. Quantum Software Testing: Current Trends and Emerging Proposals. Springer International Publishing, Cham, 167–191. doi:10.1007/978-3-031-05324-5_9 [9] Antonio García de la Barrera, Ignacio García-Rodríguez de Guzmán, Macario Polo, and Mario Piattini. 2023. Quantum software testing: State of the art. Journal of Software: Evolution and Process 35, 4 (2023), e2419. doi:10.1002/smr.2419 [10] Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 7 (Dec. 2006), 1–30. [11] Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs: A Case Study With Qiskit. IEEE Transactions on Quantum Engineering 3 (2022), 1–17. doi:10.1109/TQE.2022.3195061 [12] Daniel Fortunato, José Campos, and Rui Abreu. 2022. Mutation Testing of Quantum Programs Written in QISKit. In Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings (Pittsburgh, Pennsylvania) (ICSE ’22). Association for Computing Machinery, New York, NY, USA, 358–359. doi:10.1145/3510454. 3528649 [13] Daniel Fortunato, José Campos, and Rui Abreu. 2022. QMutPy: A Mutation Testing Tool for Quantum Algorithms and Applications in Qiskit. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 797–800. ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:34

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

doi:10.1145/3533767.3543296 [14] Sophie Fortz, Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Mohammad Reza Mousavi. 2026. Robust Mutation Analysis of Quantum Programs Under Noise. arXiv:2605.13279 [cs.SE] https://arxiv.org/abs/2605.13279 [15] Stanton A. Glantz. 2012. Primer of Biostatistics (7th edition ed.). McGraw Hill, New York. https://www.accessscience. com/content/book/9780071781503 [16] Daniel M. Greenberger, Michael A. Horne, and Anton Zeilinger. 1989. Going Beyond Bell’s Theorem. In Bell’s Theorem, Quantum Theory and Conceptions of the Universe, Menas Kafatos (Ed.). Springer Netherlands, Dordrecht, 69–72. doi:10.1007/978-94-017-0849-4_10 [17] Shahin Honarvar, Mohammad Reza Mousavi, and Rajagopal Nagarajan. 2020. Property-Based Testing of Quantum Programs in Q#. In Proceedings of the IEEE/ACM 42nd International Conference on Software Engineering Workshops (Seoul, Republic of Korea) (ICSEW’20). Association for Computing Machinery, New York, NY, USA, 430–435. doi:10. 1145/3387940.3391459 [18] Yuta Ishimoto, Masanari Kondo, Naoyasu Ubayashi, Yasutaka Kamei, Ryota Katsube, Naoto Sato, and Hideto Ogawa. 2025. Evaluating Mutation-based Fault Localization for Quantum Programs. In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering (EASE ’25). Association for Computing Machinery, New York, NY, USA, 666–671. doi:10.1145/3756681.3757022 [19] Yue Jia and Mark Harman. 2011. An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering 37, 5 (2011), 649–678. doi:10.1109/TSE.2010.62 [20] Ryoichi Kawai. 2022. A First Step to Quantum Computation with Qiskit. https://kawaihome.link/jbooks/qcompshort/cover.html [21] Marinos Kintis, Mike Papadakis, Andreas Papadopoulos, Evangelos Valvis, and Nicos Malevris. 2016. Analysing and Comparing the Effectiveness of Mutation Testing Tools: A Manual Study. In 2016 IEEE 16th International Working Conference on Source Code Analysis and Manipulation (SCAM). 147–156. doi:10.1109/SCAM.2016.28 [22] Jonas Klamroth, Max Scheerer, and Oliver Denninger. 2025. Detecting and Tolerating Faults in Hybrid Quantum Software Systems Using Architectural Redundancy. In IEEE International Conference on Quantum Software, QSW 2025, Helsinki, Finland, July 7-12, 2025, Rong N. Chang, Carl K. Chang, Jingwei Yang, Nimanthi Atukorala, Dan Chen, Sumi Helal, Sasu Tarkoma, Qiang He, Tevfik Kosar, Claudio A. Ardagna, Sebastian Feld, Elisabetta Di Nitto, and Manuel Wimmer (Eds.). IEEE, Helsinki, Finland, 162–172. doi:10.1109/QSW67625.2025.00028 [23] Michael Kölle, Tom Bintener, Maximilian Zorn, Gerhard Stenzel, Leo Sünkel, Thomas Gabor, and Claudia LinnhoffPopien. 2025. Evaluating Mutation Techniques in Genetic-Algorithm-Based Quantum Circuit Synthesis. In Proceedings of the Genetic and Evolutionary Computation Conference (NH Malaga Hotel, Malaga, Spain) (GECCO ’25). Association for Computing Machinery, New York, NY, USA, 907–915. doi:10.1145/3712256.3726402 [24] Thomas Laurent, Mike Papadakis, Marinos Kintis, Christopher Henard, Yves Le Traon, and Anthony Ventresque. 2017. Assessing and Improving the Mutation Testing Practice of PIT. In 2017 IEEE International Conference on Software Testing, Verification and Validation (ICST). 430–435. doi:10.1109/ICST.2017.47 [25] Neilson Carlos Leite Ramalho, Higor Amario de Souza, and Marcos Lordello Chaim. 2025. Testing and Debugging Quantum Programs: The Road to 2030. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 155 (May 2025), 46 pages. doi:10.1145/3715106 [26] Gushu Li, Li Zhou, Nengkun Yu, Yufei Ding, Mingsheng Ying, and Yuan Xie. 2020. Projection-based runtime assertions for testing and debugging Quantum programs. Proc. ACM Program. Lang. 4, OOPSLA, Article 150 (nov 2020), 29 pages. doi:10.1145/3428218 [27] Peixun Long and Jianjun Zhao. 2024. Testing Multi-Subroutine Quantum Programs: From Unit Testing to Integration Testing. ACM Trans. Softw. Eng. Methodol. 33, 6, Article 147 (June 2024), 61 pages. doi:10.1145/3656339 [28] Kane Meissel and Esther S Yao. 2024. Using Cliff’s delta as a non-parametric effect size measure: an accessible web app and R tutorial. Practical Assessment, Research, and Evaluation 29, 1 (2024). doi:10.7275/pare.1977 [29] Eñaut Mendiluze Usandizaga, Shaukat Ali, Paolo Arcaini, and Tao Yue. 2022. Muskit: A Mutation Analysis Tool for Quantum Software Testing. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (Melbourne, Australia) (ASE ’21). IEEE Press, 1266–1270. doi:10.1109/ASE51524.2021.9678563 [30] Eñaut Mendiluze Usandizaga, Shaukat Ali, Tao Yue, and Paolo Arcaini. 2025. Quantum circuit mutants: Empirical analysis and recommendations. Empirical Software Engineering 30, 4 (16 Apr 2025), 100. doi:10.1007/s10664-025-10643-z [31] Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali. 2026. Supplementary material for the paper “Search-Based Generation of Undetected Quantum Circuit Mutants”. https://doi.org/10.5281/zenodo.21718539 [32] Asmar Muqeet, Shaukat Ali, and Paolo Arcaini. 2024. Quantum Program Testing Through Commuting Pauli Strings on IBM’s Quantum Computers. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering (Sacramento, CA, USA) (ASE ’24). Association for Computing Machinery, New York, NY, USA, 2130–2141. doi:10.1145/3691620.3695275

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Search-Based Generation of Undetected Quantum Circuit Mutants

X:35

[33] Asmar Muqeet, Tao Yue, Shaukat Ali, and Paolo Arcaini. 2024. Mitigating Noise in Quantum Software Testing Using Machine Learning. IEEE Transactions on Software Engineering 50, 11 (2024), 2947–2961. doi:10.1109/TSE.2024.3462974 [34] Juan Manuel Murillo, Jose Garcia-Alonso, Enrique Moguel, Johanna Barzen, Frank Leymann, Shaukat Ali, Tao Yue, Paolo Arcaini, Ricardo Pérez-Castillo, Ignacio García-Rodríguez de Guzmán, Mario Piattini, Antonio Ruiz-Cortés, Antonio Brogi, Jianjun Zhao, Andriy Miranskyy, and Manuel Wimmer. 2025. Quantum Software Engineering: Roadmap and Challenges Ahead. ACM Trans. Softw. Eng. Methodol. 34, 5, Article 154 (May 2025), 48 pages. doi:10.1145/3712002 [35] Michael A. Nielsen and Isaac L. Chuang. 2016. Quantum Computation and Quantum Information (10th Anniversary edition). Cambridge University Press, Cambridge, UK. [36] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275–378. doi:10.1016/bs.adcom. 2018.03.015 [37] Mike Papadakis and Nicos Malevris. 2010. An Empirical Evaluation of the First and Second Order Mutation Testing Strategies. In 2010 Third International Conference on Software Testing, Verification, and Validation Workshops. 90–99. doi:10.1109/ICSTW.2010.50 [38] Gabriel Pontolillo, Asmar Muqeet, Shaukat Ali, and Mohammad Reza Mousavi. 2025. From Ideal to Noisy: Adapting Property-Based Testing for Real-World Noisy Quantum Computers. In 2025 IEEE International Conference on Quantum Computing and Engineering (QCE), Vol. 01. 405–416. doi:10.1109/QCE65121.2025.00053 [39] Nils Quetschlich, Lukas Burgholzer, and Robert Wille. 2023. MQT Bench: Benchmarking Software and Design Automation Tools for Quantum Computing. Quantum 7 (2023), 1062. arXiv:2204.13719 doi:10.22331/q-2023-07-20-1062 MQT Bench is available at https://mqt-bench.app/. [40] Troels F Rønnow, Zhihui Wang, Joshua Job, Sergio Boixo, Sergei V Isakov, David Wecker, John M Martinis, Daniel A Lidar, and Matthias Troyer. 2014. Defining and detecting quantum speedup. Science 345, 6195 (2014), 420–424. doi:10.1126/science.1252319 [41] David Schuler and Andreas Zeller. 2013. Covering and uncovering equivalent mutants. Software Testing, Verification and Reliability 23, 5 (2013), 353–374. [42] Birgit Schwarz, David Schuler, and Andreas Zeller. 2011. Breeding High-Impact Mutations. In 2011 IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops. 382–387. doi:10.1109/ICSTW.2011.56 [43] Rodolfo Adamshuk Silva, Simone do Rocio Senger de Souza, and Paulo Sérgio Lopes de Souza. 2017. A systematic review on search based mutation testing. Information and Software Technology 81 (2017), 19–35. doi:10.1016/j.infsof.2016.01.017 [44] Ana B. Sánchez, José A. Parejo, Sergio Segura, Amador Durán, and Mike Papadakis. 2024. Mutation Testing in Practice: Insights From Open-Source Software Developers. IEEE Transactions on Software Engineering 50, 5 (2024), 1130–1143. doi:10.1109/TSE.2024.3377378 [45] Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali. 2026. Quantum Circuit Repair by Gate Prioritisation. In 2026 IEEE International Conference on Software Testing, Verification and Validation (ICST). 744–748. doi:10.1109/ICST69053.2026.00095 [46] Willem Visser. 2016. What makes killing a mutant hard. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering (Singapore, Singapore) (ASE ’16). Association for Computing Machinery, New York, NY, USA, 39–44. doi:10.1145/2970276.2970345 [47] Jiyuan Wang, Fucheng Ma, and Yu Jiang. 2021. Poster: Fuzz Testing of Quantum Program. In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST). 466–469. doi:10.1109/ICST49551.2021.00061 [48] Xinyi Wang, Shaukat Ali, and Davide Taibi. 2025. The Landscape of Quantum Software Testing Tools. IEEE Software 42, 5 (2025), 136–140. doi:10.1109/MS.2025.3578154 [49] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Application of Combinatorial Testing to Quantum Programs. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security (QRS). 179–188. doi:10.1109/QRS54544.2021.00029 [50] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2021. Generating Failing Test Suites for Quantum Programs With Search. In Search-Based Software Engineering, Una-May O’Reilly and Xavier Devroey (Eds.). Springer International Publishing, Cham, 9–25. doi:10.1007/978-3-030-88106-1_2 [51] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Quito: A Coverage-Guided Test Generator for Quantum Programs. In Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering (Melbourne, Australia) (ASE ’21). IEEE Press, 1237–1241. doi:10.1109/ASE51524.2021.9678798 [52] Xinyi Wang, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2024. QuCAT: A Combinatorial Testing Tool for Quantum Software. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (Echternach, Luxembourg) (ASE ’23). IEEE Press, 2066–2069. doi:10.1109/ASE56229.2023.00062 [53] Xinyi Wang, Tongxuan Yu, Paolo Arcaini, Tao Yue, and Shaukat Ali. 2022. Mutation-Based Test Generation for Quantum Programs with Multi-Objective Search. In Proceedings of the Genetic and Evolutionary Computation Conference (Boston, Massachusetts) (GECCO ’22). Association for Computing Machinery, New York, NY, USA, 1345–1353. doi:10.1145/

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

X:36

Eñaut Mendiluze Usandizaga, Thomas Laurent, Paolo Arcaini, and Shaukat Ali

3512290.3528869 [54] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. http: //www.jstor.org/stable/3001968 [55] M.R. Woodward. 1993. Mutation testing—its origin and evolution. Information and Software Technology 35, 3 (1993), 163–169. doi:10.1016/0950-5849(93)90053-6 [56] Noson S Yanofsky and Mirco A Mannucci. 2008. Quantum computing for computer scientists. Cambridge University Press. [57] Jiaming Ye, Shangzhou Xia, Fuyuan Zhang, Paolo Arcaini, Lei Ma, Jianjun Zhao, and Fuyuki Ishikawa. 2024. QuraTest: Integrating Quantum Specific Features in Quantum Program Testing. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (Echternach, Luxembourg) (ASE ’23). IEEE Press, 1149–1161. doi:10. 1109/ASE56229.2023.00196 [58] Jianjun Zhao. 2020. Quantum Software Engineering: Landscapes and Horizons. CoRR abs/2007.07047 (2020). arXiv:2007.07047 https://arxiv.org/abs/2007.07047 [59] Pengzhan Zhao, Jianjun Zhao, Zhongtao Miao, and Shuhan Lan. 2021. Bugs4Q: A Benchmark of Real Bugs for Quantum Programs. In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). 1373– 1376. doi:10.1109/ASE51524.2021.9678908

ACM Trans. Softw. Eng. Methodol., Vol. X, No. X, Article X. Publication date: August 2026.

Related documents

• Robust Mutation Analysis of Quantum Programs Under Noise#180722
• Completing or Refusing Low-Dimensional Records of Structured Quantum Circuits: Measurement Loss, Compression Loss, and Hardware Drift#1070920
• Randomized and Diverse Input State Generation for Quantum Program Testing#155357
• Mapping Dynamic, Hierarchical Quantum Circuits#985467
• Automated Quantum Software and AI Engineering#124121