Flaky Test Recognition when Testing CPSs Using Hybrid Models

1

Flaky Test Recognition when Testing CPSs Using Hybrid Models

arXiv:2608.06535v1 [cs.SE] 6 Aug 2026

Zahra Sadri-Moshkenani, Justin Bradley, Gregg Rothermel

Abstract—Cyber-Physical Systems (CPSs) have many applications, ranging from simple thermostat systems to autonomous driving systems and medical devices. Like all systems, they need to function correctly. To this end, validation techniques such as testing that can effectively reveal faults are required. Also, CPSs usually operate in uncertain environments where various expected/unexpected events can affect their behaviors. These events together with timing and synchronization incidents may result in various/different CPS behaviors and may cause a CPS to pass a test under some conditions and fail it under others. Such test cases are called “flaky” test cases and we call the conditions that are responsible for them “flaky conditions” and call these behaviors “flaky behaviors”. When test cases are flaky, testing results are unreliable. To achieve more reliable test results, engineers may attempt to recognize flaky test cases and remove them. In this work, beginning with a test case generation and execution technique called H Y T EST that we created previously, we integrate TR E VA, a new technique that validates testing results provided by H Y T EST and F LA R E, a new technique that recognizes flaky test cases and flaky conditions using hybrid models during the early stages of CPS development in just one additional round of testing. We present the results of an empirical study evaluating the effectiveness of our new approach (which we call H Y T EST TF). Our results show that correctly differentiate flaky test cases from non-flaky test cases Index Terms—Cyber-Physical Systems, Embedded-Control Systems, Test Case Generation, Hybrid Models, Test Oracles, Flaky Tests, Model-Based Testing.

I. I NTRODUCTION Cyber-Physical Systems (CPSs) integrate software and hardware, communicating through sensors and actuators, typically in a feedback loop [1]. They adapt their behavior based on environmental feedback they receive in order to achieve their goals. They are often utilized in safety-critical applications such smart homes, medical devices, and autonomous driving systems; hence, they must meet specific functional and nonfunctional requirements and operate correctly. Engineers utilize processes such as reviews, testing, and formal methods to ensure that CPSs comply with their requirements [2]. This article focuses on testing methodologies, especially test case generation and execution for CPSs’ control models and software components. It is important to note that in this context, “testing” differs from “verification”. Verification offers a formal proof of specific aspects of a system’s operations across a potentially infinite set of parameters and inputs [3], whereas testing assesses whether a system has been developed correctly Zahra Sadri-Moshkenani, Justin Bradley, and Gregg Rothermel are with the Department of Computer Science, North Carolina State University. Email:{zsadrimo}@gmail.com and {zsadrim,gerother}@ncsu.edu.

with respect to its requirements and addresses end-user needs appropriately [4].1 Where testing of CPSs is concerned, several things require particular attention. First, due to CPSs frequent involvement in safety-critical operations, postponing testing until a CPS is operational in a real-world setting poses risks to both equipment and individuals. Consequently, conducting tests at various stages throughout the development process is imperative. Safety-critical CPSs are typically intricately designed, incorporating a diverse array of model abstractions used during development, which can be examined at different stages. For instance, when evaluating continuous controllers in automotive settings, a tiered testing approach may be employed, including Model in the Loop (MiL), Software in the Loop (SiL), and Hardware in the Loop (HiL) strategies [5]. These levels are explained in section II-F. This tiered testing approach ensures frequent testing at critical implementation stages, mitigating the risk of faults slipping through to later deployment phases. Additionally, CPSs commonly exhibit a blend of continuous and discrete behavior and dynamics. These hybrid systems are frequently represented as hybrid automata [6]. Generating test cases that assess both the continuous and discrete behaviors of these systems is essential, regardless of whether engineers are testing simulated or fully implemented CPSs. Furthermore, Cyber-Physical Systems (CPSs) typically function as reactive systems, aiming to sustain continuous interaction with the environment amidst various inputs, including time. Consequently, testing must accommodate the potential non-termination of the system [7]. The standard correctness requirement for such systems stipulates that all executions must be permissible based on system requirements and specifications [8]. For instance, consider a scenario in which a robot’s objective is to reach a goal target via any available path, rather than by following a specific trajectory. In such cases, traditional test oracles2 may prove ineffective at determining trajectory correctness, given the infinite possible trajectories. Instead, the oracle should focus on expected behavior, emphasizing reaching the target while avoiding obstacles, regardless of the specific path. The reactive nature of CPSs has implications for the suitability of test oracles, as traditional ones may be insufficient in this context. Finally, CPSs usually operate in uncertain and nondeterministic environments where there are many predictable/expected 1 Sections II-A and II-B discuss testing and verification further. 2 A test oracle is a method or device by which engineers can determine whether or not a test case results in program behavior that conforms to specified behavior (and thus can also indicate whether faulty output or behavior have occurred). For additional details see Section II-B.

2

and unpredictable/unexpected events. For example, to an autonomous vehicle, a sudden full stop of a car in front can be expected but likely unpredictable, because it may be interpreted as a slow-down rather than a sudden and full stop. On the other hand, a car driving backwards down a highway is unexpected but possibly predictable, because the backup lights are likely to be on and the distance between the autonomous vehicle and the oncoming car is decreasing. On the other hand, CPSs deal with the timing and synchronizing of processes and devices. Like other systems, a CPS may undergo different process timing and synchronization even over the same test scenario, and these may change the results of executing and testing the CPS. Also, some hardware components may function differently under different conditions, such as changes in external temperature. A CPS that is under test in these various, unpredictable, and changing environments may operate differently and continue to operate correctly or fail when it is tested more than once using the same test case. Such a test case is called a flaky test case. A “flaky test case” is a test case that displays a seemingly random outcome – pass or fail – when executed at different times on the same version of the code [9]–[11]. A test case can be flaky when the conditions under which the system is running change unexpectedly, causing the system to act differently. In this work we use the term “flaky conditions” to refer to the different conditions that cause the system to act differently and pass or fail on the same test case, and use the term “flaky behaviors” to refer to the different behaviors the system may exhibit once it executes under flaky conditions. Flaky test cases can have negative effects on the testing process. Because flaky test cases randomly fail or pass, reports of failures using these test cases can be inaccurate and unreliable and fault localization can be harder. Also, flaky test cases can delay the release time of the final product. The most widely-used method for recognizing flaky test cases is “Rerun”, which involves rerunning each failing test case multiple times to see whether it passes or not. If the test case passes at least once, then the test case is definitely flaky, but if no pass is observed the status of the test case is unknown. There are several disadvantages to using Rerun to identify flaky test cases in a system [9]. The most important is that flaky test cases are nondeterministic, so there is no guarantee that rerunning a flaky test case will change its outcome and therefore reveal its flakiness. Also, the efficiency of Rerun grows worse with the number of failing test cases, and in the context of CPSs with potentially expensive CPS execution/simulation time, deciding on the number of test case reruns is a challenge. Moreover, re-executing every failed test case is extremely costly when there are large numbers of them or when the system is being tested at the HiL level where any failure may damage the hardware and equipment or endanger human safety. Finally, rerun re-executes test cases that have failed ignores the possibility of test cases that have passed might be flaky – and they could be. Problem: To address the first three challenges for testing CPSs that were described above, we have proposed a test case generation and execution technique called H Y T EST [12].

H Y T EST3 allows pre-MiL level test case generation, such that the generated test cases can then be used to perform system testing of the simulation models of the CPS at the MiL and SiL levels or of the final product at the HiL level.4 As H Y T EST generates test cases, it also employs an algorithm that reduces the incidence of redundant test cases, based on the hybrid model, rendering testing more efficient. Finally, H Y T EST provides a test oracle that uses the generated test cases to test the CPS as a reactive system. In other words, H Y T EST’s test oracle checks whether the system operates correctly and in an acceptable manner, i.e., over an acceptable order of CPS states and transitions, to meet the goal or not. To address the last challenge, in this paper, we target the problem of validating CPS test results, predicting potential flaky test cases, and recognizing real flaky test cases and their corresponding flaky conditions without rerunning the test cases an indeterminate, and theoretically infinite, number of times. Prior Work: Several techniques for recognizing flaky test cases in software systems have been proposed [9], [13]; however, these focus on assessing the possible flakiness of failed test cases and trust passed tests, even though those passes may result from flaky conditions. Also, these approaches have not been applied to CPSs. There are papers, e.g. [14] that discuss the root causes of flaky behaviors in CPSs and provide some solutions for maintaining the CPSs after flaky test cases are revealed, but they do not discuss techniques for recognizing those test cases. To the best of our knowledge, there is no work on recognizing flaky test cases for CPSs without using Rerun. Our Technique: As noted earlier, CPSs are usually hybrid systems [6], and can be mathematically modeled by hybrid models [15]. Testing based on hybrid models targets both continuous and discrete CPS behaviors [6], [16]. Further, as also noted earlier, because of the complicated interactions between the cyber and physical environments of a CPS and the uncertain environment in which a CPS operates, it is difficult to predict all the incidents (unexpected events) a CPS will encounter during testing [17]. These incidents affect the CPS’s states, which are all captured in the hybrid model as invariants and guard conditions (this is also discussed further in Section II-D.) In other words, the conditions under which a CPS operates at each moment of its execution can be recognized with the help of its hybrid model; therefore, they can be used to recognize the flaky conditions that result in flaky behaviors, and consequently to recognize the potential flaky test cases. In this article, we integrate a technique for validating test results, which we call TR E VA, and a technique for handling flaky test cases, which we call F LA R E, into our H Y T EST technique [12]. We call the new technique H Y T EST TF; the technique allows us to validate the test results that H Y T EST provides, to recognize real passes and real failures5 , and to determine flaky test cases and their corresponding flaky conditions in just one additional round of testing. 3 See Section III for more details. 4 In this work, we focus primarily on the MiL level. 5 Real passes and real failures are those exhibited by test cases that are not affected by flakiness.

3

While testing the CPS, H Y T EST TF first assesses whether a test case may fail and pass at time j under different conditions and if it determines that it can, marks the test case as a potential flaky test case, and issues a failed test verdict for it. H Y T EST TF records the values that may lead the CPS to pass and fail at time j. Next, it re-tests the CPS using the potential flaky test case, while leading the CPS to pass the test, by setting the values of the CPS’s variable at time j to those that cause the CPS to pass the test. If the CPS passes the test, H Y T EST TF recognizes the test case as a real flaky test case; otherwise, it marks the observed failure as a real failure. To validate the test results for those test cases that are not marked potentially flaky, H Y T EST TF retests the CPS using those test cases. This step allows the algorithm to validate whether these failures and passes are the result of flaky conditions in the CPS or not. After the test results have been validated and any faults discovered have been localized and corrected in the CPS, H Y T EST TF checks whether the potential flaky test cases are real flaky test cases – i.e., whether they fail under some conditions and pass under different conditions. For each potential flaky test case, H Y T EST TF leads the CPS to pass once, and leads the CPS to fail once. If the CPS passes the test when a pass is expected and fails the test when a failure is expected, H Y T EST TF has revealed a real flaky test case. Otherwise, there is a real failure that needs to be corrected. Finally, H Y T EST TF recognizes the real flaky test cases that are not revealed in the previous step by recognizing potential flaky conditions with the help of the hybrid model and testing the CPS under those conditions. Just as in the previous step, if the CPS passes the test when a pass is expected and fails the test when a failure is expected, the test case is a real flaky test case, otherwise, there is a real failure. Finally, for those test cases that are recognized as real flaky test cases, H Y T EST TF provides their corresponding flaky conditions. This work provides the following contributions: A new set of terminology about, and definitions of flakiness to provide a better understanding of what flakiness is in the context of CPSs. • A novel approach for validating test results provided by our previous technique, H Y T EST, for CPSs at the MiL level. • A novel approach that recognizes real flaky test cases in just one more round of testing CPSs by checking behaviors that the CPS under test displays, and the states and transitions covered in the hybrid model, at the MiL level. • A novel approach for recognizing the “flaky conditions” under which a CPS may fail or pass a test, with the help of hybrid models at the MiL level. • An empirical study examining the application of H Y T EST TF applied to simulation models developed in Simulink [18] at the MiL level, to measure its effectiveness in recognizing real flaky test cases. Our results show that overall the approach was able to correctly differentiate flaky test cases from non-flaky test cases. •

II. BACKGROUND A. Verification Verification encompasses tasks aimed at confirming that a system accurately executes a designated function; as such, it confirms that specified requirements have been fulfilled [19]. Verification, usually, formally/mathematically proves that a system, or some aspect of a system, is correct for a (possibly infinite) set of parameters and inputs [3]. Let M be a system, P be an (infinite) set of parameters, U an (infinite) set of inputs, and Ψ a property that should hold for M . In such a system, Φ(M, p, u) is the behavior of M under parameter p and input u where p ∈ P , u ∈ U . Also, Φ(M, P, U ) is the set of all possible behaviors of M under the parameters in P and inputs in U . In such a system, verification tries to prove that Φ(M, P, U ) |= Ψ for a given set of P and U [3]. B. Falsification versus Testing Falsification techniques search for parameters and inputs from (possibly infinite) sets that falsify (violate) certain highlyspecified properties of systems [20]. In system M , the goal of falsification is to find a p ∈ P and u ∈ U such that Φ(M, P, U ) ̸|= Ψ [3]. Testing involves executing a system or component under defined conditions, observing or recording the results, and subsequently evaluating some aspect of the system or component [19]. More precisely, testing is an approach that applies specific inputs, events or other stimuli to a system, executes the system, and examines system outputs or behaviors for conformance to specifications, with the aim of determining whether a property holds for a given (finite) set of parameters and inputs [3]. In system M , the goal of testing is to find a finite subset of values and parameters that hold the property Ψ, i.e., Φ(M, P̂ , Û ) |= Ψ, where P̂ ⊆ P and Û ⊆ U [3]. Falsification involves actively seeking parameters and inputs from (potentially infinite) sets that contravene the requirements, while testing assesses whether a property holds for a specific (finite) set of parameters and inputs [20]. For system M , if falsification cannot find an input p that violates a given requirement Ψ, there is still no guarantee that there is a subset of inputs P̂ that satisfy requirement Ψ. Also, if testing determines a subset of inputs P̂ that satisfy requirement Ψ, there is no guarantee that there is no input p that violates requirement Ψ. Therefore, testing and falsification are complementary. C. Testing Terminology Here we briefly present testing terminology that is used in this paper. Software testing is an approach for assessing whether a the outputs of a software execution, using specific inputs, conforms to specifications and the required properties hold for a given (finite) set of parameters and inputs [3]. 1) Test Cases, Test Oracles, and Flaky Test Cases: A test case is a set of program inputs, execution conditions, and expected results developed to assess a particular path through the code or a specific behavior of a system [19]. Typically, a test case is a set of test data and preconditions that are used to

4

investigate whether the execution of the system using the test data and under those preconditions leads to expected results and post-conditions. A test oracle is a mechanism that compares the result of executing a test case with the expected result of running that test case on the system and issues a verdict, “passed” or “failed”, to show the system executed correctly or not [21]. A flaky test case is a test case that displays a seemingly random outcome – pass or fail – when run at different times on the same version of the code [9]–[11]. Typically, this occurs because of non-determinism, such as may be caused by synchronization or system timing, or due to the uncertain situations the system may encounter during the testing process, and more generally in the final (virtual or real) environment in which the product is supposed to operate. Fig. 1. Hybrid model for an inverted pendulum [26].

D. Models Models can express the structure and behavior of systems through conceptual or mathematical representations [22]. Models are useful for describing, developing, and validating systems such as CPSs. A formal model is a model that expresses the properties of a system at some level of abstraction [23]. Formal models typically represent systems using certain formalisms, e.g., linear temporal logic [24], and are created prior to the development and deployment of such systems [25]. A wide range of models have been utilized where software systems are concerned. Here we describe two types of models that are relevant to CPSs and that we use in this work. 1) Hybrid Models: A hybrid automaton is a formal model that is used to represent a dynamical system with discrete and continuous components [6]. A hybrid automaton is a labeled and directed graph (a finite state machine) that has the following components [6], [16]: • X: A finite set X = {x1 , ..., xn } of real-valued variables. • V : A finite set of vertices, or discrete modes, indicating a control mode/location. Traditionally these modes are “acceptable” because they are known/expected in the CPS and show the possible modes for the CPS while it is functioning. • E: A set of directed arcs or edges between vertices. An edge e ∈ E is also called a control switch or transition. • Flow condition: equations involving variables in X that describe the continuous evolution of the system. While the hybrid automaton is in control mode v ∈ V , the variables xi change according to the flow condition. • Invariant condition: A condition under which the hybrid model may reside in control mode v. • Guard: An expression involving variables in X. Each transition is associated with a guard. A transition is enabled when its assigned guard is true and its execution modifies the values of the variables in the hybrid model, as necessary, according to the flow conditions. • State: A state σ = (v, x) of the hybrid automaton consists of a mode v ∈ V and a continuous state xn ∈ X. In a hybrid automaton model of a CPS, vertices model the discrete states, or modes, of the system while edges model its

discrete dynamics or switches. Such an automaton can be used to design a controller and to develop a CPS, via simulation models or an implementation. Figure 1, taken (with minor modifications) from [26], depicts a hybrid model for a CPS. (This model and its corresponding CPS are described in Section II-G.) In Figure 1: X = {θ, θ̇, u} V = {“max left”, “max right”, “stabilize”} • E = {“max left” → “max right”, “max right” → “max left”“max left” → “stabilize”, “max right” → “stabilize”, “stabilize” → “max left”, “stabilize” → “max right”} • Flow conditions: [θ̇; θ̈] = f (θ, θ̇, u), u = z(θ, θ̇) in“stabilize”, u = −umax in“max left”, u = umax in “max right” (See [12] for more details). • Invariant conditions: {z(θ, θ̇) < −umax , z(θ, θ̇) > umax , |z(θ, θ̇)| < umax } • Guards: {z(θ, θ̇) < −umax , z(θ, θ̇) > umax , |z(θ, θ̇)| < umax }. • •

2) Simulation Models: Simulation models typically form the basis of computer simulations [27]. Simulation models can be executed in a virtual environment to demonstrate the behavior of a system before that system has been implemented. Tests can be performed as simulations on simulation models [28]. E. Model-Based Testing Model-based testing is typically performed prior to system development and deployment allowing test engineers to examine whether a system’s model conforms to the system’s specifications. Model-based testing techniques typically generate test cases from structural or behavioral models of a system. [29]. Such model-based test cases are typically abstract, and require additional detail relevant to a system’s implementation to be added to them; this transforms them into concrete test cases that can be executed on an implemented system or its more detailed models. Model-based test cases are typically easier to maintain than code-based test cases, and they can be used to measure the coverage (of the model) achieved in testing.

5

F. CPS Simulation/Test Levels CPSs can be typically be simulated and tested using models at three different levels of abstraction [5]: 1. Model-in-the-Loop (MiL) 2. Software-in-the-Loop (SiL) 3. Hardware-in-the-Loop (HiL) These three levels of abstraction can be described as follows. 1) Model-in-the-Loop (MiL): At the MiL level, CPS simulation is performed in a virtual environment, and no physical components are needed. Both controllers and plants (physical parts of a CPS) are modeled using the same notation and are connected in the same diagram. In many sectors these models are created using Matlab/Simulink [18], [30], LabView [31], or other tools that support high-level modeling, and subsequently produce code from the models, e.g., modelbased design [32]. MiL simulation and testing are performed entirely in a virtual environment and without any need for any physical components. MiL testing attempts to verify control algorithms, whether control algorithms and the plant correctly interact, functional requirements, and floating-point arithmetic computations. 2) Software-in-the-Loop (SiL): Developing and testing at the SiL level is also performed in a virtual and simulated environment, which often includes the conversion of floating point data types into fixed-point values as well as the addition of hardware-specific libraries [33]. Testing at the SiL level is still performed in a virtual and simulated environment, as with the MiL level, but the focus at the SiL level is on controller code that can run on the target platform [33]. Further, in contrast to verifying behavior, SiL testing attempts to verify the accuracy of floating point to fixed-point conversions and the conformance of code to control models, especially in contexts in which coding may be manual [33]. 3) Hardware-in-the-Loop (HiL): At the HiL level, the controller software is installed on the final platform and the plant can be a hardware component, or software that simulates that hardware component’s behavior. Testing at this level attempts to verify functional and non-functional requirements, the computer executing control (e.g., the electronic control unit), the real-time infrastructure, and whether the software and hardware work correctly together. G. Extended Example To illustrate the operations of H Y T EST and H Y T EST TF in this article, we utilize the simple example presented in [26], [34], [35]. Figure 2 depicts a system that includes an inverted pendulum of length l and mass m mounted on a cart of mass M . A force, F , is applied to the cart and drags it forward or backward to balance the pendulum. To maintain stability (i.e., a balanced pendulum), a control input is computed and sent to the motors in the cart’s wheels periodically based on feedback the sensors provide, i.e. information about the angle, θ, and angular velocity, θ̇, of the pendulum. A general template for a test case for the inverted pendulum system can be written as follows: n o testcasei,j = Fi , θi , θ̇i , xi , ẋi , θj , θ̇j , xj , ẋj .

Fig. 2. Inverted Pendulum [34]

In a system such as this, a template test case at sampling time, i, consists of a set of one or more test inputs, (possibly empty) sets of pre-condition(s), expected output(s), and expected postcondition(s). In this system, force Fi is the test input, while the angle θi and angular velocity θ̇i of the pendulum, as well as the position xi and velocity ẋi of the cart along the x-axis, are pre-conditions. The expected test outputs are θj (j can be the same as i) and xj , and the expected post-conditions are θ̇j and ẋj . Other properties such as the state of the system, its failure or success, or safety properties such as its stability at a given time j, could also be expected post-conditions – we omit these for simplicity. A specific test case replaces the variables in the template with concrete values. The hybrid model for this example, shown in Figure 1, has three acceptable modes (shown as ellipses) that model its discrete states, and six edges/transitions between modes that model its discrete dynamics or switches. Inside each mode, the first line displays the mode’s name. The next two lines display flow conditions, and the fourth displays an invariant condition. The labels on each edge display guard conditions. The function z(θ, θ̇), which calculates the total energy of the system, together with umax , which is a constant and shows the maximum total energy that the CPS can have in order to remain stable, determine whether the CPS should remain in the same mode or change to another one. In other words, if the absolute value of z(θ, θ̇) is in the range of −umax to umax , then the inverted pendulum is stable; otherwise it is falling down, either left or right. This hybrid model describes how the inverted pendulum functions, as follows. The CPS can begin in any of the acceptable modes. Here we assume that the pendulum is initially in the upright position, i.e., mode stabilize, and when released the pendulum moves towards the left. Suppose force F moves the cart to the right to balance the total energy of the CPS and keep the pendulum stable, captured by the mode stabilize; in this case, the pendulum moves to the right. Now force F moves the cart to the left to balance the total energy of the CPS and stabilize the pendulum. If, for some reason, the total energy of the CPS is less than −umax , i.e., the pendulum is falling to the left, the system will be in mode max left. Depending on the acceptable deviation the controller designer has considered for the modes max left and max right, the controller may or may not be able to stabilize the pendulum and return the CPS to mode stabilize.

6

III. H Y T EST In this section, we summarize our previous work on H Y T EST, which is a test case generation technique that generates test cases based on hybrid models, accompanied by appropriate test oracles, for use in testing CPSs early in their development cycle. We limit ourselves to text that we believe is necessary to understand our new technique. Complete details on the H YTEST algorithms and examples of their operation can be found in [12] A. The CPS Development Process Before introducing H Y T EST, it’s essential to understand the CPS development process in which it is intended to be applied. Irrespective of the chosen development methodology, the CPS development process typically begins with a “Requirements Engineering” step. Here, the system’s goals, properties, specifications, and requirements are established. For CPSs, this step often involves determining and modeling the system’s dynamics, usually through mathematical models like equations of motion. Once these models are defined, the hybrid model of the CPS can be designed without additional effort beyond what’s already required in constructing the mathematical models and the requirements engineering process. The subsequent step in the CPS development process involves designing and testing the controller, leading to the integration of the controller’s dynamics into the system. At this juncture, H Y T EST utilizes the hybrid model, the system’s dynamics, and other information extracted during the requirements engineering process to generate test cases. These test cases are then used to test the simulation model at the MiL level, providing results that identify any failures. Engineers correct these faults, and then H Y T EST is reapplied to ensure the model passes all test cases. This iterative testing and correction process continues until no test case fails. At the SiL level, the controller’s code is developed and unit tested, with H Y T EST employed to test the CPS using the same generated test cases. Testing and adjustment continue until no test case fails, ensuring the CPS’s readiness for deployment in the real environment, where H Y T EST can also be utilized for testing. Importantly, H Y T EST can generate test cases regardless of the CPS development methodology, offering flexibility and consistency in testing approaches. Checking the correctness of the hybrid model, including reachability and other issues [36], before using H Y T EST, is not addressed in this work. However, in observed cases where the hybrid model was flawed, H Y T EST detected errors, displayed error messages, and terminated execution. Additionally, as H Y T EST assumes every state can be the CPS’s start state, reachability checking of the hybrid model to ensure each state is reachable from a specific start state is unnecessary.

The Test Case Generation Module receives data about the hybrid model, unacceptable conditions, system and CPS dynamics, and simulation data—these data are usually available after the requirement engineering phase or can be derived easily from the outcome of that phase—in the form of inputs that the user provides to answer the questions that the module asks and in the form of text files. Using this data, the Test Condition Graph Generation step generates a “Test Condition Graph” that shows all possible modes and transitions (whether acceptable or not), in the form of passing (if they are acceptable) or failing (if they are unacceptable) conditions. Because the graph includes all acceptable or unacceptable conditions, it captures the effect of any event (whether expected or not) that may occur in the CPS, whether or not the event is known a priori. Next, the Test Condition Generation step combines each condition in the test condition graph with other information about the conditions, such as the mode of the CPS for each condition, creating a set of test conditions. This step generates test conditions for all conditions of the transitions and the modes, in order to achieve complete coverage of the acceptable and unacceptable conditions. When test conditions are ready, the Test Case Generation step uses these test conditions to partition the input space, i.e., it classifies the inputs based on test conditions, and generates test cases. After test cases have been generated, the Test Oracle Module receives them along with the information about the CPS’s variables and simulation data, such as simulation time. Using these, the module performs a Test Execution step on the CPS, followed by a Test Result Inspection step, which checks the execution outputs in order to issue the test verdicts. Algorithm 1 provides an overview of H Y T EST. H Y T EST receives information about a CPS’s hybrid model, conditions that indicate the CPS’s failure, simulation parameters, information on the CPS’s dynamics, and a simulation model of the CPS. All of these are either required to design and develop the CPS or can be obtained easily from its requirements. The algorithm outputs the failures that are revealed during the CPS testing process. Algorithm 1: H Y T EST

B. Overview of H Y T EST

Inputs: hModel: Hybrid Model, unConditions: Unacceptable Conditions, simData: Simulation Data, sysDyn: System and CPS Dynamics, simulationModel: Simulation model Output: failures: Detected failure in the CPS 1 hModel=getHybridModelData(); 2 unConditions= getUnacceptableConditions(); 3 simData= getSimulationData(); 4 sysDyn= getDynamics(); 5 cGraph= genConditionGraph(hModel,unConditions); 6 tConditions= genTestConditions(cGraph); 7 testcases=genTestCases(tConditions,simData,sysDyn); 8 [failures]=testCPS(testcases,simData,tConditions,simulationModels); 9 return f ailures

Figure 3 provides an overview of H Y T EST. As depicted in the figure, H Y T EST consists of two main modules: test case generation and test oracle modules.

H Y T EST begins (line 1) by retrieving data about the hybrid model of the CPS. This data includes the total number of

7

Fig. 3. H Y T EST Overview

modes, the CPS goal(s), invariant conditions, guard conditions that show the transitions between modes, and variables in the hybrid model including their acceptable value range and precision. Next (line 2), H Y T EST obtains a list of “unacceptable conditions”, provided by the system designer or test engineer: these are conditions that are not expected to occur during the operation of the CPS, or that would lead to a failure. If no unacceptable conditions are provided for the CPS, H Y T EST temporarily sets the variable unConditions to null until a later set of operations can adjust it. H Y T EST next (lines 3-4) retrieves data about the simulation model used to test the CPS, and about the system’s and CPS’s dynamics including its statespace representation; transfer functions or Matlab code that implements the system’s dynamics; and initial values, inputs, and simulation time, provided as Matlab “init” files. Using this information, H Y T EST creates a “condition graph” (line 5) and using this graph, generates test conditions (line 6).

and the displacement of the cart is not more than 3m, which here is an unacceptable condition, then the CPS’s stays in the mode “stabilize”. For each edge in the graph, H Y T EST obtains its source and destination modes and concatenates them as a formatted string, which is easy to parse when the approach needs to extract specific data from the test conditions. H Y T EST concatenates the edge’s label, which shows under what conditions the CPS changes mode from source to destination, to the formatted string. Finally, it appends the type of the destination mode (of the edge) to the formatted string, which helps the test oracle issue the correct test verdict for the test inputs. If the destination mode is: 1) final then the type is “passed”, 2) failing then type is “failed”, and 3) acceptable then the type is “acceptable”. The following line illustrates a sample test condition that is generated by H Y T EST for our example and corresponds to the self-loop on the mode stabilize.

H Y T EST uses test conditions to partition the input space and then generate test cases (line 7). It passes these test cases, along with simulation data, to the testCPS function (line 8). The testCPS function implements a testing framework with which to test the CPS by running its simulation model, and ultimately returns the failures that are revealed by test cases during testing.

stabilize, stabilize # (|z(θ, θ̇)| < umax ) & ∼ (z(θ, θ̇) > umax ) & ∼ (z(θ, θ̇) < −umax ) & ∼ (x > 3)@passed

To generate test cases using information about the hybrid model and unacceptable conditions, H Y T EST generates a specific type of graph that we call a condition graph. A condition graph is a representation of a CPS that shows all acceptable, final, and failing modes in the CPS, along with all conditions that change the mode of the CPS from one to another and may lead the CPS to a failure or success. An example of condition graph is shown in Figure 4. As we see, H Y T EST categorizes the modes as acceptable (any mode that is neither final nor failing), final, or failing, to show whether the CPS has reached its goal (final modes), has failed (failing mode), or is working toward its goal (acceptable modes). As an example, the edge from the mode “stabilize” to “stabilize” shows that if the absolute/value of the total energy of the CPS, z(θ, θ̇), is less than the maximum total energy that the CPS can have in order to remain stable, umax , and the total energy of the CPS is neither greater that umax nor less that −umax

Using these test conditions, H Y T EST partitions the input space. It examines the test inputs against the test conditions and finds all test conditions a test input may fit. Each test input may lead the CPS to each of failing, final, or acceptable modes, which is recognizable using the type of the destination mode in the test condition they fit. A concrete test case for our inverted pendulum example looks like this: testcasei = {−0.495502602162482, 2.88465092291609, 0.529831722628956, 1.33637906750083, passed}. The members of this set are the pendulum’s angle, the pendulum’s angular velocity, the cart position, the cart velocity, and the expected type of the mode that the CPS will be in when the its variables are set to these values (in this case the type of mode is “passed” which shows the CPS starts from a final mode). Using these test cases, H Y T EST begins the test execution process. H Y T EST starts the CPS in different states, as starting state of the execution, and checks whether the CPS operates correctly or not. To this end, first H Y T EST sets the value of the CPS’s variables to the values in a test case and starts the CPS simulation. As an example, using the concrete test case

8

Fig. 4. Condition Graph Generation, Step 2

We next introduce and define some new terminology which we use in explaining our approach.

sampling time j. We call this condition a flaky condition. If the condition causes the CPS to fail the test we call it a failing flaky condition. Otherwise, it is a passing flaky condition. For example, in the test condition graph that H Y T EST has generated for the inverted pendulum (see Figure 4), for mode “stabilize”, c & ∼ a & ∼ b & ∼ d which is (|z(θ, θ̇)| < umax ) & ∼ (z(θ, θ̇) > umax ) & ∼ (z(θ, θ̇) < −umax ) & ∼ (x > 3) is a potential passing flaky condition, and d, which is sim(x > 3), is a potential failing flaky condition. We use term potential here because according to the test condition graph, the CPS may fail or pass when it is in the mode “stabilize”. Further steps are required to recognize it as a real passing/failing flaky test condition, as we explain in Section V.

A. Flaky Condition

B. Flaky Behavior

Flaky test cases are those test cases on which the CPS can randomly fail or pass. This happens because of a change in the condition under which the CPS is functioning at some

As noted earlier, CPSs are reactive systems, i.e., they function according to the feedback they receive through their sensors and from the environment. When an incident (an

for our inverted pendulum CPS, H Y T EST starts simulating the CPS from a “passing”, i.e., final, state by setting the initial values of the CPS’s variables to the values in the test case and simulates the CPS for the entire simulation time. Then, it gets the test output values and checks what test condition they fit and recognizes the type of the states of each moments of the CPS’s simulation. Using this information H Y T EST issues the test verdict. IV. T ERMINOLOGY

Fig. 5. H Y T EST TF Overview

9

unexpected event) happens, whether within the CPS or in its functioning environment, the CPS reacts to it and adapts its behavior to the condition related to that incident. If the condition is a flaky one, no matter whether passing or failing, we call the CPS adaptation a flaky behavior. In other words, a flaky behavior is the CPS’s reaction to a flaky condition. In Figure 4, when the CPS is in mode “stabilize”, if a failing flaky condition is encountered the CPS may (we use the word “may” because these flaky conditions are just potential flaky conditions) change the mode to “failing” and if a passing flaky condition occurs the CPS may stay in the mode “stabilize”, which is passing. These behaviors are flaky behaviors because they are the results of flaky conditions.

Algorithm 2 presents a modified version of testCPS, with the original lines shown in black, and the new, added flakiness inspection steps shown in green. The new testCPS algorithm, referred to here as testCPS´ , takes the generated test suite, test conditions, and simulation data as inputs and returns test results, including passes, failures, and potential flaky test cases revealed during testing. testCPS´ begins by retrieving test cases and extracting test inputs (line 2) and the initial modes (line 3). H Y T EST TF starts the CPS in different modes and transitions. To this end, for each test case, testCPS´ (lines 4-9) Algorithm 2: Test execution and test oracle testCPS´ (testcases, simData, tConditions, simM odel) inputs : testcases: Test Cases, simData: Simulation Data, tConditions: Test Conditions, simModel: Simulation Model of CPS output: results: Test results 2 testInputs=extractTestInputs(testcases); 3 initialMode= extractInitialMode(testcases); 4 foreach test input inputi ∈ testInputs do 5 results.values ← inputj 6 results.simT ime ← simData.simT ime 7 results.modes ← initialM odei 8 try: 9 testOutput=simulateModel(inputi , simData); 10 possibleCPSModes=recognizeCPSModes(... ...testOutput, tConditions); 11 foreach mode modej ∈ possibleCP SM odes do 12 if modej == null then 13 print(“The hybrid model was not designed correctly.”); 14 else 15 if ~isAllowed(modej , modej+1 ) then 16 resultsi .case ← “f ailed′′ 17 else if modej contains “failing” then 18 if modej ~contains “passing” then 19 resultsi .case ← “f ailed′′ 20 else if modej contains “passing” then 21 resultsi .case ← “f laky ′′ 22 resultsi .values ← valuesj 23 resultsi .simT ime ← simData.simT ime − j 24 resultsi .modes ← modej 25 end 26 if possibleCP SM odes ~contains “failing” then 27 lastM ode ← thelastmode in possibleCP SM odes; 28 if lastM ode contains final then 29 if initialM odei == f inal || initialM odei == acceptable then 30 resultsi .case ← “passed′′ 31 else 32 resultsi .case ← “f laky ′′ 33 resultsi .values ← inputi 34 resultsi .simT ime ← simData.simT ime 35 resultsi .modes ← initialM ode 36 else 37 resultsi .case ← “f laky ′′ 38 resultsi .values ← valuesendT ime 39 resultsi .simT ime ← simData.simT ime 40 resultsi .modes ← modeendT ime 41 catch Exception: 42 resultsi .case ← “f ailed′′ 43 end 44 end 45 return results; 1

C. Flaky Modes As noted before, hybrid models and the test condition graph that H Y T EST generates have several modes. If the CPS displays flaky behavior while in any mode, we call that mode a flaky mode. In Figure 4 the states “stablize”, “max left”, and “max right” are potential flaky modes. V. A PPROACH In this section we explain our Test Results Validation (TR E VA) and Flaky Test Recognition (F LA R E) approaches, which as noted before, are integrated into H Y T EST to form H Y T EST TF. A. Overview of Approach Figure 5 provides an overview of H Y T EST TF and how TR E VA and F LA R E are integrated into H Y T EST. As shown, TR E VA depends on the module Flakiness Inspection, which is integrated into H Y T EST’s test oracle. While H Y T EST TF is testing the CPS, it marks those test cases that fail but, under different conditions, have a chance to pass, as potential flaky tests. Next, it sends test results, which include results from potential flaky test cases as well, to TR E VA. TR E VA validates the failures and passes by distinguishing real failures and passes from flaky ones. F LA R E receives test conditions and test cases. First, using test conditions, it finds all modes that could be flaky. Next, using test cases, it examines the potential flaky modes and checks whether any of them can be real flaky modes. For each mode it checks whether it passes and fails the test case under passing and failing flaky conditions. If so, the test cases that are related to those flaky conditions are real flaky test cases, the mode is a real flaky mode, and the flaky conditions are real flaky conditions under which the CPS may fail while it is operating in the real environment. Using this information, the CPS developer can take preventative steps to avoid failures that occur due to failing flaky conditions. B. H Y T EST TF’s Test Oracle and Flakiness Inspection The final step of H Y T EST’s Algorithm 1 (line 8), presented in Section III-B, invokes algorithm testCPS, into which the flakiness inspection module is integrated. testCPS executes generated test cases on the CPS and inspects the test results and flakiness.

10

initially sets the values of the variables to the test input values of test cases to put the CPS in every possible acceptable, final, or failing mode and then it monitors the behavior of the CPS until the end of the simulation. As an example, using the concrete test case for our inverted pendulum CPS (presented in Section III-B), H Y T EST TF starts simulating the CPS from a “passing”, i.e., final, mode by setting the initial values of the CPS’s variables to the values in the test case, and simulating the CPS for the entire simulation time. Also, for each test input on lines 5-7, it initializes results of each test run that will be used by TR E VA to validate the test results. For each test input, testCPS´ receives the simulation output(s) as one or more sets of values with the length of the (simulation time)/(sampling time). Then, testCPS´ inspects the simulation outputs to see whether the CPS has stopped at a final mode through sequences of allowed transitions and whether the values of the variables fit the goal conditions. The test oracle behaves as follows. First, using the test conditions and the test output signal, the test oracle recognizes the possible modes of the CPS at any sampling time (line 10) by checking the test outputs against test conditions. Since a test output at time j may fit in several test conditions, the CPS at time j can be in different modes and the real mode of the CPS depends on its real condition at testing time. At this point (lines 11-13), testCPS´ checks whether there are any test outputs with no modes assigned to them. If there are, the hybrid model was designed incorrectly (line 13), because there is a behavior in the CPS that does not fit any modes/transitions represented in the hybrid model: i.e., the hybrid model omitted a mode/transition. According to the modes in the condition graph, the modes and transitions (based on their destinations) of the CPS can be categorized into three groups: acceptable, final, and failing. When the possible modes the CPS can be in at a sampling time are recognized, the flakiness inspection part of the test oracle inspects the test results to find test cases that may be flaky. To this end, it looks for one of the following cases: • At any sampling time j, the CPS has more than one possible mode; i.e., the test outputs at sampling time j fit in more than one test condition, and the set of possible CPS modes contains at least one “failing” and at least one “passing” mode (lines 20-24). • The CPS has passed the test while we expect a failure (lines 32-35). In this case, if the CPS is retested for the whole simulation time while it starts under a flaky failing condition, the test may fail instead of pass, which is expected. • The CPS has not been in any “failing” mode during the whole simulation time, and we do not expect the CPS to fail the test, but the CPS was not able to reach its target, i.e. the CPS’s mode at the end of its simulation is “acceptable” and not “final”. In this case, although the CPS has failed the test, it may pass it if it is given more time (lines 37-40). If any of these cases occur, the test oracle marks the test case as a “potential flaky test case” and saves the values at sampling time j, the rest of simulation time, and the possible modes of the CPS at time j. Using this information, TR E VA

can start the CPS from the state in which the test oracle has recognized the flakiness and continue testing the CPS for the rest of the simulation time to see whether the CPS passes or fails the test and whether the failure is a real or a flaky one. Based on the potential cases of flakiness, to issue the correct test verdict for each test case, the test oracle works as follows: • Using the condition graph, the test oracle determines (lines 15-16) whether transitions between consecutive modes are allowed or not, i.e., transitions between modes of the test output at times i and i + 1. If a transition is not allowed then a failure occurred, e.g., there is an overshoot, (a sudden change in a value at two consecutive sampling times) in the CPS output values. • If the possible modes of the CPS include “failing” in at least one sampling time and do not contain “passing” at the same sampling time, the test oracle issues a “failed” verdict for the test case (line 17-19), but if they contain “passing” at the same sampling time, then the failure, or the pass, or in general the test run, is “flaky” (lines 20-24). • If the test verdict is not “failed”, the possible modes of the CPS at the last moment of the simulation include “final”, and the initial mode in the test case includes “final” or “acceptable”, then the test oracle issues a “passed” verdict (lines 26-30). • If the test verdict is not “failed”, the possible modes of the CPS at the last moment of the simulation includes “final”, and the initial mode in the test case includes “failing”, then the test oracle issues a “flaky” verdict (lines 32-35). • If the test verdict is not “failed” and the possible modes of the CPS at the last moment of the simulation include “acceptable”, then the CPS has failed the test (lines 3740), because the CPS was not able to reach its target, i.e. the final mode or goal conditions, within the simulation time (the time within which the CPS must reach the goal condition), but since the CPS may reach the goal in a longer time and pass the test, so the test oracle issues a “flaky” verdict. In our inverted pendulum system, although the modes “max left” and “max right” are both acceptable, the goal of designing such a CPS is to keep it in the “stabilized” mode; therefore, if this CPS is not in the mode “stabilized” at the end of the simulation, the CPS failed to reach its goal. In this case, although the CPS is in an acceptable mode at the end of the simulation, H Y T EST TF recognizes a failure because the goal was not reached. • If the simulation encounters any problem that leads to an exception, the test oracle issues a “failed” verdict (lines 41-42). C. TR E VA: Test Results Validation After testCPS finishes the testing and returns the test results, TR E VA (Algorithm 3) receives the results, test conditions, and simulation data and model as inputs and returns validated test results, which include real passes, real failures, and real flaky test executions. TR E VA begins by retrieving values and start modes from the set of results (line 3). Using the values and start mode,

11

Algorithm 3: TReVa

Algorithm 4: FlaRe

TReVa (results, simData, tConditions, simM odel) inputs : results: Test Results, simData: Simulation Data, tConditions: Test Conditions, simModel: Simulation Model of CPS output: validatedResults: Validated Results 2 [testcases, simData]=extractVlaues(results, ... 3 ...newSimData); 4 [newResults, ∼ ]=testCPS(testcases, newSimData, ... 5 tConditions, simM odel); 6 foreach new result newResi ∈ newResults do 7 if newResi .case == “passed” && resultsi .case == “passed” then 8 validatedResultsi ← “passed” 9 else if newResi .case ~= “passed” && resultsi .case ~= “passed” then 10 validatedResultsi ← “failed” 11 else 12 validatedResultsi ← “flaky” 13 end 14 return validatedResultsi ;

FlaRe (testcases, simData, tConditions, simM odel) inputs : testcases: Test Cases, simData: Simulation Data, tConditions: Test Conditions, simModel: Simulation Model of CPS that have no real failures recognized by H Y T EST TF output: flakyTests: Flaky Test Cases, flakyModes: Flaky Modes 2 [failures, ∼ ] = testCPS(testcases, simData, ... 3 tConditions, simM odels); 4 modes = getP otF lakyM odes(tConditions); 5 foreach test case testi ∈ testcases do 6 startM ode = extractStartState(testi ); 7 if f ailuresi then 8 modesstartM ode .append(“f ailed′′ ) 9 else 10 modesstartM ode .append(“passed′′ ) 11 end 12 foreach flaky mode modej ∈ modes do 13 if modej contains “failed” and “passed” then 14 add modej to f lakyM odes; 15 foreach test case testi ∈ testcases do 16 if testi ’s start mode is modej then 17 add testi to f lakyT ests; 18 end 19 end 20 return f lakyT ests, f lakyM odes;

1

it generates a new set of test cases. TR E VA uses this new test suite to start testing the CPS from the states in which flakiness has been recognized for the test cases in the original test suite, which was generated by H Y T EST, for those test runs that are marked potentially flaky. Also, for each potential flaky test case, TR E VA updates the simulation time to the value that is retrieved from it. for the test runs that are not marked potentially flaky, TR E VA uses this new test suite to validate the results for those test runs. Once TR E VA has the new test suite and updated simulation time for each test case, it calls testCPS (line 4-5) to retest the CPS using the new test suite. When testCPS returns the new results, in lines 6-12, TR E VA compares them with those results it received from the test oracle. If the CPS has passed the test both times (lines 7-8), then TR EVA issues a passed test verdict for the test case. If the CPS has not passed the test both times, i.e. it has failed the test or the test run is a potential flaky one, (lines 9-10), then TR EVA recognizes the test case as a real failed one. Remember that a potentially flaky test run is a test run that has failed, but it has a chance to pass. On line 12, if the CPS has not passed the test either times and passed it the other time, then TR E VA marks the test case as a real flaky test case. It is worth mentioning that the faults that caused the real failures may be the cause of flaky behaviours in the CPS, too. and fixing them may resolve the flaky behaviours, as well; Therefore, at this stage, a test engineer should resolve the real failures and ignore those that are recognized as really flaky. D. F LA R E: Flaky Test Case Recognition Once the faults have been fixed and H Y T EST FT does not reveal real failures on the fixed model, it is time to determine the real flaky test cases. It is important to reveal real flaky test cases because every real flaky test case is the consequence of at least one failing and one passing flaky condition, starting from the same mode. When there is no real failure in the CPS, these flaky conditions are the result of incidents within

1

the CPS or in the environment that the CPS is executing. Recognizing these flaky conditions can provide insights for CPS developers, allowing them to take preventive measure against those incidents. Algorithm 4 shows our proposed approach, F LA R E. F LA R E receives test cases that are generated by H Y T EST, test conditions, and simulation data as inputs and returns real flaky test cases and real flaky modes. First, in lines 2-3, F LA R E calls testCPS to test the corrected simulation model of the CPS using the test cases generated by H Y T EST. Next, on line 4, using the test conditions, which are the conditions on edges in the vstest condition graph, it finds all the modes that are likely flaky. F LA R E marks a mode as a potentially flaky mode if one of the following cases is true. If the mode has at least one outgoing edge to a failing mode and and at least one outgoing edge or edge to a passing mode. • If the mode has at least one outgoing edge to a failing mode and and at least one outgoing edge or edge to an acceptable mode. • If the mode has at least one outgoing edge to a passing mode and and at least one outgoing edge or edge to an acceptable mode. •

In the second and third cases, if the mode has an outgoing edge to an acceptable mode, then it may have a path to a passing mode or a failing mode or both; Hence, F LA R E considers the mode as a potentially flaky mode. Then for each test case, it extracts the start mode (lines 5-6). If the test case has failed/passed the test case, F LA R E appends a “failed”/“passed” to its start mode to show that the CPS has failed/passed the test case starting from this mode

12

(lines 7-10). In the next step, lines 12-13, F LA R E looks for flaky modes. If a mode has a “passed” and “failed” it means the CPS has passed and failed the test case starting from this mode, so the mode is flaky and F LA R E adds it to the set of real flaky modes (line 14). Each test case that has a flaky start mode is a real flaky test case and is added to the set of real flaky test cases (lines 15-17). Finally, F LA R E returns the real flaky test cases and flaky modes. Since the test cases include their related test conditions and expected test verdicts, F LA R E, implicitly, returns failing and passing flaky conditions as well. E. Implementation Details We used the H Y T EST’s implementation described in our previous work [12] and integrated the flakiness inspection module into it. Also, I implemented TR E VA and F LA R E using Java 1.8.0 and Matlab R2022. Condition graph generation and test condition generation algorithms were implemented using Java and the rest of the implementation is in Matlab. The Java code takes a hybrid model and other inputs, parses them, and generates a condition graph and test conditions. The Matlab code obtains the test conditions and the rest of the inputs, generates and selects test cases, executes these test cases on the Simulink model of the CPS, returns and validate the test results and flaky test cases, modes, and conditions. VI. E MPIRICAL S TUDY To evaluate H Y T EST TF, with a particular focus on the performance of TR E VA and F LA R E, and the flakiness inspection performed by H Y T EST TF‘s test oracle, we conducted an empirical study that focused on the following questions: RQ1: How effective is H Y T EST TF at recognizing real failures, real passes, and real flaky failures? RQ2: How effective is H Y T EST TF at revealing distinct faults? RQ3: How effective is H Y T EST TF at revealing real flaky modes? RQ1 is important because the correct recognition of real failures and real flaky failures can reduce fault localization and debugging efforts and consequently testing costs, by allowing test engineers to focus on actual failures and the faults that cause them, and not be distracted by the results of flaky test cases. Test cases that lead to flaky executions, having been identified, can also be repaired, rendering them useful, or discarded, ensuring that they do not continue to be a distraction when test suites are used later to test the system at different levels. Finally, real passes are useful in that they help establish whether the CPS has been designed and operates in accordance with its specifications. While differentiating test cases that are flaky from those that are not is important, investigating whether real test failures reveal real faults within the CPS, and whether ignoring flaky test failures does not result in missing faults, is of higher importance. RQ2 helps us assess this; by assessing the numbers of distinct faults in the CPS that are related to the real failures it provides insights into how well H Y T EST TF does at detecting faults in the CPS.

TABLE 1 O BJECTS OF S TUDY

Cruise Control Inverted Pendulum Hexapod Rooms and Heaters Automatic Transmission

# of Simulink Blocks 16 11 348 24 14

# of Modes in Hybrid Model 2 3 5 6 4

When faults in the CPS are fixed and H Y T EST TF exhibits no more real failures, it is important to investigate whether the CPS can still exhibit flaky behavior. Given information on flaky modes, a CPS developer can implement measures to prevent the CPS from failing due to flaky conditions. RQ3 helps us determine the extent to which this could happen. A. Objects of Study To conduct our study we require CPSs. Just like H Y T EST, H Y T EST TF performs testing at the MiL level, and requires simulation models of the systems being tested. In this study, we used the same set of CPSs used to evaluate H Y T EST, for which simulation models are available.6 We provide a description of those CPSs here. Table 1 provides data on the selected CPSs. The second column shows the number of blocks in the Simulink model of each CPS and the third column shows the number of modes in their hybrid models. “Cruise Control” [37] monitors the speed of a vehicle through sensors, increases and decreases the vehicle’s speed to match a set speed, and maintains that speed. “Inverted Pendulum” is the system [38] introduced in Section II-G and used to illustrate the operation of H Y T EST. “Hexapod” [39] is an autonomous legged robot that is able to move around its environment with high flexibility and stability; it can reach a preset goal and avoid preset obstacles on its way to the goal. “Rooms and Heaters” [40] controls the transfer of moving heaters among adjacent rooms to maintain the temperature of the rooms within a desired range. Finally, “Automatic Transmission” controls an automobile’s speed and engine rpm [41]. The simulation models for all five of these CPSs are (hybrid) Simulink models, and they contain a wide range of block types including Integrator, Merge, Add, Transfer Function, If, State Chart, Relational and Logical Operators. These Simulink models have each been designed using different types of controllers; this allows us to examine whether H Y T EST can generate test cases for CPS’s with different controllers. For Inverted Pendulum we retrieved a hybrid model from [26] and for Cruise Control we followed the description provided in [37] to recreate its model. The hybrid model for Hexapod is from [39]. The hybrid models for Rooms and Heaters and Automatic Transmission are from [42] and [43]. To answer our research questions we required information on fault detection and flakiness recognition for our object 6 The CPSs we selected are all available via links provided in the relevant monographs, all of which are cited in this subsection.

13

CPSs. Unfortunately, no faults have been reported for these systems, and finding events and conditions that cause flakiness in our CPSs was not feasible. Thus, for the fault detection step of our study, we used the faulty models that were generated for use in evaluating H Y T EST [12]. These faulty models are models of CPSs that are mutated using a comprehensive list of Simulink fault patterns based on experiences reported by Delphi Engineers and from their review of the literature on CPSs [44]. We also required “flakiness” in our test cases. Although there are several techniques for injecting flakiness into test cases [45]–[47] and some collections of real world flaky test cases exist [48], [49], they all involve flakiness software systems that are written in specific programming languages such as Java or JavaScript, and do not involve simulation models. Furthermore, the flakiness involved is limited to known/identified sources of flakiness in those types of systems, and this omits other sources of flakiness. Therefore, we chose a different approach to reproduce the effects of flakiness: we injected random noises into simulation models of our CPSs. To reproduce the effects of flakiness, we used the original (unmutated) simulation models of the CPSs and added a noise block to the feedback loop of each CPS model to inject noise into the feedback values. To inject noise, for each CPS, we randomly selected a random number n that is used to randomly select n output variables from the list of the CPS’s outputs. Then for each randomly selected output variable, we selected a random value and added it to the value of that CPS’s output variable at a randomly selected time. Because the noise value could be negative or positive, the addition is an algebraic sum. To determine how many noisy models to create, we measured the time required to test the faulty models and used that to guide the creation and testing of noisy models; this allowed us to conduct the study in a feasible amount of time. For example, the time required to test the faulty models of the Inverted Pendulum was around one minute, so we generated and tested as many noisy models as possible in one minute. Tables 2.1 - 2.5, which are presented in Section VI-E, provide statistics on the number of faulty models (second column) and the number of noisy models (third column) obtained through this process for each of our objects of study. B. Variables and Measures The independent variable in our study is the testing technique utilized. H Y T EST TF is one such technique, and as a baseline for comparison we chose H Y T EST. H Y T EST is a logical choice because it is the only other extent technique for testing CPSs using simulation models. Choosing H Y T EST as a baseline allows us to assess the extent to which H Y T EST TF succeeds in revealing useful information related to test flakiness, when considering research questions RQ1 and RQ2. To perform the evaluation, we used the test results obtained by applying H Y T EST to faulty models of our subject CPSs in our previous study of H Y T EST. Then we tested the noisy models of all the CPSs that we generated as described above. Our dependent variables are the number of real failures, the number of real passes, the number of real flaky failures, the number of distinct faults that the real failures reveal, and

the number of flaky modes. These are measured to evaluate the effectiveness of H Y T EST TF in validating test results. To measure them we wrote a script that counted the number of failures, passes, and distinct faults revealed by H Y T EST and H Y T EST TF on both faulty and noisy models. We wrote a second script to count the number of real flaky test failures and real flaky modes recognized by H Y T EST TF. To evaluate the effectiveness of H Y T EST TF in recognizing real flaky modes, we measured the number of modes that it recognized as such. C. Study Operation In practice, a user (test engineer, control designer, or CPS developer) of H Y T EST TF would run it on a simulation model or CPS they designed to retrieve the validation results along with information on real flaky modes, real flaky test cases, and real flaky conditions. To evaluate the effectiveness of H Y T EST TF in the context of this study, however, we used a three-step process applied to the faulty (mutated) and noisy models that were created for use in our study. The three-step process allows us to more easily capture the values of the dependent variables that we are concerned with. In the first step of our process, for each of the five objects of study, H Y T EST TF uses the same approach used by H Y T EST to generate test cases and test faulty models of the CPS. H Y T EST TF, like H Y T EST, puts the CPS in different initial states by setting the model’s initial values to the test inputs, providing the initial mode for the test oracle, and executing the faulty models. We record the number of failures, potentially flaky failures, and potential passes revealed by this step. Then, we run TR E VA to validate H Y T EST TF’s test results on faulty models and we capture the numbers of real failures, real flaky test cases, real passes, and distinct faults (mutations) that were revealed by real failures. In the second step of our process, we generate noisy models (as discussed before) and repeat what we did on the faulty models in the first step of the process. Because H Y T EST T does not recognize any potential flaky failures on some sets of noisy models when it is testing CPSs, we executed this second step 10 times. We did this in order to make sure we had noisy models that cover cases in which TR E VA did and did not recognize potential flaky failures. In this step, we also measured one more parameter – the number of generated noisy models in each round of execution. At this point of the study we have the results on how well H Y T EST TF has done in providing validated test results. In the third step of the process, we evaluate F LA R E’s ability to recognize flaky modes. We use test cases generated by H Y T EST TF in testing the original (unmutated) simulation models of the CPSs to conduct this step. We measure the number of real flaky modes that F LA R E recognizes. Finally, to assess the accuracy of the results, we compared the results obtained from H Y T EST TF with those obtained by a manual assessment method, through the following process. To validate measures for the variables from H Y T EST TF where TR E VA was concerned, we inspected the test results from the test oracle and the validated results that H Y T EST TF provided. If a failure occurred on faulty models both times,

14

we recognized it as a real failure. To obtain measures for other variables we implemented code to inspect the results. The code checked whether the output values fit any failing conditions, and if they did, we marked the test execution as a potential failure. If the output values did not fit any failing conditions and the output values fit the goal of the CPS at the end of the simulation, we marked the test execution as a potential pass. When test executions were marked differently across executions we counted them as real flaky test cases. When test executions were marked the same across executions we counted them as real passes or real failures. To obtain measures for the variables from H Y T EST TF where F LA R E was concerned, we looked in the condition graph for modes that had at least one outgoing passing transition and at least one outgoing failing transition. Then, we used the test cases related to those transitions to test the CPS and checked the results. If the CPS passed the test using the passing test case and failed the test using the failing test case, then we marked the mode as a real flaky mode and marked those test cases as flaky test cases. The results of our manual validation matched the results returned by H Y T EST TF for all of our subject CPSs. D. Threats to Validity External validity threats concern the generalization of our findings. As objects of study we selected five CPSs, so our results pertain to those CPSs and may not generalize beyond them. To answer our research questions we relied on the insertion of mutations into Simulink models. Our mutations are based on those defined in earlier research in which they were derived based on experiences from Delphi Engineers and other sources [44], so they have some relevance to natural faults; nevertheless, results obtained using these may not match the results that could be obtained on natural faults occurring in practice. The hybrid models obtained for our objects represent only five specific models. However, we did verify the simulation models against the hybrid models and specifications and made sure both models behaved in accordance with their corresponding specifications. Internal validity threats concern uncontrolled factors that may have affected our results. Errors in our implementation of H Y T EST TF and the manual assessment method we used to verify the correctness of that implementation could do so. Construct validity threats concern our metrics and measures, and how well they actually capture the effects we are trying to measure. The metrics we capture concern numbers of real failures, real passes, real flaky failures, distinct faults revealed, and flaky modes. The extent to which these metrics reflect potential efficiency gains in the underlying testing and debugging process is not directly examined. E. Results Tables 2.1, 2.2, 2.3, 2.4, and 2.5 show the results of our study for each of the five object CPSs in turn. In these tables, the number of test cases that H Y T EST and H Y T EST TF have generated and used to test those CPSs is noted in the caption. In a given table, each row shows the results of one of the

ten rounds that were conducted to generate noisy models for that object CPS, test them using H Y T EST TF, and run T RE VA to validate the test results (rounds are numbered in Column 1). Column 2 gives the number of faulty models created for the CPS, and Column 3 gives the number of noisy models generated in each round. Columns 4, 6, and 8 show the numbers of real failures, real passes, and real flaky runs recognized by H Y T EST TF, while Column 5 and 7 show the numbers of real failures and real passes recognized by H Y T EST. Column 9 shows the percentage of test runs that are real flaky runs. Columns 10 and 11 show the number of distinct faults, i.e., the number of mutations plus the number of real faults (if any) found in the original model7 that were detected by H Y T EST TF and H Y T EST, respectively. We now consider research questions RQ1 and RQ2. We begin with the data for the Cruise Control CPS, shown in Table 2.1. As Table 2.1 shows, during the first round of the study, 26 faulty and 16 noisy models were tested by H Y T EST and H Y T EST TF; the results of testing were validated by H Y T EST TF. Because the number of test cases used was 11, and each model was tested using all of the test cases, the total number of test runs for both H Y T EST and H Y T EST TF is 11 × (26 + 16) = 462. Among these test runs H Y T EST recognized 248 failures and 214 passes while H Y T EST TF recognized 148 real failures, 144 real passes, and 170 flaky test runs. For both H Y T EST and H Y T EST TF, these numbers add up to 462, which is the total number of test runs. This shows that H Y T EST TF has validated the results of all test runs for Cruise Control during the first round. Considering the results across all of Tables 2.1 to 2.5, a substantial portion of the test runs (from 26% to 56.1%) are flaky across all of our subject CPSs. For Cruise Control the mean percentage of flaky tests is 40.3%. The median value of 40.0% is very close to the mean and indicates a fairly symmetric distribution of flaky test percentages, with no extreme outliers skewing the data significantly. This is true for all other CPSs except for Inverted Pendulum, where the median (34.6%) is nearly 1% apart from mean (33.7%). These levels of flakiness can undermine the overall effectiveness and efficiency of the testing process. Also, the data shows that for all CPSs, as for Cruise Control, H Y T EST TF validated the results of all test runs during the first round. For another look at the data pertaining to flaky test runs, Table 3 provides descriptive statistics for the flaky run percentages, i.e., data in Column 9 in Tables 2.1 to 2.5, for all of our CPSs. Column 1 names the CPSs. Columns 2 through 5 show the minimum and maximum values, range, mean, median, and standard deviation of the flaky runs, respectively. For example, the first row of the table provides data for Cruise Control. As the Table 3 shows, for Cruise Control, the minimum percentage of flaky test runs was 36.6%, so at best, 63.4% of the test runs were not flaky. The maximum percentage was 44.6%, so at best, of the 55.4% of the test runs were not flaky. Thus, the validity of the test results varied by 8% 7 As we reported in our previous work [12], H Y T EST revealed some actual failures while we were executing our implementation of it using the original models, i.e. models that were not mutated. Those failures were related to one fault in each of the CPSs.

15

TABLE 2.1 R ESULTS OF TESTING C RUISE C ONTROL USING 11 TEST CASES Round 1 2 3 4 5 6 7 8 9 10

Faulty Models 26 26 26 26 26 26 26 26 26 26

Noisy Models 16 16 15 16 16 18 18 18 18 17

Failures (H Y T EST TF) 148 150 153 156 149 162 146 162 162 146

Failures (H Y T EST) 248 278 291 262 254 308 286 277 278 279

Passes (H Y T EST TF) 144 115 97 137 140 113 122 144 143 120

Passes (H Y T EST) 214 184 160 200 208 176 198 207 206 194

Flaky Runs 170 197 201 169 173 209 216 178 179 207

Flaky Runs Percentage 36.8% 42.6% 44.6% 36.6% 37.5% 43.2% 44.6% 36.8% 37.0% 43.8%

Distinct Faults (H Y T EST TF) 27 27 27 27 27 27 27 27 27 27

Distinct Faults (H Y T EST) 27 27 27 27 27 27 27 27 27 27

Flaky Runs Percentage 39.0% 33.0% 26.0% 26.4% 35.6% 36.5% 37.2% 33.8% 40.8% 28.4%

Distinct Faults (H Y T EST TF) 21 21 21 21 21 21 21 21 21 21

Distinct Faults (H Y T EST) 21 21 21 21 21 21 21 21 21 21

Flaky Runs Percentage 41.5% 41.6% 44.8% 44.0% 44.7% 45.4% 45.2% 42.4% 46.6% 44.4%

Distinct Faults (H Y T EST TF) 192 192 192 192 192 192 192 192 192 192

Distinct Faults (H Y T EST) 192 192 192 192 192 192 192 192 192 192

Distinct Faults (H Y T EST TF) 57 57 57 57 57 57 57 57 57 57

Distinct Faults (H Y T EST) 57 57 57 57 57 57 57 57 57 57

TABLE 2.2 R ESULTS OF TESTING I NVERTED P ENDULUM USING 13 TEST CASES Round 1 2 3 4 5 6 7 8 9 10

Faulty Models 20 20 20 20 20 20 20 20 20 20

Noisy Models 22 24 19 20 21 23 23 23 23 23

Failures (H Y T EST TF) 176 182 180 186 175 175 194 165 188 196

Failures (H Y T EST) 348 329 282 295 326 334 352 300 370 325

Passes (H Y T EST TF) 157 201 195 197 169 180 157 205 143 204

Passes (H Y T EST) 198 243 225 225 207 225 207 259 189 234

Flaky Runs 213 189 132 137 189 204 208 189 228 159

TABLE 2.3 R ESULTS OF TESTING H EXAPOD USING 20 TEST CASES Round 1 2 3 4 5 6 7 8 9 10

Faulty Models 191 191 191 191 191 191 191 191 191 191

Noisy Models 26 26 26 24 18 18 20 26 23 29

Failures (H Y T EST TF) 1202 1202 1135 1149 1136 1108 1118 1186 1111 1178

Failures (H Y T EST) 2073 2077 2088 2073 2042 2033 2048 2082 2086 2125

Passes (H Y T EST TF) 1335 1331 1260 1258 1175 1173 1194 1312 1175 1268

Passes (H Y T EST) 2267 2263 2252 2227 2138 2147 2172 2258 2194 2275

Flaky Runs 1803 1807 1945 1893 1869 1899 1908 1842 1994 1954

TABLE 2.4 R ESULTS OF TESTING ROOMS AND H EATERS USING 43 TEST CASES Round 1 2 3 4 5 6 7 8 9 10

Faulty Models 56 56 56 56 56 56 56 56 56 56

Noisy Models 10 9 11 9 9 9 9 9 10 7

Failures (H Y T EST TF) 562 556 627 613 556 613 559 559 560 559

Failures (H Y T EST) 1092 1085 1081 1067 1085 1067 1094 1076 1083 1062

Passes (H Y T EST TF) 1024 993 1131 1059 993 1059 997 1001 1026 945

Passes (H Y T EST) 1746 1710 1800 1728 1710 1728 1701 1719 1755 1647

Flaky Runs 1252 1246 1123 1123 1246 1123 1239 1235 1252 1205

Flaky Runs Percentage 44.1% 44.6% 39.0% 40.2% 44.6% 40.2% 44.3% 44.2% 44.1% 44.5%

16

TABLE 2.5 R ESULTS OF TESTING AUTOMATIC T RANSMISSION USING 25 TEST CASES Round 1 2 3 4 5 6 7 8 9 10

Faulty Models 66 66 66 66 66 66 66 66 66 66

Noisy Models 51 16 47 37 9 42 38 44 44 43

Failures (H Y T EST TF) 680 439 792 571 339 607 580 560 607 662

Failures (H Y T EST) 2105 1430 2055 1825 1275 1910 1840 1930 1940 1945

Passes (H Y T EST TF) 665 496 606 599 487 636 611 647 648 613

Passes (H Y T EST) 820 620 770 750 600 790 760 820 810 780

Flaky Runs 1580 1115 1427 1405 1049 1457 1409 1543 1495 1450

Flaky Runs Percentage 54.0% 54.4% 50.5% 54.6% 56.0% 54.0% 54.2% 56.1% 54.4% 53.2%

Distinct Faults (H Y T EST TF) 66 66 66 66 66 66 66 66 66 66

Distinct Faults (H Y T EST) 67 67 67 67 67 67 67 67 67 67

TABLE 3 D ESCRIPTIVE S TATISTICS OF F LAKY RUN P ERCENTAGES Cruise Control Inverted Pendulum Hexapod Rooms and Heaters Automatic Transmission

Min-Max 36.6% - 44.6% 26.0% - 40.8% 41.5% - 46.6% 39.0% - 44.6% 50.5% - 56.1%

Range 8.0% 14.8% 5.1% 5.6% 5.6%

across different runs, with a standard deviation of 3.5%. This 8% range together with a standard deviation of 3.5% indicate a moderate variability in the percentage of flaky test runs across different test runs. For Inverted Pendulum the range is 14.8% and the standard deviation is 4.9%. Compared to Cruise Control, it shows a higher variability around the mean. For Hexapod, Rooms and Heaters, and Automatic Transmission this variability is around 5%, which is less than the range for the first two CPSs. Also, for these systems the standard deviation is less than for the first two CPSs. This suggests less variability in the percentage of flaky test runs in these CPSs, as well. Figure 6 provides a graphical view of the data from Table 3, using boxplots to depict the ranges of percentages of flaky test runs observed on the five CPSs studied. As the boxplots show, the largest variability in results occurred on Inverted Pendulum, followed closely by Cruise Control. The other three CPSs display substantially less variation in percentages of flaky test runs observed. To answer RQ1, on all five CPSs, since H Y T EST TF was

Mean 40.3% 33.7% 44.1% 43.5% 54.1%

Median 40.0% 34.6% 44.6% 44.1% 54.3%

Standard Deviation 3.5% 4.9% 1.6% 2.2% 1.5%

able to differentiate real failures and passes from those that were flaky, H Y T EST TF was more effective than H Y T EST in recognizing real failures, real passes, and real flaky failures. Next we turn to research question RQ2. As noted earlier, when evaluating H Y T EST TF, while it is important to differentiate real failures and passes from flaky failures or passes, it is also important to asses the effectiveness of the approach in revealing faults.8 . To this end, we recorded the distinct faults that each failure could reveal. Data in Columns 10 and 11 of Tables 2.1 to 2.5 shows the number of distinct faults, which is the number of faulty models plus the number of real faults in the original model for each object of study, that H Y T EST TF and H Y T EST revealed. This data shows that discarding the real flaky failures did not reduce the fault revealing ability of H Y T EST TF on any CPS other than Automatic Transmission; across the other four CPSs H Y T EST TF revealed as many faults that H Y T EST, i.e., all the faults we had injected or recognized previously. For Automatic Transmission, we first confirmed that H Y T EST TF could not reveal one of the faults, but after carefully and manually investigating the results, we noticed that by changing the conditions under which the Automatic Transmission was working, it alternately passed and failed the test, which means those test runs were real flaky ones. Although initially it seems that H Y T EST TF, and specifically TR E VA has missed one of the faults, it is worth noting that after validating the test results, H Y T EST TF, specifically F LA R E recognizes real flaky modes, test cases, and conditions, and provides these data for the test engineer or system developer, so they can execute the CPS under each of the flaky failing conditions and localize their relevant faults. This means finally that H Y T EST TF reveals all faults, no matter they are flaky faults or not. Finally, we turn to research question RQ3. Table 4 shows the results of our study for H Y T EST TF (specifically for F LA R E) 8 By “revealing faults” we mean that the approach determines that a fault must exist; this does not mean that it localizes the fault

Fig. 6. Percentage of flaky test runs in all CPSs

17

TABLE 4 R ESULTS FOR F LAKY-M ODES R ECOGNITION OF H Y T EST TF

Cruise Control Inverted Pendulum Hexapod Rooms and Heaters Automatic Transmission

Modes in Test Condition Graph Graph 3 4 6 7 5

Potential Flaky Modes 2 3 5 6 4

Real Flaky Modes 2 3 5 6 4

for each of the CPSs. The first column of the table contains names of the CPSs. The second column shows the number of modes that are contained in the condition graph generated by H Y T EST TF for each CPS. The third column shows the number of potentially flaky modes that H Y T EST TF has recognized for each CPS. The fourth column shows the number of real flaky modes that H Y T EST TF recognized. Our manual validation process confirmed the correctness of these results. The results show that H Y T EST TF was able to correctly recognize all of the flaky modes in all of the CPSs studied. VII. D ISCUSSION In this section, we discuss the results of our empirical study, and observations about the results. A. Implications for Efficiency and Correctness Efficiency involves achieving a result with minimal extraneous or redundant effort [19]. In testing CPSs, this result includes generating test cases, executing the system using those test cases, detecting failures, localizing faults, and fixing them; hence, a testing techniques that is efficient accomplishes these things with minimal effort. Correctness, as an attribute of software, is the extent to which the software, documentation, or other items fulfill user needs and expectations, whether explicitly stated or not [19]. As noted earlier, the focus of RQ1 is on reducing the effort required for fault localization/debugging of CPSs and consequently reducing the effort required for testing CPSs, as well as providing insights on whether the CPS operates as it is desired by the end user or not. The former focuses on the efficiency of the test process, the latter on the CPSs correctness. Regarding reduction of testing effort, real flaky failures involve test runs that failed during the testing phase but did not fail during the validation process. When real flaky failures

are not identified, and when the test engineer receives the test results on failures, they begin the debugging process to find the fault(s) that have caused the failures. When a failure is a flaky failure, since the flaky failing condition is unknown and the CPS randomly fails and passes the test, it can be difficult to debug the CPS under the same failing condition and find faults. Although there are several techniques that can recognize flaky test cases in software systems or can detect root causes of flakiness in CPSs, there are limitations for applying them to CPSs (as we discuss in Section VIII) that increases the chances of failing to recognize flaky test cases. Regarding the issue of correctness, real flaky passes involve test runs that passed during the testing phase but did not pass during the validation process. Using the real passes, a CPS designer can obtain more accurate and reliable insights into whether the CPS they have designed conforms to the CPS specification, i.e., to the end user’s expectations and needs. When real flaky passes are not identified, such insights are not reliable because it is unclear whether the CPS unconditionally responds to the end user’s needs or just responds to their needs under some conditions and not under others. To measure the number of real flaky failures in our study, we subtracted the number of real failures that H Y T EST TF recognized from the number of failures that H Y T EST reported. Similarly, to measure the number of real flaky passes in our study, we subtracted the number of real passes that H Y T EST TF recognized from the number of passes that H Y T EST reported. Note that these two numbers add up to the total number of real flaky test runs. We report the results of this step in Table 5. Column 1 in Table 5 shows the rounds of execution. The rest of the table has five sections, one for each object CPS. In each section, the first column shows the total number of real flaky test runs recognized by H Y T EST TF, while the second and third columns show the percentages of real flaky failures and real flaky passes that made up those test runs, respectively. As an example, the first row in the first section shows that on Cruise Control, during the first round of execution, H Y T EST TF recognized 170 real flaky runs, of which 58.8% were real flaky failures and 41.2% were real flaky passes. As the table shows, the minimum percentage of real flaky failures within a set of flaky runs is 40.0%, while the maximum percentage is 90.2%. By discarding this number of real flaky failures, the focus of fault localization and debugging will be on the real failures; Hence, the testing effort and costs will be

TABLE 5 F LAKY T EST RUNS

Round 1 2 3 4 5 6 7 8 9 10

Flaky Runs

170 197 201 169 173 209 216 178 179 207

Cruise Control Flaky Flaky Failures Passes

58.8% 65.0% 68.7% 62.7% 60.7% 69.9% 64.8% 64.6% 64.8% 64.3%

41.2% 35.0% 31.3% 37.3% 39.3% 30.1% 35.2% 35.4% 35.2% 35.7%

Inverted Pendulum Flaky Flaky Flaky Runs Failures Passes

Flaky Runs

Hexapod Flaky Failures

Flaky Passes

Rooms and Heaters Flaky Flaky Flaky Runs Failures Passes

Automatic Transmission Flaky Flaky Flaky Runs Failures Passes

213 189 132 137 189 204 208 189 228 159

1803 1807 1945 1893 1869 1899 1908 1842 1994 1954

48.3% 48.4% 49.0% 48.8% 48.5% 48.7% 48.7% 48.6% 48.9% 48.5%

51.7% 51.6% 51.0% 51.2% 51.5% 51.3% 51.3% 51.4% 51.1% 51.5%

1252 1246 1123 1123 1246 1123 1239 1235 1252 1205

1580 1115 1427 1405 1049 1457 1409 1543 1495 1450

80.8% 77.8% 77.3% 79.6% 79.9% 77.9% 76.0% 71.4% 79.8% 81.1%

19.2% 22.2% 22.7% 20.4% 20.1% 22.1% 24.0% 28.6% 20.1% 18.9%

42.3% 42.5% 40.4% 40.4% 42.5% 40.4% 43.2% 41.9% 41.8% 41.7%

57.7% 57.5% 59.6% 59.6% 57.5% 59.6% 56.8% 58.1% 58.2% 58.3%

90.2% 88.9% 88.5% 89.3% 89.2% 89.4% 89.4% 88.8% 89.2% 88.5%

9.8% 11.1% 11.5% 10.7% 10.8% 10.6% 10.6% 11.2% 10.8% 11.5%

18

reduced by using H Y T EST TF. Also, the minimum percentage of real flaky passes is 9.8% and the maximum is 59.6%, i.e. around 10% to 60% of real flaky runs, over all CPSs, falsely show that the CPS works as it is expected while under a different condition it does not respond to the end user’s needs. B. Faults Versus Design Problems As we noted in Section VI-E, the number of distinct faults that were revealed by real failures was the sum of the number of faulty models we generated for the empirical study in our previous work [12] and the number of faults that H Y T EST could reveal in the original simulation models of the CPSs when we were testing the correctness of our implemented approach. Here it is worth noting that while a failure is an event where a system or system component fails to perform a required function within specified limits [19], a fault is an actual defect in a system. One fault may cause zero or more failures and one failure may be the reason of one or more faults. Failures that H Y T EST revealed in the original models occurred because of a specific fault that was present in each of the subject CPSs: the CPS kept executing when they reached failing conditions. Given that none of our subject CPSs are supposed to continue running when they fail because doing so may put devices or humans under risk, we consider this a problem in the design of the CPSs and we count this as one fault. C. Insights about Flaky Behaviors Although the results of our study for F LA R E do not demonstrate it, it is possible for the number of potential flaky modes recognized by F LA R E to be greater than the number of real flaky modes it recognizes. In this case, if F LA R E does not mark a potentially flaky mode or test case as a real one, it does not necessarily mean that F LA R E has missed them and they are false positives. It may be the case that based on the condition graph they are found to be potentially flaky and the system designer/developer has already considered preventive measures in their CPS design. Such a case is a true negative, i.e., F LA R E has correctly not inferred that the potentially flaky mode or test case is flaky. Finally, when F LA R E recognizes real flaky modes, it returns the test cases that cause flakiness starting from those modes. Since each test case that is generated by H Y T EST TF contains the test conditions along with values that fit in those conditions, the test engineer can use those values to start the CPS from the flaky modes under those flaky conditions, run the CPS, and investigate the CPS’ behavior, in order to find preventative measures by which those flaky failing conditions can be avoided. VIII. R ELATED W ORK Several techniques have been proposed for recognizing flaky test cases in software systems [9], [13], [50]–[52]. There are several challenges, however, for applying these techniques to CPSs. First, they operate on specific types of systems, mainly software systems, so more work is required to determine

whether they are applicable to CPSs and whether they can recognize flaky test cases in testing CPSs. Second, these techniques focus on recognizing flaky test cases during regression testing, whereas flakiness can occur in CPSs at each level and in each type of testing. Third, these techniques usually rely on older versions of source code or test data from previous testing rounds, which provide pre-defined or learned anti-patterns in the test cases. This, again, is possible in a regression testing setting but not directly applicable to the scenarios we consider here. Furthermore, this approach limits the techniques’ ability to a source-code-dependent subset of all possible flakiness causes. Finally, the cited work just investigates the flakiness of failed tests, whereas it is possible for a test of a CPS to pass in a given execution and yet actually be flaky, a possibility that our technique detects. There has been work investigating the root causes of flaky behaviors in CPSs [14], [53] or embedded systems [54], that provides some solutions for maintaining CPSs after flaky tests are revealed, but the papers cited do not discuss approaches for recognizing flakiness. To the best of our knowledge, there has been no work on recognizing flaky tests for CPSs using any approach other than “Rerun”. IX. C ONCLUSION We have presented H Y T EST TF, an approach for testing CPSs, that integrates the HyTest test case generaton and test oracle approach [12] with additional steps that perform test results validation (TR E VA) and flaky test recognition (F LA R E). To evaluate the effectiveness of H Y T EST TF in recognizing real failures, real passes, real flaky test cases, and real flaky modes, we studied its application on a set of CPSs and compared the results with the results obtained by applying H Y T EST to all of these CPSs. Our results show that H Y T EST TF was able to correctly validate the test results and differentiate test runs that failed under some condition(s) and passed under other condition(s) from those that passed or failed unconditionally. We have studied H Y T EST TF in relation to the testing of CPSs at the MiL level; however, we believe that H Y T EST TF could be applied to CPSs at the SiL and HiL levels as well to validate the test results and recognize flaky modes, flaky test conditions, and flaky test cases in those cases. Additional studies could be performed to assess this. In our empirical study, we evaluated H Y T EST TF on five CPSs of various (low, medium, high) complexity and our results are limited to these. Although we do not have any theoretical or practical reason to think that H Y T EST TF’s applicability and scalability might be limited on larger and more complex industrial CPSs, additional empirical work is needed to accurately determine this. R EFERENCES [1] R. Alur, Principles of Cyber-Physical Systems. MIT press, 2015. [2] M. S. Fisher, Software Verification and Validation: An Engineering and Scientific Approach. Berlin, Heidelberg: Springer-Verlag, 2006.

19

[3] J. Kapinski, J. V. Deshmukh, X. Jin, H. Ito, and K. Butts, “Simulation-based approaches for verification of embedded control systems: An overview of traditional and advanced modeling, testing, and verification techniques,” IEEE Control Systems Magazine, vol. 36, no. 6, pp. 45–64, 2016. [4] R. Pressman, Software Engineering: A Practitioner’s Approach, 7th ed. USA: McGraw-Hill, Inc., 2009. [5] R. Matinnejad, S. Nejati, L. Briand, T. Bruckmann, and C. Poull, “Automated model-in-the-loop testing of continuous controllers using search,” in Search Based Software Engineering, G. Ruhe and Y. Zhang, Eds., 2013, pp. 141–157. [6] T. A. Henzinger, M. K. Inan, and R. P. Kurshan, The Theory of Hybrid Automata. Berlin: Springer, 2000, pp. 265–292. [7] D. Harel and H. Kugler, Some Thoughts on the Semantics of Biocharts. Berlin, Heidelberg: Springer-Verlag, 2010, p. 185–194. [8] P. Černý, T. A. Henzinger, and A. Radhakrishna, Quantitative Simulation Games. Berlin, Heidelberg: Springer-Verlag, 2010, p. 42–60. [9] J. Bell, O. Legunsen, M. Hilton, L. Eloussi, T. Yung, and D. Marinov, “Deflaker: Automatically detecting flaky tests,” in 2018 IEEE/ACM 40th International Conference on Software Engineering, 2018, pp. 433–444. [10] M. Eck, F. Palomba, M. Castelluccio, and A. Bacchelli, “Understanding flaky tests: The developer’s perspective,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 830–840. [Online]. Available: https://doi.org/10.1145/3338906.3338945 [11] W. Lam, S. Winter, A. Wei, T. Xie, D. Marinov, and J. Bell, “A large-scale longitudinal study of flaky tests,” Proc. ACM Program. Lang., vol. 4, no. OOPSLA, nov 2020. [Online]. Available: https://doi.org/10.1145/3428270 [12] Z. Sadri-Moshkenani, J. Bradley, and G. Rothermel, “Test case generation and test oracle support for testing cpss using hybrid models,” 2023. [13] W. Lam, R. Oei, A. Shi, D. Marinov, and T. Xie, “idflakies: A framework for detecting and partially classifying flaky tests,” in 2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST), 2019, pp. 312–322. [14] F. Zampetti, V. Nardone, and M. Di Penta, “Problems and solutions in applying continuous integration and delivery to 20 open-source cyber-physical systems,” in 2022 IEEE/ACM 19th International Conference on Mining Software Repositories (MSR), 2022, pp. 646–657. [15] W. M. Taha, A.-E. M. Taha, and J. Thunberg, What is a Cyber-Physical System? Springer International, 2021, pp. 3–18. [16] P. J. Antsaklis and X. D. Koutsoukos, “Hybrid system control,” in Encyclopedia of Physical Science and Technology (Third Edition), third edition ed., R. A. Meyers, Ed. New York: Academic Press, 2003, pp. 445–458. [17] M. A. Schneider, M.-F. Wendland, and L. Bornemann, “Gaining certainty about uncertainty,” in Risk Assessment and Risk-Driven Quality Assurance, J. Großmann, M. Felderer, and F. Seehusen, Eds., 2017, pp. 129–142. [18] “Simulink website,” https://www.mathworks.com/products/simulink.html, accessed: 2020-8-24. [19] “Iso/iec/ieee international standard - systems and software engineering – vocabulary,” ISO/IEC/IEEE 24765:2010(E), pp. 1–418, Dec 2010. [20] B. Hoxha, H. Bach, H. Abbas, A. Dokhanchi, Y. Kobayashi, and G. Fainekos, “Towards formal specification visualization for testing and monitoring of cyber-physical systems,” 2014. [21] A. Memon, I. Banerjee, and A. Nagarajan, “What test oracle should i use for effective gui testing?” in Proceedings of the IEEE International Conference on Automated Software Engineering, Oct 2003, pp. 164–173. [22] “Britannica website,” https://www.britannica.com/science/scientific-modeling, accessed: 2019-3-13. [23] A. V. Lamsweerde, “Formal specification: A roadmap,” in Proceedings of the Conference on The Future of Software Engineering, 2000, pp. 147–159. [24] E. A. Emerson, “Temporal and modal logic,” in Formal Models and Semantics. Elsevier, 1990, pp. 995–1072. [25] T. Nummenmaa, A. Tiensuu, E. Berki, T. Mikkonen, J. Kuittinen, and A. Kultima, “Supporting agile development by facilitating natural user

interaction with executable formal specifications,” SIGSOFT Software Engineering Notes, vol. 36, no. 4, pp. 1–10, Aug. 2011. [26] “Inverted Pendulum,” https://www.cs.bgu.ac.il/∼inhs122/wiki.files/hybrid-automata3.pdf, accessed: 2022-2-22. [27] J. Duran, “What is a simulation model?” Minds and Machines, 03 2020. [28] G. Dubois, Modeling and Simulation: Challenges and Best Practices for Industry. CRC Press, 2018. [29] M. Broy, B. Jonsson, J.-P. Katoen, M. Leucker, and A. Pretschner, Part III. Model-Based Test Case Generation. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005, pp. 277–279. [30] “Matlab website,” https://www.mathworks.com/products/matlab.html, accessed: 2020-8-24. [31] “LabView website,” https://www.ni.com/en-us/shop/labview.html, accessed: 2020-8-24. [32] J. C. Jensen, D. H. Chang, and E. A. Lee, “A model-based design methodology for cyber-physical systems,” in 2011 7th International Wireless Communications and Mobile Computing Conference. IEEE, 2011, pp. 1666–1671. [33] R. Matinnejad, S. Nejati, L. Briand, T. Bruckmann, and C. Poull, “Search-based automated testing of continuous controllers: Framework, tool support, and case studies,” Information and Software Technology, vol. 57, pp. 705 – 722, 2015. [34] O. Boubaker, “The inverted pendulum: A fundamental benchmark in control theory and robotics,” in Proceedings of the International Conference on Education and e-Learning Innovations, 2012, pp. 1–6. [35] Z. Sadri-Moshkenani, J. Bradley, and G. Rothermel, “Survey on test case generation, selection and prioritization for cyber-physical systems,” Software Testing, Verification and Reliability, vol. 32, no. 1, p. e1794, 2022. [36] T. A. Henzinger, “What’s decidable about hybrid automata?” Journal of Computing Systems Science, vol. 57, no. 1, p. 94–124, aug 1998. [37] “Crusie Control,” https://ctms.engin.umich.edu/CTMS/index.php? example=CruiseControl&section=SystemModeling, accessed: 2022-1-19. [38] “Inverted Pendulum,” https://ctms.engin.umich.edu/CTMS/index.php? example=InvertedPendulum&section=SystemModeling, accessed: 2022-1-19. [39] M. Khazaee, M. Sadedel, and A. Davarpanah, “Behavior-based navigation of an autonomous hexapod robot using a hybrid automaton,” Journal of Intelligent Robotics Systems, no. 2, jun 2021. [40] A. Fehnker and F. Ivančić, “Benchmarks for hybrid systems verification,” in Hybrid Systems: Computation and Control, R. Alur and G. J. Pappas, Eds., 2004, pp. 326–341. [41] Qianchuan Zhao, B. H. Krogh, and P. Hubbard, “Generating test inputs for embedded control systems,” IEEE Control Systems Magazine, vol. 23, no. 4, pp. 49–57, 2003. [42] “Hybrid Model for Automatic Transmission and Thermostat1,” https://web.ece.ucsb.edu/∼hespanha/ece229/Lectures/Lecture2.pdf, accessed: 2023-1-11. [43] “Hybrid Model for Automatic Transmission and Thermostat2,” https://people.kth.se/∼kallej/papers/eolss 6.43.28.1.pdf, accessed: 2023-1-11. [44] R. Matinnejad, S. Nejati, L. C. Briand, and T. Bruckmann, “Automated test suite generation for time-continuous simulink models,” in Proceedings of the International Conference on Software Engineering, 2016, pp. 595–606. [45] Y. Chen, A. Yildiz, D. Marinov, and R. Jabbarvand, “Transforming test suites into croissants,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2023. New York, NY, USA: Association for Computing Machinery, 2023, p. 1080–1092. [Online]. Available: https://doi.org/10.1145/3597926.3598119 [46] M. Cordy, R. Rwemalika, A. Franci, M. Papadakis, and M. Harman, “Flakime: laboratory-controlled test flakiness impact assessment,” in Proceedings of the 44th International Conference on Software Engineering, ser. ICSE ’22. New York, NY, USA: Association for Computing Machinery, 2022, p. 982–994. [Online]. Available: https://doi.org/10.1145/3510003.3510194 [47] S. Habchi, M. Cordy, M. Papadakis, and Y. L. Traon, “On the use of mutation in injecting test order-dependency,” 2021. [48] M. Gruber and G. Fraser, “Flapy: Mining flaky python tests at scale,” ser. ICSE ’23. IEEE Press, 2023, p. 127–131. [Online]. Available: https://doi.org/10.1109/ICSE-Companion58688.2023.00039

20

[49] W. Lam, S. Winter, A. Astorga, V. Stodden, and D. Marinov, “Understanding reproducibility and characteristics of flaky tests through test reruns in Java projects,” in ISSRE 2020: 31st IEEE International Conference on Software Reliability Engineering, Virtual Event, October 2020, pp. 403–413. [50] T. M. King, D. Santiago, J. Phillips, and P. J. Clarke, “Towards a bayesian network model for predicting flaky automated tests,” 2018 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C), pp. 100–107, 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:52005045 [51] W. Lam, P. Godefroid, S. Nath, A. Santhiar, and S. Thummalapenta, “Root causing flaky tests in a large-scale industrial setting,” ser. ISSTA 2019. New York, NY, USA: Association for Computing Machinery, 2019, p. 101–111. [Online]. Available: https://doi.org/10.1145/3293882.3330570 [52] G. H. L. Pinto, B. Miranda, S. Dissanayake, C. Treude, A. Bertolino, and M. d’Amorim, “What is the vocabulary of flaky tests?” 2020 IEEE/ACM 17th International Conference on Mining Software Repositories (MSR), pp. 492–502, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:219165666 [53] F. Zampetti, D. Tamburri, S. Panichella, A. Panichella, G. Canfora, and M. Di Penta, “Continuous integration and delivery practices for cyber-physical systems: An interview-based study,” vol. 32, no. 3, apr 2023. [Online]. Available: https://doi.org/10.1145/3571854 [54] P. E. Strandberg, T. J. Ostrand, E. J. Weyuker, W. Afzal, and D. Sundmark, “Intermittently failing tests in the embedded systems domain,” in Proceedings of the 29th ACM SIGSOFT International Symposium on Software Testing and Analysis, ser. ISSTA 2020. New York, NY, USA: Association for Computing Machinery, 2020, p. 337–348. [Online]. Available: https://doi.org/10.1145/3395363.3397359

Related documents

• Statistical Scanner Model Validation#493192
• A Dataset of Reproducible Flaky-Test Failures#216895
• Software Testing in Computable Analysis#442730
• Going in the Right Direction: A Tutorial to Directional Hypothesis Testing Using the BFpack Module in JASP#31064
• Quasilinear Equivalence Checking for Detector Error Models#271900