Failure-Based Testing for Deep Reinforcement Learning Agents
arXiv:2606.31372v1 [cs.SE] 30 Jun 2026
WEIBIN LIN, Beihang University, China JIANGTAO MENG, Beihang University, China ZHENG ZHENG∗ , Beihang University, China Deep Reinforcement Learning (DRL) agents have been widely adopted across diverse domains to address challenging decision-making problems, such as autonomous driving and robotic control. Given that many of these applications are safety- and security-critical, rigorous testing of DRL agents is indispensable. Existing testing methods are typically guided by reward signals to detect failures. However, for well-trained agents, whose performance approaches optimal levels in standard operating conditions, reward signals remain generally high, making current methods ineffective at uncovering critical failures. To address these challenges, we propose a novel failure-based method that leverages task-induced failure insights to enhance failure detection capability while reducing the number of tests required. Since DRL agents are inherently designed with human-defined tasks, they provide valuable cues about task difficulty. Intuitively, a DRL agent is more likely to fail when confronted with a more difficult task; therefore, PRT prioritizes these tasks. Building on this foundation, we propose Prior Random Testing, a black-box failure-based testing method that enables targeted prioritization while preserving the diversity of generated test cases. Guided by task-induced failure insights, PRT prioritizes failure-prone regions of the input domain, thereby facilitating efficient failure detection. PRT is evaluated on four widely used benchmarks and compared with different state-of-the-art methods including fuzzing, search-based and generative-based methods. PRT ranks among the top performers in terms of both the cost of finding the first failure and the diversity of test cases. Notably, compared to random testing, PRT achieves better diversity and reduces the testing cost by over 50%. CCS Concepts: • Software and its engineering → Software testing and debugging. Additional Key Words and Phrases: Software testing, failure-based testing, deep reinforcement learning ACM Reference Format: Weibin Lin, Jiangtao Meng, and Zheng Zheng. 2026. Failure-Based Testing for Deep Reinforcement Learning Agents. Proc. ACM Softw. Eng. 3, FSE, Article FSE178 (July 2026), 22 pages. https://doi.org/10.1145/3808185
1
Introduction
Artificial intelligence (AI) has shown impressive performance in solving sequential decision-making tasks using deep reinforcement learning (DRL) techniques. DRL agents are widely employed in complex tasks such as autonomous driving [17], robot control [28], traffic control [40], and gaming [26, 32]. Despite these advances, the reliability of DRL agents remains uncertain [41]. Wrong decisions made by agents could lead to catastrophic consequences, particularly in safety-critical ∗ Corresponding author
Authors’ Contact Information: Weibin Lin, Beihang University, Beijing, China, [email protected]; Jiangtao Meng, Beihang University, Beijing, China, [email protected]; Zheng Zheng, Beihang University, Beijing, China, Zhengz@buaa. edu.cn.
This work is licensed under a Creative Commons Attribution 4.0 International License. © 2026 Copyright held by the owner/author(s). ACM 2994-970X/2026/7-ARTFSE178 https://doi.org/10.1145/3808185
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:2
Weibin Lin, Jiangtao Meng, and Zheng Zheng
domains [10]. This growing reliance on DRL agents underscores the urgent need to rigorously test these agents to ensure their quality. Real-world incidents highlight this necessity. For instance, a Tesla autonomous taxi was recently involved in its first recorded accident, colliding with a parked car [1]. Although commercial products are generally released by well-trained individuals, even rare failures can be disastrous. Therefore, efficiently testing almost flawless DRL agents is particularly important. Existing methods that can be applied to test different types of DRL agents are typically guided by reward signals [16, 24, 29]. Reward-based guidance exploits the property that low rewards are often correlated with failures. However, when agents demonstrate well-trained performance, rewards are consistently high, offering little useful signal. For example, in the Cart Pole environment [3], all successful test cases output exactly the same reward, providing no distinction for guiding testing. When reward signals provide limited guidance, failure-prone regions of the input domain can often be identified more directly. Since DRL agents are inherently task-driven and defined by human-specified oracles, it is possible to distinguish between easier and more difficult tasks, where the latter are more likely to induce failures. We define this intuition as task-induced failure insights, referring to the identification of input domain locations corresponding to the most difficult tasks within a given environment. A natural concern, however, is whether testing only the most difficult tasks suffices. Eniser et al. [12] show that an agent may succeed in a difficult task yet fail in an easier one. This observation underscores broad exploration: when the location of failure-prone regions is unknown a priori, evenly distributed test cases over the input space become an effective surrogate strategy for failure discovery, serving as a principled fallback when testing the most difficult tasks fails to expose failures. This naturally connects to the idea of failure-based testing, first introduced by Chen et al. [6]. Failure-based methods select test cases based on prior knowledge of failure patterns, including geometric shapes, sizes, or locations of failure regions [6]. Adaptive Random Testing (ART) [7] is a representative example that assumes that failure-causing test cases form contiguous regions, while passing test cases also exhibit clustering behavior. Therefore, given a set of previously executed test cases that have not exposed failures, selecting new test cases farther from these existing ones increases the likelihood of discovering failures [6]. While preserving this sparsity-driven exploration strategy, we further leverage failure location information to guide failure-based testing. Building on this, we propose Priority Random Testing (PRT), a black-box failure-based test case generation method that enables prioritization with uniformity. Before testing, PRT needs to design the hyperparameters in three steps: (1) Failure-prone Region: For every dimension of the input domain, identify the failure-prone value for prior testing, here corresponding to the most difficult tasks. (2) Mapping F : Design a mapping F (·) for every dimension, mapping the boundary to the failure-prone value. (3) Confidence 𝜆: Define the certainty level for that failure-prone value. A larger value of 𝜆 indicates a higher prior confidence in the failure-prone designation. PRT analytically generates new test cases within the sparsest regions of existing test cases to ensure uniformity, naturally resulting in a default priority on boundary exploration. For example, in one-dimensional input domain, regardless of the location of the first point, the farthest point from it (i.e., the sparsest location) necessarily lies at the boundary. We briefly introduce the workflow of PRT as follows. Assume that we already have a test case set C = {𝜀 1, 𝜀 2, · · · , 𝜀𝑛 }. When generating a new test case 𝜀𝑛+1 , PRT decomposes the problem across dimensions and generates values dimension by dimension. It first computes the largest interval 𝐸𝑖 of each dimension 𝑖 in the set C. It then selects the sparsest dimension 𝑘 = arg max𝑖 (𝐸𝑖 ) and 𝑘 generates the corresponding solution 𝜀𝑛+1 (the optimal solution for the 𝑘-dimensional largest interval 𝐸𝑘 ). We term this process Dimension Reduction. Next, PRT filters existing test cases to a
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:3
𝑘 }. This step, is referred to as Local Recombination. By alternating set C ′ = {𝑥 ∈ C | 𝑥 𝑘 close to 𝜀𝑛+1 Dimension Reduction and Local Recombination, a new test case 𝜀𝑛+1 is obtained. In addition, we incorporate a mapping F to shift boundary regions into other areas of the domain. Though PRT prioritizes testing the boundary by default, mapping F enables customized prioritization of other regions according to different failure insights. We implement PRT with a time complexity of approximately 𝑂 (𝑀𝑁 2 ) for generating 𝑁 test cases in an 𝑀-dimensional domain. In summary, this paper makes the following contributions:
• We propose a failure-based method, PRT, with a time complexity 𝑂 (𝑀𝑁 2 ), capable of leveraging the task-induced failure insights on DRL agents to efficiently find failures. • We evaluate PRT against four state-of-the-art (SOTA) methods—including fuzzing, search-based, and generative-based methods—on three representative testing subjects with distinct reward structures. PRT consistently ranks among the top performers in terms of both effectiveness and efficiency for detecting the first failure while achieving the best diversity of test cases. • We further empirically investigate the failure patterns of different DRL agents to evaluate the effectiveness and generalizability of PRT. Our findings reveal two main types of patterns: blockshaped and point-shaped. PRT addresses these through task-induced failure insights and uniform distribution, respectively. 2 2.1
Background Preliminaries
Markov Decision Process (MDP). MDP is a mathematical framework used for modeling decisionmaking, which is composed of a tuple < 𝑆, 𝐴, T , R, 𝜋 >. 𝑆 is a set of all possible states during MDP. 𝐴 is a set of actions available to the decision-making agent. T (𝑠𝑡 +1 |𝑠𝑡 , 𝑎𝑡 ) → [0, 1] defines the probability from the present state 𝑠𝑡 to the next state for every 𝑠𝑡 +1 ∈ 𝑆 with action 𝑎𝑡 . The reward function R (𝑠𝑡 , 𝑎𝑡 ) → R is used to evaluate the quality of an action 𝑎𝑡 under a specified state 𝑠𝑡 . Specifically, an episode reward means the cumulative reward from the starting state 𝑠 0 to the end of the episode. 𝜋 (𝑎𝑡 |𝑠𝑡 ) → [0, 1] gives the probability of taking action 𝑎𝑡 under the state 𝑠𝑡 , representing the behavior of an agent. Deep Reinforcement Learning (DRL). Based on the mathematical MDP model, DRL gives a solution of 𝜋 (𝑎𝑡 |𝑠𝑡 ) [2]. Unlike deep learning, which directly labels an action as right or wrong, RL only scores an action with a reward value and lets the agents learn how to obtain the cumulative reward in a task as high as possible. The environment in RL refers to the tuple < 𝑆, 𝐴, T , R >, and the agent refers to the policy 𝜋. The environment is responsible for giving the observed state and the corresponding reward to the agent, and the agent gives an action to the environment for the next state transition. As the deep neural network (DNN) demonstrates excellent performance in many areas, DRL employs DNN as the policy function 𝜋 (𝑎𝑡 |𝑠𝑡 ). Popular DRL techniques include DQN [37], PPO [31], TQC [19], and so on. Test Cases for DRL. Though our testing subject, the DRL agent, takes states as inputs, our test cases are defined as the initial configurations of the environment. Once the agent takes the first action, we will not change anything, so as to ensure the authenticity of the scenario. Input Domain. The input domain is defined as the set of all feasible test cases. It is emphasized that the test case is not the input 𝑠 for testing subjects. Test Oracle. We give a setting of the environment as the test case at the beginning. Then the agent interacts with the environment. If the state 𝑠𝑡 satisfies some conditions (depended on the environment), the test case fails or passes at the 𝑡 step. Failure Pattern. We refer to the failure pattern as the characteristics of the failure test case distribution in the input domain, especially the geometric ones after visualization. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:4
2.2
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Task-Induced Failure Insights on DRL Agents
A concern is whether task-induced failure insights can generalize across different DRL agents. We demonstrate that such insights are broadly applicable because they originate from properties shared by all DRL agents. Specifically, every DRL agent is designed to solve a man-made task, and these tasks inherently contain cues about difficulty [34]. The core intuition is that agents are more likely to fail in more difficult tasks, with difficulty intuited directly from the task definition. For example, in games, difficulty can be increased by adding more obstacles or enemies, while in control problems, test cases can be designed farther from the stable state. Amal et al. make a comprehensive review on testing DRL agents and summarizes the testing environments [33]. We select the environments tested by more than two papers and annotate their tasks along with the failure insights on the failure patterns in Table. 1, so as to show that we can easily locate the failure-prone regions of the input domain. We also attach the cited numbers counted by Amal et al. [33]. Table 1. Task-induced insights about failures in different environments [33]
Environment Frozen Lake
Task Walk from start to goal without falling into any holes Mountain Car2 Accelerate a car from the bottom of a sinusoidal valley to the top of the right hill Cart Pole2 Balance a pole attached by an un-actuated joint to a cart Lunar Lander2 Safely land on the middle pad from the sky Super Mario Arrive at the destination without collision with the enemies or falling into the pits Coop Navi Several agents need to cover the same num(Simple Spread) ber of landmarks while avoiding collisions Bipedal Walker Balance a 4-joint robot to step over obstacles CARLA Control a car to arrive at the destination without collision Navigate a car to pick up a passenger and Taxi drop him off at the destination
Insights1 More holes
Papers 3
Starting from the middle with no velocity Higher initial velocity
12
Higher initial velocity More enemies and pits
7 3
Agents are closer to each other Larger size of obstacles More vehicles on roads
3
The locations are far from each other
8
4 4 3
1 Insights indicate how to design a test case with a higher failure probability. 2 For these three environments, we introduce more details in Sec. 4.2.
3
Approach
PRT consists of two cooperative mechanisms: dimension reduction and local recombination. Dimensionality reduction provides a global view that identifies sparse regions among the candidate set, while local recombination examines which region is still dense and offer that as the new candidate set. Each time PRT generates a new test case 𝜀, it performs dimension reduction by splitting the test case space along individual dimensions: 𝜀 = (𝜀 1, 𝜀 2, · · · , 𝜀𝑚 ). Each time, PRT selects the sparsest dimension (assumed to be the 𝑘 1 -th dimension) from the candidate set C to generate the corresponding optimal solution 𝑣𝑎𝑙𝑢𝑒 so that 𝜀 𝑘1 = 𝑣𝑎𝑙𝑢𝑒. Then PRT reconstructs a candidate set C1 ⊂ C by only including the test cases in C close to 𝜀 𝑘1 in the 𝑘 1 -th dimension (local recombination). Next round we will have 𝜀 𝑘2 assigned to the 𝑣𝑎𝑙𝑢𝑒 corresponding to the sparsest 𝑘 2 -th dimension, and a smaller candidate set C2 ⊂ C1 , and so on up to 𝑘𝑚 , where {𝑘 1, 𝑘 2, · · · , 𝑘𝑚 } = {1, 2, · · · , 𝑚}. At last we can obtain a new test case 𝜀. 𝐹 𝑃 (𝐼𝑛𝐷) = {𝑓 𝑝 1, 𝑓 𝑝 2, · · · , 𝑓 𝑝𝑚 } (1) Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:5
By default, PRT prioritizes testing the boundary. However, the region we want to prioritize (corresponding to the most difficult tasks) does not always lie in the boundary. At the beginning of applying PRT, we need to identify the failure-prone (𝐹 𝑃) region of the input domain (𝐼𝑛𝐷) as Eq. 1. More specifically, for every dimension of the input domain 𝐼𝑛𝐷 𝑖 , we need to specify the failureprone value 𝑓 𝑝 𝑖 . By default, the value is set as the boundary, both the left side and the right side. We can change PRT’s exploring strategy by a well-designed mapping F , shifting the prior testing region. We regard F to be a hyperparameter as it is manually defined before testing. After setting the other hyperparameters, we can begin testing with PRT. 3.1
Approach Overview
Alg. 1 shows the workflow of PRT. It takes the number of test cases 𝑁 and test case dimension 𝑀 as inputs along with hyperparameters F , 𝜆 ∈ [1, +∞), and gives the generated test case set T as the output. Line 2 randomly generates the first test case 𝜀 as the initial reference. Line 3 constructs the output test case set T. The outer while loop, lines 4–17, generates a new test case 𝜀 at each round, where line 16 adds it to the output test case set T. Line 5 creates a candidate set, initialized as the existing test case set T. Line 6 creates an empty container for the generating test case 𝜀 and Line 7 assigns an adaptive hyperparameter for the inner while loop. Line 8 creates an index set for indicating which dimensional value of 𝜀 has not been generated. The inner while loop, lines 9–15, generates a dimensional value for 𝜀 at each round. Each round line 10 picks the sparsest dimension 𝑘 (i.e., with the largest interval) from the candidate set C and generates the corresponding value resulting from the largest interval, which is assigned to 𝛬[𝑘]-dimensional of 𝜀 (i.e., 𝜀 𝛬[𝑘 ] ) at line 11. Line 12 uses such information to select test cases similar to 𝜀 𝛬[𝑘 ] , where the new candidate set C is a subset of the old one. As we have generated 𝑘-dimensional data, line 13 reduces that dimension of C and the index set 𝛬 removes that dimension (e.g., remove the second dimension of {(1, 2, 3), (4, 5, 6)} → {(1, 3), (4, 6)}). Until all dimensional data have been generated, we can obtain a new test case 𝜀. Lines 18–20 make a mapping F on every test case in T. Algorithm 1 Prior Random Testing Input: expected number of test cases 𝑁 , test case dimension 𝑀, mapping F , confidence 𝜆 Output: test case set T 1: function PRT(𝑁 , 𝑀, F , 𝜆) 2: 𝜀𝑟𝑎𝑛𝑑𝑜𝑚 ← randomly generate an M-dimensional test case 3: T ← {𝜀𝑟𝑎𝑛𝑑𝑜𝑚 } 4: while |T| < 𝑁 do ⊲ |T|: the number of T 5: C ← 𝑐𝑜𝑝𝑦 (T) 6: 𝜀 ← an M-dimensional empty array ⊲ the next test case 7: 𝜏 ← (|T| + 1) −1/𝑀 ⊲ an adpative hyperparameter for LocalRecombination 8: 𝛬 ← {1, 2, · · · , 𝑀 } ⊲ a dimensional index set 9: while 𝛬 is not empty do 10: 𝑘, 𝑜𝑝𝑡𝑖𝑚𝑎𝑙_𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 ← DimensionReduction(C, 𝑀, 𝜆) 11: 𝜀 𝛬[𝑘 ] ← 𝑜𝑝𝑡𝑖𝑚𝑎𝑙_𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 ⊲ 𝛬[𝑘]: the 𝑘-th element of 𝛬 12: C ← LocalRecombination(C, 𝑘, 𝑜𝑝𝑡𝑖𝑚𝑎𝑙_𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛, 𝜏) 13: delete 𝑘-dimensional data of C 14: 𝛬 ← 𝛬\{𝑘 } ⊲ remove 𝑘 from 𝛬 15: end while Ð 16: T ← T {𝜀} 17: end while 18: for 𝑘 = 1 → |T| do 19: T[𝑘] ← F (T[𝑘]) 20: end for 21: return T 22: end function Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:6
3.2
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Dimension Reduction
As PRT generates a new test case dimension by dimension, dimension reduction compares the largest intervals of every dimension in the candidate set C, generates the corresponding value. More specifically, given a 𝑀-dimensional test case set C = {𝜀 1, 𝜀 2, · · · , 𝜀𝑛 }, the largest interval of 𝑖-th dimension is defined as 𝐸𝑖 : max
𝐸𝑖
𝑥 ∈InD𝑖
s.t.
(2) 𝐸𝑖 ≤ |𝑥 − 𝜀𝑘𝑖 |, 𝑘 = 1, 2, . . . , 𝑛
where 1 ≤ 𝑖 ≤ 𝑀, 𝐼𝑛𝐷 𝑖 is the 𝑖-dimensional input domain, 𝜀𝑘𝑖 is the 𝑖-dimensional value of 𝜀𝑘 . And the optimal solution 𝑥 is referred to the corresponding solution for 𝐸𝑖 . Idea. Given 𝑛 points in high-dimensional domain, finding the farthest point from them is an NP-hard (Nondeterministic Polynomial time) problem. However, the problem is easy to be solved in one-dimensional domain. For example, assuming the input domain is [0, 1], we have four existing points (colored blue): 𝑝 1 = 0, 𝑝 2 = 0.1, 𝑝 3 = 0.3, 𝑝 4 = 0.7. We can compute their interval as Eq. 3. (𝑝 2, 𝑝 3, 𝑝 4 ) − (𝑝 1, 𝑝 2, 𝑝 3 ) = (0.1, 0.2, 0.4)
(3)
It is easy to find a large interval between 𝑝 3 and 𝑝 4 . The corresponding solution is 𝑝 = (𝑝 3 + 𝑝 4 )/2 = 0.5 and the interval is (𝑝 4 − 𝑝 3 )/2 = 0.2. However, by taking the infimum and the supremum into account, we find that 𝑝 = 1 is the optimal solution because of its larger interval 1 − 𝑝 4 = 0.3 > (𝑝 4 − 𝑝 3 )/2 = 0.2. So the new generated point for this round is 𝑝 = 1 (colored green) and the maximal interval is 0.3. Similarly, we can get the next generated point 𝑝 = 0.5 (colored orange) and its corresponding interval 0.2. Furthermore, the sparsest point is treated as a probabilistic rather than deterministic solution, as it may not always be the optimal choice. We design a probabilistic function as Eq. 4, where 𝑖𝑛𝑓 and 𝑠𝑢𝑝 are the infimum and the supremum respectively, 𝜆 ≥ 1 is a hyperparameter, and 𝜃 ∈ [0, 1] follows a uniform distribution. Eq. 4 makes the generated point more likely close to (𝑖𝑛𝑓 + 𝑠𝑢𝑝)/2 and 𝜆 controls the probability. The larger 𝜆 is, the more likely 𝑓 (𝜃 | 𝑖𝑛𝑓 , 𝑠𝑢𝑝, 𝜆) is close to (𝑖𝑛𝑓 + 𝑠𝑢𝑝)/2. When 𝜆 = 1, 𝑓 (𝜃 | 𝑖𝑛𝑓 , 𝑠𝑢𝑝, 𝜆) follows a uniform distribution. ( 𝑖𝑛𝑓 +𝑠𝑢𝑝 𝑠𝑢𝑝 −𝑖𝑛𝑓 − (1 − 2𝜃 ) 𝜆 · , 0 ≤ 𝜃 ≤ 0.5 2 2 𝑓 (𝜃 | 𝑖𝑛𝑓 , 𝑠𝑢𝑝, 𝜆) = 𝑖𝑛𝑓 +𝑠𝑢𝑝 (4) 𝑠𝑢𝑝 −𝑖𝑛𝑓 𝜆 · + (2𝜃 − 1) , 0.5 < 𝜃 ≤ 1 2 2 Here (𝑖𝑛𝑓 + 𝑠𝑢𝑝)/2 is the middle point. As 𝜃 follows a uniform distribution 𝑈 (0, 1), the generated point moves left when 𝜃 < 0.5, and moves right when 𝜃 > 0.5. |1 − 2𝜃 | is an item ranging from 0 to 1. The larger 𝜆, the smaller |1 − 2𝜃 |𝜆 and the more likely 𝑓 (𝜃 | 𝑖𝑛𝑓 , 𝑠𝑢𝑝, 𝜆) close to the middle point. We refer 𝜆 as the confidence level. It is correlated with the probability function of new generating points. More specifically, in the case of left boundary, 𝜃 is required to follow a uniform distribution 𝑈 (0.5, 1). The larger 𝜆, the more likely new points close to the boundary. And so like for the right boundary. Design. Algo. 2 formulates the workflow to find the sparsest dimension and the corresponding value. Lines 5–30 record the maximal interval and the corresponding solution dimension by dimension. Lines 7–8 aim to only consider the boundary (lines 18–27). Lines 10–12 compute the intervals, and lines 13–14 record the largest one. Line 15 samples a value from the uniform distribution between 0 and 1. Line 16 computes the generated value according to Eq. 4. Lines 18–27 take a similar method to deal with the boundary case, where 𝑖𝑛𝑓 𝑖 and 𝑠𝑢𝑝 𝑖 are the infimum and the supremum of 𝑖-dimensional input domain. Lines 28–29 record the analytical result of the 𝑖-th dimension after normalization. Line 31 selects the dimension with the maximal interval. Line 32 obtains the solution by the dimension number. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:7
Algorithm 2 Dimension Reduction Input: test case set C, dimension 𝑀, confidence 𝜆 Output: the sparsest dimension 𝑘, the corresponding solution 𝑣𝑎𝑙𝑢𝑒 1: function DimensionReduction(C, 𝑀, 𝜆) 2: 𝑐 1, 𝑐 2, · · · , 𝑐𝑛 ← all test cases in C 3: 𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠 ← ∅ 4: 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛𝑠 ← ∅ 5: for 𝑖 = 1 → 𝑀 do 6: 𝑝 1, 𝑝 2, · · · , 𝑝𝑛 ← 𝑠𝑜𝑟𝑡 (𝑐 𝑖1, 𝑐 𝑖2, · · · , 𝑐𝑛𝑖 ) ⊲ in ascending order 7: if 𝑛 == 1 then 8: 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 ← 0 9: else 10: for 𝑗 = 1 → 𝑛 − 1 do 11: 𝑑 𝑗 ← 𝑝 𝑗+1 − 𝑝 𝑗 12: end for 13: 𝑘 ← arg max 𝑗 (𝑑 𝑗 ) 14: 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 ← (𝑝𝑘+1 − 𝑝𝑘 )/2 15: 𝜃 ← 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚𝑆𝑎𝑚𝑝𝑙𝑒 (0, 1) ⊲ Sample a value from a uniform distribution [0, 1] 16: 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 ← 𝑓 (𝜃 | 𝑝𝑘 , 𝑝𝑘+1, 𝜆) 17: end if 18: if 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 < 𝑝 1 − 𝑖𝑛𝑓 𝑖 then ⊲ 𝑖𝑛𝑓 𝑖 : the infimum of the 𝑖-th dimension 𝑖 19: 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 ← 𝑝 1 − 𝑖𝑛𝑓 20: 𝜃 ← 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚𝑆𝑎𝑚𝑝𝑙𝑒 (0.5, 1) 21: 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 ← 𝑓 (𝜃 | 2 · 𝑖𝑛𝑓 𝑖 − 𝑝 1, 𝑝 1, 𝜆) 22: end if 23: if 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 < 𝑠𝑢𝑝 𝑖 − 𝑝𝑛 then ⊲ 𝑠𝑢𝑝 𝑖 : the supremum of the 𝑖-th dimension 𝑖 24: 𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙 ← 𝑠𝑢𝑝 − 𝑝𝑛 25: 𝜃 ← 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚𝑆𝑎𝑚𝑝𝑙𝑒 (0, 0.5) 26: 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛 ← 𝑓 (𝜃 | 𝑝𝑛 , 2 · 𝑠𝑢𝑝 𝑖 − 𝑝𝑛 , 𝜆) 27: end if Ð 28: 𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠 ← 𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠 Ð{𝑚𝑎𝑥_𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙/(𝑠𝑢𝑝 𝑖 − 𝑖𝑛𝑓 𝑖 )} 29: 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛𝑠 ← 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛𝑠 {𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛} 30: end for 31: 𝑘 ← arg max(𝑖𝑛𝑡𝑒𝑟𝑣𝑎𝑙𝑠) 32: 𝑣𝑎𝑙𝑢𝑒 ← 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛𝑠 [𝑘] 33: return 𝑘, 𝑣𝑎𝑙𝑢𝑒 34: end function 3.3
Local Recombination
Idea. Algo. 2 aims to identify the largest interval containing a given number of points across different dimensions. However, this task becomes challenging when the number of points is large and the dimensionality is high. Moreover, Algo. 2 fails to preserve the relationships among test cases across dimensions. Therefore, we aim to reconstruct the candidate set C. And we regard that the generation process should focus on points close to the newly generated 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛. Design. Eq. 5 defines the local recombination strategy, where 𝑘 1, 𝑘 2, · · · , 𝑘𝑚 are the generated dimensions of 𝜀, 𝑄𝜏 denotes 𝜏-quantile. 𝑚 n o Ù C ′ = 𝑅𝑒𝑐𝑜𝑚𝑏𝑖𝑛𝑒 (T|𝜀, 𝜏) = 𝑥 ∈ T |𝑥 𝑘𝑖 − 𝜀 𝑘𝑖 | ≤ 𝑄𝜏 { |𝑦𝑘𝑖 − 𝜀 𝑘𝑖 | : 𝑦 ∈ T } (5) 𝑖=1
If 𝑘𝑚+1 is newly generated dimension and C is the last recombination candidate set, Eq. 5 can be simplifed as only computing the 𝑘𝑚+1 -th dimension: n o C ′ = 𝑅𝑒𝑐𝑜𝑚𝑏𝑖𝑛𝑒 (C|𝜀, 𝜏) = 𝑥 ∈ C |𝑥 𝑘𝑚+1 − 𝜀 𝑘𝑚+1 | ≤ 𝑄𝜏 { |𝑦𝑘𝑚+1 − 𝜀 𝑘𝑚+1 | : 𝑦 ∈ C } (6) Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:8
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Algorithm 3 Local Recombination Input: test case set C, target dimension number 𝑘 and new value 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛, resolution 𝜏 Output: the recombined test case set C ′ 1: function LocalRecombination(C, 𝑘, 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛, 𝜏) 2: 𝐷𝑖𝑠𝑡 ← ∅ 3: for 𝑐 ∈ C do Ð 4: 𝐷𝑖𝑠𝑡 ← 𝐷𝑖𝑠𝑡 {|𝑐 𝑘 − 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛|} 5: end for 6: 𝑞𝜏 ← 𝑄𝜏 (𝐷𝑖𝑠𝑡) 7: C ′ ← 𝑐 ∈ C |𝑐 𝑘 − 𝑠𝑜𝑙𝑢𝑡𝑖𝑜𝑛| ≤ 𝑞𝜏 8: return C ′ 9: end function Algo. 3 formulates the local recombination process. Line 6 obtains the 𝜏-quantile of the distance set. A simple method is to sort the points and obtain the 𝜏 |𝐷𝑖𝑠𝑡 |-th point, whose time complexity is 𝑂 (𝑛 log 𝑛). Here we leverage Introselect algorithm to reduce the time complexity to 𝑂 (𝑛). There is a question on how to select the hyperparameter 𝜏. From Algo. 3, we can get |C| = ⌈𝜏 |C| ′ ⌉ ≈ 𝜏 |C| ′ . Initially in the inner while loop of Algo. 1, there are |T| points and the iteration will have 𝑀 times. By designing 𝑁 · 𝜏 𝑀 = 1, we have a recommended setting: 1
𝜏 = 𝑁−𝑀 3.4
(7)
Boundary Priority
In this section, we explain why PRT prioritizes boundary regions by default. First, Algo. 2 assigns a boundary interval that is twice as large as that of a middle region, prioritizing generating boundary points. For example, suppose the input domain is [0, 1] and the existing points are 0.3, 0.8. The interval corresponding to the candidate solution at 0.55 is only 0.25, whereas the interval at the boundary point 0 is 0.3. Thus, the next generated point will be 0. Moreover, Algo. 3 further reinforces boundary exploration across dimensions. For instance, assume the existing points lie on the boundary in the 𝑖-th dimension but in the middle region of the 𝑗-th dimension. Although the new point may be generated in the middle region along the 𝑖-th dimension, it will be positioned on the boundary along the 𝑗-th dimension. 3.5
Linear Mapping
We demonstrate that PRT generates evenly distributed test cases but prioritizes testing the boundary by default (F (𝑡) = 𝑡). However, sometimes we may expect PRT to prioritize other regions. Here we give an example of prioritizing the middle domain. Eq. 8 assumes 𝑡 is a scalar value with infimum 𝑖𝑛𝑓 and supermum 𝑠𝑢𝑝 and maps the boundary to the middle (𝑖𝑛𝑓 + 𝑠𝑢𝑝)/2. For high-dimensional data, this process is repeated for every dimension. ( F1 (𝑡) =
3𝑖𝑛𝑓 +𝑠𝑢𝑝 − 𝑡, 2 𝑖𝑛𝑓 +3𝑠𝑢𝑝 − 𝑡, 2
𝑖𝑛𝑓 +𝑠𝑢𝑝
𝑖𝑛𝑓 ≤ 𝑡 ≤ 2 𝑖𝑛𝑓 +𝑠𝑢𝑝 < 𝑡 ≤ 𝑠𝑢𝑝 2
(8)
Note that we make a linear transformation to ensure the evenly distributed property. Other transformations can also be adopted for specific purposes. 3.6
Realization
If we follow the steps of Alg. 2 and Alg. 3, the time complexity is 𝑂 (𝑀𝑁 2 log 𝑁 ). We use some data structure techniques to accelerate the program. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:9
Assume that we want to generate 𝑁 𝑀-dimensional test cases and we already have 𝑘 test cases during the while loop in Alg. 1. First, an 𝑀 × 𝑁 matrix 𝐷 is used to store test cases. 𝑖th row records the 𝑖-dimensional data in an ascending order, despite the relationship between different dimensions. That makes the time complexity of Alg. 2 line 6 𝑂 (1) and that of the whole Alg. 2 𝑂 (𝑀𝑘). Secondly, use a 𝑀 × 𝑘 matrix 𝛼 to record which number of test case 𝐷 (𝑖, 𝑗) is from, and use a 𝑘 × 𝑀 matrix 𝛽 to record which index of 𝐷 the 𝑘-th test case is in. They satisfy 𝛼 (𝑚, 𝛽 ( 𝑗, 𝑚)) = 𝑗, ∀1 ≤ 𝑗 ≤ 𝑘, 1 ≤ 𝑚 ≤ 𝑀. That makes the time complexity of the whole Alg. 3 𝑂 (𝜏𝑘 (𝑀 −1)). At the next iteration, the number of test cases approximate 𝜏𝑘 and the time complexity of Algo. 2 and Algo. 3 will be 𝑂 (𝜏𝑘 (𝑀 − 1)) and 𝑂 (𝜏 2𝑘 (𝑀 − 2)) respectively. As the time complexity of Algo. 3 is lower than that of Algo. 2, the time complexity of Lines 8–14 in Algo. 1 is 𝑂 𝜏 𝑡 𝑘 (𝑀 − 𝑡) , where 𝑘 and 𝑡 implies the 𝑘-th outer while loop and the 𝑡-th inner while loop. By traversing 𝑘 from 1 to 𝑁 we can get the result 𝑂 (𝑀𝑁 2 ). 𝑁 𝑀 −1 ∑︁ ∑︁
𝑂 𝜏 𝑡 𝑘 (𝑀 − 𝑡)
𝜏=𝑘
𝑘=2 𝑡 =0
4
1 −𝑀
≈
𝑁 ∑︁
𝑂 (𝑀𝑘) = 𝑂 (𝑀𝑁 2 )
(9)
𝑘=2
Experimental Setup
4.1
Baselines
PRT is a black-box setting method and can be applied to test any type of DRL agents. Our baselines include SOTA fuzzing methods MDPFuzz [29] and CureFuzz [16], search-based method QD [24], generative-based method G-Model [20] as well as random testing (RT). For QD, we use the MapElites algorithmic version as it performs the best in most cases analyzed by the authors [24]. Note that though these methods are all black-box, most of them need preparation before testing. Besides, the information used for testing also varies. Table 2. Applicable conditions of different methods.
PRT (ours) MDPFuzz [29] CureFuzz [16] G-Model [20] QD [24] RT
Failure Insights √
Execution
× × × × ×
× √
× √ √
×
Model Training × √ √ √ × ×
State
Action
Reward
× √ √
× √ √
× √
× × × × √
×
×
×
× √
Tab. 2 summarizes their differences. PRT needs failure insights about the task. MDPFuzz, CureFuzz and QD need to execute a number of test cases before testing. MDPFuzz, CureFuzz and G-Model will train machine learning models before testing. Furthermore, these methods access different information during testing. MDPFuzz and CureFuzz require the executing data, including state and reward, while QD requires state, action and reward. For MDPFuzz [29] and CureFuzz [16], which require executing test cases before testing, we randomly generate 2000 passing test cases, where we choose 2000 to give the method enough information and the passing condition is their inner algorithmic step. Though QD [24] needs execution before testing, it incorporates the execution into the testing cost. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:10
4.2
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Testing Subjects
We select the three most popular environments in Tab. 1 and one high-dimensional environment for testing, which are all developed by OpenAI in Gymnasium platform [36]. The three most popular environments are also chosen because of the different reward patterns (RQ1 in Sec. 5.2, Fig. 2). We introduce the environments’ tasks, the test case definitions and the test oracles as follows. To comprehensively test different types of agents, we select the discrete version of Cart Pole and the continuous versions of Lunar Lander and Mountain Car.
(a) Cart Pole
(b) Lunar Lander
(c) Mountain Car
(d) Humanoid
Fig. 1. Different environment examples
4.2.1 Cart Pole. This environment corresponds to the version of the cart-pole problem described by Barto et al. [3]. In Fig. 1a, a pole is attached by an unactuated joint to a cart, which moves along a frictionless track. The test case is defined as its four initial states (𝑥, 𝑥, ¤ 𝜃, 𝜃¤): the cart’s position 𝜋 𝜋 𝑥 ∈ [−0.9, 0.9], the cart’s velocity 𝑥¤ ∈ [−0.3, 0.3], the pole’s angle 𝜃 ∈ [− 60 , 60 ] and the pole’s 𝜋 𝜋 angular velocity 𝜃¤ ∈ [− 90 , 90 ]. The agent can apply either a left or a right force to the cart. Once the pole’s angle is not within ±12◦ or the cart’s position is not within ±2.4, the task fails. If the agent fails at 𝑡 step, it will have a reward 𝑡. If the system keeps balance during 500 steps, we will terminate testing, give a reward 500 and regard the test case as passing. 4.2.2 Lunar Lander. This is a classic rocket trajectory optimization problem. In Fig. 1b, a lander descends from the middle sky towards a designated landing pad on the lunar surface. The test case is defined as the initial force (𝑓𝑥 , 𝑓𝑦 ) applied to the lander along 𝑥 direction 𝑓𝑥 ∈ [−1500, 1500] and 𝑦 direction 𝑓𝑦 ∈ [−1500, 1500]. The agent can make one of four discrete actions: do nothing, fire the left orientation engine, fire the right orientation engine, or fire the main engine. A task fails if the lander crashes (e.g., outside the landing pad or with excessive speed), while it succeeds if the lander safely lands on the middle pad. The reward is calculated according to the distance to the landing pad, the speed, the angle, and the leg contacts. A successful landing within the designated area yields a high reward of 100, while crashing results in a large negative reward of -100. 4.2.3 Mountain Car. This environment first appeared in Andrew Moore’s PhD thesis (1990) [27]. In Fig. 1c, a car is situated between two hills and must build momentum to reach the flag positioned at the top of the right hill. The test case is defined by its two initial states (𝑥, 𝑣): the car’s horizontal position 𝑥 ∈ [−1, 0] and velocity 𝑣 ∈ [−0.05, 0.05], where the position of the valley is − 𝜋6 ≈ −0.52 and the position of the flag is 0.5. The agent can apply one of three discrete actions: accelerate left, accelerate right, or do nothing. The agent receives a constant reward of −1 per step until termination. If the car reaches the flag at time step 𝑡, the task succeeds with a cumulative reward of −𝑡; but if 𝑡 > 200, the test case is regarded as a failure. 4.2.4 Humanoid-v4. This environment is a 3D bipedal robot simulation based on the work of Tassa et al. [35]. In Fig. 1d, a humanoid robot with a torso, a pair of arms and legs (each leg has three parts, each arm has two parts) is designed to simulate human locomotion. The robot starts in a standing pose, and the test case is defined as the bias added to the standing pose. The input domain Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:11
is [−0.01, 0.01] 47 , including 24 position values and 23 velocity values. The agent controls the robot by applying torques to its 17 hinge joints, including 3 for the abdomen, 3 for each hips, 1 for each knees, 2 for each arm’s shoulder, and 1 for each elbow, totally 3 + (3 + 1 + 2 + 1) × 2 = 17. The goal is to make the robot walk forward as fast as possible without falling. The task fails if the robot falls (the torso height drops below a threshold). At each time step, the agent receives a reward comprised of a forward velocity bonus, a healthy reward for staying upright, and penalties for excessive control forces. If the agent survives until the time horizon (1000 steps), the test case is regarded as passing. Validity of test cases. We regard that a test case 𝜀 is valid if and only if there exists a feasible action sequence enabling the agent to pass the test case. We have verified that the most difficult test cases (i.e., the most difficult tasks of specific environments) within the designed input domain can successfully be solved, which implies all failures are caused by incorrect software logic (i.e., incorrect agent actions) rather than flaws in the input domain design. In addition, as we change the initial state bound of Cart Pole, Lunar Lander and Mountain Car, we upload three oracle agents to our open-source repository for verifying that all failures can be solved. Task-induced failure insights. In the Cart Pole environment (Fig. 1a), the agent must balance the cart and the pole with (𝑥, 𝑥, ¤ 𝜃, 𝜃¤) around (0, 0, 0, 0), so the most difficult task corresponds to the 𝜋 𝜋 boundary (±0.9, ±0.3, ± 60 , ± 90 ). In the Lunar Lander environment (Fig. 1b), the lander is required to safely land on the middle pad, so the most difficult task also corresponds to the boundary (±1500, ±1500). In the Mountain Car environment (Fig. 1c), the car must climb the right hill to reach the flag within specific steps, so the most difficult task corresponds to the middle point (−0.5, 0). In the Humanoid environment (Fig. 1d), the robot needs to maintain the standing pose, so the most difficult task corresponds to the boundary (±0.01, ±0.01, · · · , ±0.01). 4.3
Hyperparameters
Mapping F . For Cart Pole, Lunar Lander and Humanoid, the mapping F does not change anything F (𝑡) = 𝑡. For the Mountain Car environment, we shift the boundary to the middle with Eq. 8 F1 . Confidence 𝜆. We set 𝜆 = 20. Under this setting, assuming the parameter 𝜃 follows a uniform distribution 𝜃 ∼ 𝑈 (0, 1), the deviation term (1 − 2𝜃 ) 𝜆 in Eq. 4 exceeds 10% with probability 10.9%, and exceeds 2% with probability 17.8%. 4.4
Implementation
Stochasticity of test case execution. Stochasticity is a common challenge in testing DRL agents. To minimize its impact on the outputs, we generate test cases using different random seeds but execute all test cases under a fixed random seed, which means the test case generation is nondeterministic but the test case execution is deterministic. Experimental platform. Our experimental platform is 24.04.1-Ubuntu, with hardwares consisting of an AMD Ryzen 9 9950x CPU and two 32 GB 5200 Hz RAMs. We do not use any GPU for testing. 5
Evaluation
In this section, we present three research questions (RQs) to study the effectiveness, efficiency and generalization of PRT. RQ1: How effective and efficient is PRT for detecting failures? We evaluate PRT against SOTA fuzzing, search-based, and generative methods on DRL agents with failure rates below 1% measured by random testing under 100,000 tests. The comparison focuses on the testing cost required to detect the first failure. In addition, we examine the conditions under which PRT outperforms the baselines, as well as those in which the baselines demonstrate superior performance. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:12
Weibin Lin, Jiangtao Meng, and Zheng Zheng
RQ2: How diverse are test cases generated by different methods? As we claim that PRT preserves the diversity of generated test cases while enabling prioritization, RQ2 evaluates the extent of this diversity, particularly in comparison with the baselines. In addition, we visualize the distributions of all generated test cases to provide further insights into the strengths and limitations of different algorithms beyond scalar diversity metrics. RQ3: What are the failure patterns of different agents, and how does PRT address them? As a failure-based testing method, the effectiveness of PRT is strongly correlated with the failure pattern of testing subjects. To this end, we train agents with different algorithms and analyze their failure patterns. We then investigate why PRT can effectively detect failures across different DRL agents. 5.1
Diversity Metric
We quantify the diversity of generated test cases using differential entropy, which measures the dispersion of samples in continuous state spaces. Given a continuous random variable 𝑋 ∈ R𝑑 with probability density function 𝑝 (𝑥), its differential entropy is defined as: ∫ 𝐻 (𝑋 ) = − 𝑝 (𝑥) log 𝑝 (𝑥) 𝑑𝑥 (10) Differential entropy characterizes the effective volume occupied by a distribution and increases as samples become more dispersed within the same support [8]. Under identical dimensionality and normalization, higher entropy indicates broader exploration of the state space rather than closeness to a specific target distribution. Since the underlying distribution of test cases is unknown and may exhibit complex, non-Gaussian structures, we adopt a non-parametric k-nearest neighbor (kNN) entropy estimator. Specifically, we use the estimator proposed by Kozachenko and Leonenko [18]. Given a set of 𝑛 samples 𝑥𝑖 𝑛𝑖=1 ⊂ R𝑑 , the entropy is estimated as: 𝑛 𝑑 ∑︁ 𝐻ˆ𝑘 = 𝜓 (𝑛) − 𝜓 (𝑘) + log 𝑉𝑑 + log 𝜀𝑘 (𝑥𝑖 ) 𝑛 𝑖=1
(11)
where 𝜓 (·) denotes the digamma function, 𝑉𝑑 is the volume of the unit ball in R𝑑 , and 𝜀𝑘 (𝑥𝑖 ) is twice the Euclidean distance from 𝑥𝑖 to its 𝑘-th nearest neighbor. Unless otherwise stated, we fix the neighborhood size to 𝑘 = 10 in all experiments, following common practice in non-parametric entropy estimation [4, 14, 18]. To ensure meaningful comparison, all entropy values are computed under identical state dimensionality, normalization, and neighborhood size, and are compared only within the same environment. Under these controlled conditions, the estimated entropy serves as a relative measure of sample dispersion, reflecting the extent to which an algorithm explores the continuous state space. 5.2
RQ1: Effectiveness and Efficiency for Detecting Failures
At the beginning, we construct some well-trained agents for testing. We employ PPO [31] to train Cart Pole and Lunar Lander agents, TQC [19] to train Mountain Car agents, and DSAC-T [11] to train Humanoid agents. We repeat training and random testing for 100,000 times until two conditions are both satisfied: (1) there is at least one failure test case under 100,000 tests, and (2) the failures are less than 500. Specifically, the numbers of failures under 100,000 tests are 226, 164, 56, 112 for Cart Pole, Lunar Lander, Mountain Car and Humanoid respectively. We use the F-measure (the expected number of tests required to detect the first failure), proposed by Chen et al. [7], as an effectiveness metric, as it directly reflects the failure-detection capability of a testing algorithm. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:13
Table 3. Testing cost for finding the first failure with corresponding (𝑝-value, effect size)
Cart Pole PRT
Lunar Lander
Mountain Car
Humanoid
Number
Time
Number
Time
Number
Time
Number
Time
62.65
1.85
70.15
1.60
444.3
2.20
524.9
165.9 400.11 (0.02,0.69)
2341.4
465.7
616.1
92.7
96.0
4.68
1268.01
MDPFuzz
(0.00,0.98)
(0.00,1.00)
(0.00,0.92)
(0.00,0.98)
(1.00,0.19)
(0.00,0.79)
(0.03,0.68)
2182.6
68.4
613.5
15.1
1502.4
8.42
1116.6
352.4
CureFuzz
(0.00,0.93)
(0.00,0.93)
(0.00,0.91)
(0.00,0.92)
(0.00,0.80)
(0.00,0.82)
(0.21,0.58)
(0.20,0.58)
657.5
24.5
836.3
24.2
96192+2
968+2
4008.2
1310
G-Model
(0.00,0.81)
(0.00,0.83)
(0.02,0.69)
(0.01,0.71)
(0.00,1.00)
(0.00,1.00)
(0.00,0.94)
(0.00,0.94)
1616.5
48.6
566.9
13.2
656.1
4.23
1789.3
572.3
QD
(0.00,0.89)
(0.00,0.89)
(0.00,0.88)
(0.00,0.88)
(0.01,0.74)
(0.00,0.78)
(0.17,0.59)
(0.14,0.60)
491.5
15.2
532.8
12.7
2177.1
10.26
1261.3
393.2
RT
(0.00,0.89)
(0.00,0.90)
(0.00,0.89)
(0.00,0.89)
(0.00,0.79)
(0.00,0.79)
(0.01,0.75)
(0.01,0.75)
The item ranking first is colored red, while the second is colored blue. The statistics with 𝑝-value larger than 5% is colored orange. Number and Time mean the number of generated test cases and the corresponding time (in seconds, including algorithmic computation and environment simulation) for finding the first failure. 1 In the Humanoid environment, MDPFuzz requires approximately 553 seconds to analyze a single test case, implying that preparing 2,000 seeds would take more than 12 days before testing can even begin. Due to this prohibitive computational cost, we disable the freshness-guided component of MDPFuzz in the Humanoid environment. 2 Across 20 independent experiments, G-Model detected a failure only once, occurring at the 23,851st test case. In the remaining 19 experiments, testing was terminated after 100,000 trials without identifying any failures. Consequently, the minimum cost required for G-Model to detect a failure is (23,851+100,000*19)/20=96,192 in terms of test executions and 968 in terms of accumulated time cost.
Under the low failure rate condition, we compare the testing cost of different methods to find the first failure test case. We consider the testing cost from two aspects: the number of generated test cases and the total testing time. For simulated programs, we usually want to find failures as fast as possible. However, for some practical applications, we may consider the cost of executing a test case, where the number of test cases is more important. For each experiment, we repeat it 20 times and report the average cost together with the p-value and the effect size. For each experiment, we repeat the evaluation 20 times and report the average cost, together with the corresponding 𝑝-value and effect size. We employ Mann-Whitney U test to compute the 𝑝-value, under the one-sided assumption that the cost of the baselines is 𝑙𝑜𝑤𝑒𝑟 than that of PRT. The effect size between the baselines and PRT is measured using the 𝐴ˆ21 statistic [38]. Tab. 3 reports the average testing cost required by different methods to detect the first failure across various environments. In terms of average cost, PRT achieves the best performance in all cases, except for the number cost in Mountain Car. On one hand, the failure insights towards Mountain Car are deviated, where the failures are not precisely located in the central region. On the other hand, the reward pattern has strong guidance in this environment and MDPFuzz makes an aggressive strategy. Further analysis will be provided later in Fig. 2 and Fig. 3. Nevertheless, compared with RT, PRT consistently reduces the testing cost by at least 40%, requiring significantly fewer test cases to detect the first failure. Moreover, PRT incurs low computational overhead, achieving the shortest time cost across all testing subjects. In Mountain Car, although the number cost of PRT is nearly six times that of MDPFuzz, it still detects the first failure faster. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:14
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Tab. 3 also reports the corresponding 𝑝-values and effect sizes. Overall, PRT performs statistically better than RT across all environments. However, in some cases, the advantage of PRT is not statistically significant, particularly in the Humanoid environment. On one hand, the low average cost achieved by PRT can be attributed to its smaller maximal cost. In Humanoid, PRT’s maximal number cost is 1461, whereas RT, MDPFuzz, CureFuzz, and QD exhibit maximal number costs of 3069, 5135, 7858, and 7992, respectively. As will be shown in RQ2, PRT achieves the highest diversity, which may contribute to its stability. On the other hand, the statistical insignificance can be partly attributed to the high variance and limited repetitions in the experiments. The results indicate that, under limited experimental runs, some baselines can achieve competitive performance with PRT in specific scenarios, especially in the high-dimensional Humanoid environment. Focusing on the average cost, we observe that MDPFuzz achieves the best number cost in Mountain Car. Notice that MDPFuzz, CureFuzz, and QD all benefit from reward guidance (primarily, though not exclusively, driven by reward signals). To better understand the underlying mechanisms, we further analyze the reward patterns.
(a) Cart Pole
(b) Lunar Lander
(c) Mountain Car
(d) Humanoid
Fig. 2. Reward patterns of different environments
We draw the heat map as Fig. 2. For Humanoid, we execute 100,000 random test cases and select 10,000 test cases with the lowest rewards. For the other environments, we execute 10,000 random test cases. As the input domain of Cart Pole is four-dimensional, we present two plots (more details in Sec. 4.2). For Humanoid, since the input domain has 47 dimensions, we enumerate all 47 2 = 1081 dimension pairs and select the pair with the lowest diversity among failure test cases, as computed by Eq. 11, which implies failures are clustered most at that projections. The failure test cases achieve the lowest rewards and are colored dark purple. More specifically, the failure test cases of Cart Pole (Fig. 2a) are on the left side, those of Lunar Lander (Fig. 2b) appear in the lower-right region, those of Mountain Car (Fig. 2c) are concentrated near the center, close to (−0.3, 0), and those of Humanoid (Fig. 2d) are mainly distributed in the lower-right and the upper-left regions. We observe that in Cart Pole (Fig. 2a), the rewards of all passing test cases are identical, meaning the reward signal offers no guidance for distinguishing between passing and failing cases. In this setting, PRT and RT achieve the best performance. In Lunar Lander (Fig. 2b), although the rewards vary across the input domain, the passing test cases in the central region yield low rewards, which may mislead methods guided by reward signals. In this case, reward-driven baselines perform much better than those in Cart Pole, yet still fall short of PRT. In Mountain Car (Fig. 2c), passing test cases exhibit diverse rewards. Fortunately, the failure region is surrounded by areas of low reward, allowing MDPFuzz and QD to efficiently detect failures. Nevertheless, PRT consistently ranks second in terms of the number cost among all methods. In Humanoid (Fig. 2d), part of the failure test cases are located near the medium-reward test cases (green points), particularly in the upper-left region. However, these failure test cases are also surrounded by high-reward ones. As a result, MDPFuzz and CureFuzz can detect failures quickly in some cases, but in others it requires significantly more testing cost to detect failures. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:15
Answer to RQ1: Compared with RT, PRT reduces the testing cost by at least 50% across all testing subjects. Compared with DRL-based testing techniques, PRT’s advantage is particularly evident in the environments where reward signals are either uninformative or misleading. 5.3
RQ2: Diversity of Test Cases
We leverage entropy (Eq. 11) to evaluate the diversity of generated test cases by different methods. Each method is required to generate 10,000 test cases, and every experiment is repeated 20 times. To mitigate the threat of conclusion validity, we show the results with hyperparameters 𝑘 = 5, 𝑘 = 10 and 𝑘 = 20 in Eq. 11. We also report the standard variance of each experiment. Beyond such distance metric, we also visualize the test case distribution of different methods as Fig. 3. On one hand, this can implement the distance metric, making our conclusion more reliable. On the other hand, more importantly, that visualization intuitively shows the exploring strategies of different methods, enabling deeper analysis on their advantages and disadvantages. Note that, during the 20 experiments, we select the one with the lowest number cost finding the first failure for visualization, where the green and the red points represent the passing and the failure test cases respectively. Table 4. Entropy of 10,000 generated test cases under different 𝑘
Method
𝑘
Cart Pole
Lunar Lander
Mountain Car
Humanoid
PRT MDPFuzz CureFuzz G-Model QD RT
5
0.5266 ± 0.0043 −0.6569 ± 0.0205 −0.6567 ± 0.0203 −22.6375 ± 2.6734 −4.4818 ± 0.4454 0.1765 ± 0.0068
0.2306 ± 0.0014 0.2120 ± 0.0018 −0.0446 ± 0.0092 −0.5768 ± 0.0184 −0.0289 ± 0.0129 −0.0320 ± 0.0135 −9.6081 ± 0.7587 −10.0524 ± 1.0541 −1.2159 ± 0.1458 −1.2140 ± 0.1607 0.0157 ± 0.0053 0.0157 ± 0.0053
14.3975 ± 0.4199 −40.4884 ± 0.6664 −31.1420 ± 0.6763 −59.9535 ± 2.8108 −83.1634 ± 0.7219 14.2386 ± 0.0387
PRT MDPFuzz CureFuzz G-Model QD RT
0.4513 ± 0.0028 −0.3097 ± 0.0210 −0.3093 ± 0.0203 10 −22.0423 ± 2.7388 −4.3464 ± 0.4261 0.2142 ± 0.0071
0.1054 ± 0.0015 −0.0360 ± 0.0082 −0.0104 ± 0.0082 −9.0635 ± 0.7885 −1.1395 ± 0.1427 0.0215 ± 0.0057
0.0880 ± 0.0019 14.9516 ± 0.3636 −0.5727 ± 0.0176 2.3939 ± 0.5797 −0.0148 ± 0.0102 11.6711 ± 0.3869 −9.5217 ± 1.0936 −52.3097 ± 2.8589 −1.1994 ± 0.1615 −80.6062 ± 1.2369 0.0215 ± 0.0057 14.6885 ± 0.0383
PRT MDPFuzz CureFuzz G-Model QD RT
0.4044 ± 0.0027 0.0911 ± 0.0009 −0.0061 ± 0.0187 −0.0292 ± 0.0077 −0.0049 ± 0.0177 0.0012 ± 0.0083 20 −21.2772 ± 2.7295 −8.5240 ± 0.8100 −4.1132 ± 0.3829 −1.0618 ± 0.1529 0.2589 ± 0.0069 0.0290 ± 0.0037
0.0731 ± 0.0010 15.5194 ± 0.3137 −0.5719 ± 0.0177 14.6547 ± 0.1209 −0.0039 ± 0.0073 14.8709 ± 0.0826 −9.0049 ± 1.1442 −42.7788 ± 2.9618 −1.1918 ± 0.1622 −77.3323 ± 1.8267 0.0290 ± 0.0037 15.1581 ± 0.0371
From Tab. 4, PRT consistently achieves the highest diversity across all environments, indicating that the test cases generated by PRT are more well-dispersed and uniformly distributed in the input space. This observation is further supported by the visualizations in Fig. 3b and Fig. 3c, where PRT produces a more evenly distributed set of test cases than RT. In the other high-dimensional environments, though diversity of PRT is high in Tab. 4, Fig. 3 cannot reveal the dimensional combinations, resulting in little difference on the two-dimensional prejections. Moreover, as shown Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:16
Weibin Lin, Jiangtao Meng, and Zheng Zheng
(a) Cart Pole
(b) Lunar Lander
(c) Mountain Car
(d) Humanoid Fig. 3. Test case distributions of different methods Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:17
in Fig. 3c, PRT tends to focus more on the middle region of the input domain due to the mapping defined in Eq. 8 in the Mountain Car environment. However, since the failure region in Mountain Car deviates from the center of the input space, this bias reduces the efficiency of PRT in detecting the first failure, reported in Tab. 3. For Humanoid, since the input domain has 47 dimensions, we enumerate all 47 2 = 1081 dimension pairs and select the two pairs with the highest and lowest entropy among 10,000 test cases, as reported in Fig. 3d. From Fig. 3, we can observe that different algorithms exhibit distinct preferences. MDPFuzz seems to be more interested in the middle region in Lunar Lander (Fig. 3b) and the circle region in Mountain Car (Fig. 3c), which fit the reward pattern shown in Fig. 2b and Fig. 2c. It corresponds to the best number cost in Tab. 3 Mountain Car environment. CureFuzz achieves the third-highest diversity, following PRT and RT. G-Model exhibits very low diversity. He et al. observed that G-Model tends to concentrate on boundary regions [16]. Moreover, a GitHub issue reported that the generative model fails to converge during training, a problem we also encountered, leading to test cases outside the input domain. Following the solution proposed by Li et al., we project invalid test cases onto the boundary of the input domain. QD demonstrates a low diversity but finds many failures. That is because QD randomly samples a cluster of test cases but always selects the one with the lowest reward for executing. Upon detecting a failure, QD concentrates on mutating the failure test case. Both G-Model and QD take an aggressive strategy—they focus on specific domains with a large number of test cases—but they may miss some failures. G-Model miss failures in Mountain Car. QD does not find the failures around (𝑥, 𝑥) ¤ = (−0.9, 0.0) in Cart Pole environment (the middle left domain of the first line). Moreover, even if we add more test cases, it is still difficult for these methods to find the failures in such domain. That highlights the importance of test case diversity, which can help us miss fewer failures in testing. Answer to RQ2: PRT achieves the best diversity of test cases among all methods. Notably, PRT consistently performs better than RT across all testing subjects. 5.4
RQ3: Failure Patterns
In this section, we examine the failure patterns exhibited by different agents. Specifically, we empirically study the regions of the input domain where failures occur. Chen et al classified the patterns of failure test cases into three categories: point, strip and block patterns [7]. However, the dimensionality of Humanoid input domain is 47. Limited number of evenly distributed test cases cannot reveal the failures’ geometric characteristics. Even if we only place two points in every dimension, the combinations can be 247 ≈ 1011 . So we just explore failure patterns of the other three environments. For Cart Pole (discrete version), we select agents trained with A2C [25], QRDQN [9], and PPO [31]; for Lunar Lander (continuous version), SAC [15], TD3 [13], and PPO [31]; and for Mountain Car (continuous version), DDPG [21], TRPO [30], and TQC [19]. In total, eight algorithms are considered to ensure the robustness of our conclusions. Testing is performed using PRT, which achieves the highest diversity according to Tab. 4. For each testing subject, PRT generates up to 100,000 test cases to reduce the likelihood of missing failures. Note that PRT is characterized by: (1) prioritizing specific regions (boundary in Cart Pole, Lunar Lander and Humanoid), and (2) uniform distribution. That implies PRT is significantly effective for failures aligned with the manually defined regions and block-shaped failures. Fig. 4 illustrates the distribution of passing test cases (green) and failure test cases (red). We find that the testing subjects exhibit block-shaped (clustered) failures aligned with the task-induced failure insights. That is the reason why PRT achieve significant effectiveness in Tab. 3. Moreover, Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:18
Weibin Lin, Jiangtao Meng, and Zheng Zheng
(a) Cart Pole
(b) Lunar Lander
(c) Mountain Car
Fig. 4. Failure patterns of different agents.
we observe that such aligned and block-shaped patterns appear across different training algorithms in the same environment, which implies the task-induced failure insights are largely environmentdependent and agent-independent and such failure insight is a general tool for testing DRL agents. However, there are also point-shaped (dispersed) failures not aligned with the failure insights. Though not efficiently, PRT detects them by uniform distribution. Answer to RQ3: We empirically observe that there are block-shaped failures aligned with the task-induced failure insights across different training algorithms in the same environment, where PRT detects such failures by testing priority. There are also pointshaped failures not aligned with the task-induced failure insights. PRT can detect them by uniform distribution. 6
Discussion
The applicable conditions of PRT. Though PRT is capable of generating evenly distributed points within a hypercube (defined as Eq. 12), certain techniques can be applied to broaden its applicability. D = {𝑥 ∈ R𝑚 |𝑥 𝑖 ∈ [𝑖𝑛𝑓𝑖 , 𝑠𝑢𝑝𝑖 ], 𝑖 = 1, 2, · · · , 𝑚} (12) A hypercube is essentially a domain in which each dimension is independently bounded. To satisfy the boundedness assumption, a sufficiently large value can be chosen to approximate the infimum or the supermum. For the independence assumption, if the domain has a complex shape, a larger hypercube can be used to enclose the domain. Points generated outside the target domain are filtered out, while those within are retained. How to understand the hyperparameters 𝜏 and 𝜆? PRT assumes that the failure region is a hypercube with side length 𝜏, and it seeks to prevent two test cases from residing within the same 𝜏-length hypercube. Nevertheless, owing to randomness, it is challenging to strictly determine whether two points lie within such a region. Hence, assuming the existing points are evenly Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:19
distributed, we employ a ratio model as an approximation. As PRT prioritizes testing a specific domain, 𝜆 exhibits confidence in that task-induced failure insight. If 𝜆 = 1, the generating process follows a uniform distribution and that means the task-induced failure insight plays little role in the test case generation process. Can PRT be applied to test other software systems? The answer is yes. PRT is a failure-based testing method. We use PRT to test DRL agents because we can easily locate their failure-prone input domains. If we know about the size or the potential failure domain of other softwares, we can also leverage PRT to test them. Why some methods perform worse than RT in Tab. 3? Firstly, the evaluation metrics differ: while most methods are assessed by the total number of failures detected, we measure the cost for finding the first failure. Secondly, the methods have different search preferences. Many are effective at uncovering clustered failures, whereas RT distributes its effort across the entire input domain, avoiding excessive testing in any local region. At last, we have explained that in some environments they lack effective reward guidance, which will make their performance worse. 7
Threats to Validity
Internal Validity. A potential threat to internal validity lies in the setting of hyperparameters. For PRT, we describe the hyperparameters in Sec. 4.3. For the baselines, we strictly follow the settings reported by their authors whenever available. For MDPFuzz [29], Mazouni et al. confirmed that the hyperparameters are set as 𝐾 = 10, 𝜏 = 𝛾 = 0.01 [23]. For CureFuzz [16], the authors did not report the hyperparameter settings. We therefore examined its GitHub repository and found that the settings differ from the formulas in the paper. In our experiments, we followed the GitHub settings, normalized the rewards of different environments to [0, 1], and set 𝛼 = 5, 𝛽 = 0.01, 𝛾 = 1 for all experiments. For G-Model [20], all hyperparameters are clearly stated in the paper, and we adopt them directly. For QD [24], we refer to its GitHub repository. As it also uses the Lunar Lander environment, we take the same settings for that task. For Cart Pole, Mountain Car and Humanoid, we derive the settings by analogy from those of Taxi. External Validity. A limited number of testing subjects poses a threat to external validity. One concern is how to design task-induced failure insights for other environments. As discussed in Sec. 2.2, such insights can be derived from the tasks, which are intrinsic characteristics of DRL agents. Another concern is the generalizability of our findings. To mitigate this threat, we analyze the underlying mechanisms of different testing methods from multiple perspectives (e.g., reward patterns, test case distributions, and failure patterns), with the aim of explaining why the observed behaviors are expected, in order to generalize beyond the studied benchmarks. Construct Validity. A threat to construct validity lies in the diversity metric. Eq. 11 can be unstable due to large dimension 𝑑 or different hyperparameter 𝑘. However, 𝑑 only influcences the constant item, and that is equivalent when we only compare the entropy under the same environment and the same test case number. To mitigate the threat from 𝑘, we report 𝑘 = 5, 10, 20 in RQ2. Conclusion Validity. A threat to conclusion validity may come from the randomness of test execution. To mitigate this, we repeat each experiment 20 times and report the averaged results. In addition, we report (𝑝-value, effect size) in RQ1, and standard variance in RQ2. To further improve reproducibility, we fix the random seed in our open-source implementation. 8
Related Works
Fuzz testing. Pang et al. proposed MDPFuzz [29], which leverages rewards to calculate seed sampling probabilities and employs a Gaussian Mixture Model (GMM) to estimate state freshness. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:20
Weibin Lin, Jiangtao Meng, and Zheng Zheng
Mazouni et al. verified that while the GMM strategy is effective, it is also time-consuming [23]. He et al. developed CureFuzz [16], a curiosity-driven fuzzing approach. Wan et al. introduced DRLFuzz [39], a white-box fuzzing method that uses a coverage-guided algorithm and the Q-network gradient to mutate seeds for testing DRL agents. Search-based Testing. Zolfagharian et al. proposed a white-box search-based testing method STARLA [42]. It leverages the Q-value of agents to encode the state sequence into a binary sequence and applies a random forest algorithm to predict the failure probability of that sequence as the fitness function. Biagiola and Tonella introduced Indago [5], searching for failure environment configurations with a failure predictor to give the failure probability. However, both STARLA and Indago rely on training data—test cases and their corresponding passing or failure labels. Mazouni et al. characterize a sequence with a 2-dimensional array as the agents’ behavior, so as to guide genetic search for failures [24]. Ma et al. proposed MASTest [22], a diversity-guided testing method for multi-agent systems. Generative-based Testing. Li et al. introduced the diffusion model to generate test cases [20]. In every epoch, random test cases are sampled to fine-tune the diffusion model, which then generates a batch of new test cases for evaluation. 9
Conclusion and Future Work
In this work, we focus on well-trained DRL agents and propose a failure-based testing technique PRT. Our key point is that DRL agents are task-oriented and we can intuitively have the failure insights about which task is harder and the corresponding region has a higher failure probability. With the task-induced failure insights, we leverage PRT to prioritize testing the specific regions of the input domain, in order to uncover failures as fast as possible. Our empirical experiments show that PRT can effectively and efficiently find failures in terms of the number of test cases and the test execution time. Its advantage is particularly evident in the environments where reward signals are either uninformative or misleading for reward-guided algorithms. Furthermore, we investigate the failure patterns of several agents across different environments and we find that the task-induced failure insights are empirically aligned with the block-shaped failures while the characteristic of uniform distribution can help PRT to find the point-shaped failures. In the future, we plan to apply PRT to testing other DRL agents according to their specific failure insights as well as other types of software. For example, the failure-prone regions of codes can be the boundary or branch conditions in the program, which can be served as the failure-insights for PRT to test. Besides, since PRT is efficient in finding the first failure and some methods can exploit specific failure characteristics, we will further investigate the feasibility of combining PRT with these methods, aiming to improve the effectiveness of detecting clustered failures while maintaining high efficiency and uniform distribution. Acknowledgements This work was partially supported by the National Key R&D Program of China (Grant No. 2024YFB33 11503) and the National Natural Science Foundation of China (Grant No. 62372021). We sincerely thank Tsong Yueh Chen (Swinburne University of Technology), Yuechen Li (Beihang University), Yi Cai (Beihang University), and Tianjie Zhou (Beijing Normal University) for helpful discussions and insightful comments. Data Availability We open our source code at https://github.com/Avagnes/PRT-DRL-Experiments/tree/main to show more details and help for further researches. Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
Failure-Based Testing for Deep Reinforcement Learning Agents
FSE178:21
References [1] 2025. Tesla Robotaxi Hits Parked Car in First Recorded Accident. PC Magazine. [2] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine 34, 6 (2017), 26–38. [3] Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. 1983. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Transactions on Systems, Man, and Cybernetics SMC-13, 5 (1983), 834–846. doi:10.1109/TSMC.1983.6313077 [4] Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, et al. 1997. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences 6, 1 (1997), 17–39. [5] Matteo Biagiola and Paolo Tonella. 2024. Testing of Deep Reinforcement Learning Agents with Surrogate Models. ACM Trans. Softw. Eng. Methodol. 33, 3, Article 73 (March 2024), 33 pages. doi:10.1145/3631970 [6] Tsong Yueh Chen, Fei-Ching Kuo, Robert G. Merkel, and T.H. Tse. 2010. Adaptive Random Testing: The ART of test case diversity. Journal of Systems and Software 83, 1 (2010), 60–66. doi:10.1016/j.jss.2009.02.022 SI: Top Scholars. [7] T. Y. Chen, H. Leung, and I. K. Mak. 2005. Adaptive Random Testing. In Advances in Computer Science - ASIAN 2004. Higher-Level Decision Making, Michael J. Maher (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 320–329. [8] Thomas M Cover and Joy A Thomas. 2006. Elements of Information Theory. Vol. 1. John Wiley & Sons. [9] Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos. 2018. Distributional reinforcement learning with quantile regression. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence (New Orleans, Louisiana, USA) (AAAI’18/IAAI’18/EAAI’18). AAAI Press, Article 353, 10 pages. [10] Yi Dong, Xingyu Zhao, Sen Wang, and Xiaowei Huang. 2024. Reachability Verification Based Reliability Assessment for Deep Reinforcement Learning Controlled Robotics and Autonomous Systems. IEEE Robotics and Automation Letters 9, 4 (2024), 3299–3306. doi:10.1109/LRA.2024.3364471 [11] Jingliang Duan, Wenxuan Wang, Liming Xiao, Jiaxin Gao, Shengbo Eben Li, Chang Liu, Ya-Qin Zhang, Bo Cheng, and Keqiang Li. 2025. Distributional Soft Actor-Critic With Three Refinements. IEEE Transactions on Pattern Analysis and Machine Intelligence 47, 5 (2025), 3935–3946. doi:10.1109/TPAMI.2025.3537087 [12] Hasan Ferit Eniser, Timo P. Gros, Valentin Wüstholz, Jörg Hoffmann, and Maria Christakis. 2022. Metamorphic relations via relaxations: an approach to obtain oracles for action-policy testing. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (Virtual, South Korea) (ISSTA 2022). Association for Computing Machinery, New York, NY, USA, 52–63. doi:10.1145/3533767.3534392 [13] Scott Fujimoto, Herke Hoof, and David Meger. 2018. Addressing function approximation error in actor-critic methods. In International conference on machine learning. PMLR, 1587–1596. [14] Weihao Gao, Sreeram Kannan, Sewoong Oh, and Pramod Viswanath. 2017. Estimating Mutual Information for Discrete-Continuous Mixtures. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.), Vol. 30. Curran Associates, Inc. https: //proceedings.neurips.cc/paper_files/paper/2017/file/ef72d53990bc4805684c9b61fa64a102-Paper.pdf [15] Tuomas Haarnoja, Aurick Zhou, P. Abbeel, and Sergey Levine. 2018. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ArXiv abs/1801.01290 (2018). https://api.semanticscholar.org/ CorpusID:28202810 [16] Junda He, Zhou Yang, Jieke Shi, Chengran Yang, Kisub Kim, Bowen Xu, Xin Zhou, and David Lo. 2024. Curiosity-Driven Testing for Sequential Decision-Making Process. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–14. [17] David Isele, Reza Rahimi, Akansel Cosgun, Kaushik Subramanian, and Kikuo Fujimura. 2018. Navigating occluded intersections with autonomous vehicles using deep reinforcement learning. In 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2034–2039. [18] Leonid F. Kozachenko and Nikolai N. Leonenko. 1987. Sample estimate of the entropy of a random vector. Problems of Information Transmission 23, 2 (1987), 95–101. [19] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, and Dmitry Vetrov. 2020. Controlling overestimation bias with truncated mixture of continuous distributional quantile critics. In International Conference on Machine Learning. PMLR, 5556–5566. [20] Zhuo Li, Xiongfei Wu, Derui Zhu, Mingfei Cheng, Siyuan Chen, Fuyuan Zhang, Xiaofei Xie, Lei Ma, and Jianjun Zhao. 2023. Generative model-based testing on decision-making policies. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 243–254. [21] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971 (2015). [22] Xuyan Ma, Yawen Wang, Junjie Wang, Xiaofei Xie, Boyu Wu, Shoubin Li, Fanjiang Xu, and Qing Wang. 2024. Enhancing multi-agent system testing with diversity-guided exploration and adaptive critical state exploitation. In Proceedings of Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.
FSE178:22
Weibin Lin, Jiangtao Meng, and Zheng Zheng
the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1491–1503. [23] Quentin Mazouni, Helge Spieker, Arnaud Gotlieb, and Mathieu Acher. 2024. Policy Testing with MDPFuzz (Replicability Study). In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1567–1578. [24] Quentin Mazouni, Helge Spieker, Arnaud Gotlieb, and Mathieu Acher. 2024. Testing for Fault Diversity in Reinforcement Learning. In Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024) (Lisbon, Portugal) (AST ’24). Association for Computing Machinery, New York, NY, USA, 136–146. doi:10.1145/3644032.3644458 [25] Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In International Conference on Machine Learning. https://api.semanticscholar.org/CorpusID:6875312 [26] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 (2013). [27] Andrew William Moore. 1990. Efficient Memory-based Learning for Robot Control. Technical Report. University of Cambridge. [28] Hai Nguyen and Hung La. 2019. Review of deep reinforcement learning for robot manipulation. In 2019 Third IEEE international conference on robotic computing (IRC). IEEE, 590–595. [29] Qi Pang, Yuanyuan Yuan, and Shuai Wang. 2022. Mdpfuzz: testing models solving markov decision processes. In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis. 378–390. [30] John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan, and Pieter Abbeel. 2015. Trust region policy optimization. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37 (Lille, France) (ICML’15). JMLR.org, 1889–1897. [31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [32] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 362, 6419 (2018), 1140–1144. [33] Amal Sunba, Jameleddine Hassine, and Moataz Ahmed. 2026. Testing reinforcement learning systems: A comprehensive review. Journal of Systems and Software 231 (2026), 112563. doi:10.1016/j.jss.2025.112563 [34] Richard S. Sutton and Andrew G. Barto. 2018. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA. [35] Yuval Tassa, Tom Erez, and Emanuel Todorov. 2012. Synthesis and stabilization of complex behaviors through online trajectory optimization. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems. 4906–4913. doi:10.1109/IROS.2012.6386025 [36] Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulao, Andreas Kallinteris, Markus Krimmel, Arjun KG, et al. 2024. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032 (2024). [37] Hado Van Hasselt, Arthur Guez, and David Silver. 2016. Deep reinforcement learning with double q-learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 30. [38] András Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the "CL" Common Language Effect Size Statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics 25, 2 (2000), 101–132. http: //www.jstor.org/stable/1165329 [39] Xiaohui Wan, Tiancheng Li, Weibin Lin, Yi Cai, and Zheng Zheng. 2024. Coverage-guided fuzzing for deep reinforcement learning systems. Journal of Systems and Software 210 (2024), 111963. [40] Cathy Wu, Abdul Rahman Kreidieh, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. 2021. Flow: A modular learning framework for mixed autonomy traffic. IEEE Transactions on Robotics 38, 2 (2021), 1270–1286. [41] Hitoshi Yoshioka and Hirotada Hashimoto. 2024. A Reliability Quantification Method for Deep Reinforcement Learning-Based Control. Algorithms 17, 7 (2024). doi:10.3390/a17070314 [42] Amirhossein Zolfagharian, Manel Abdellatif, Lionel C Briand, Mojtaba Bagherzadeh, and S Ramesh. 2023. A searchbased testing approach for deep reinforcement learning agents. IEEE Transactions on Software Engineering 49, 7 (2023), 3715–3735.
Received 2026-02-24; accepted 2026-03-24
Proc. ACM Softw. Eng., Vol. 3, No. FSE, Article FSE178. Publication date: July 2026.