Symmetry-Aware Surrogate-Assisted Testing of UAVs

arXiv:2611.08012v1 [cs.RO] 5 Nov 2026

Symmetry-Aware Surrogate-Assisted Testing of UAVs 1st Anas Shrinah

2nd Chris Bennett

3rd Kerstin Eder

Department of Software Engineering Applied Science Private University Amman, Jordan a shrinah@asu.edu.jo

Department of Computer Science University of Bristol Bristol, United Kingdom christopher.bennett@bristol.ac.uk

Department of Computer Science University of Bristol Bristol, United Kingdom Kerstin.Eder@bristol.ac.uk

Abstract—We investigate a symmetry-aware surrogate-assisted testing approach for UAV autopilots. The proposed approach exploits geometric symmetries between flight missions by transforming obstacle configurations into a shared canonical coordinate frame. This enables a neural-network surrogate trained on a single reference mission to be reused across arbitrary missions, reducing the amount of high-fidelity simulation data required for training. The surrogate model is trained to predict the minimum clearance distance between a UAV and surrounding obstacles and is used to guide test generation towards safety-critical scenarios. We evaluate this approach using the Aerialist UAV test bench and benchmark its performance against techniques from the 2024 to 2026 UAV Testing Competitions. Results show that the approach achieves an average critical-test-case hit rate of 42% across the evaluated missions, compared with approximately 10% for the constrained-random baseline used to generate the training data. This approach also achieves high input diversity and competitive benchmark performance. However, our findings reveal limitations in transferring surrogate predictions across missions with differing flight-path characteristics, highlighting the boundaries of geometric canonicalization. To support reproducibility and future research, we provide the trained surrogate model and a dataset comprising more than 2,500 high-fidelity simulations. Index Terms—UAV testing, simulation-based testing, surrogateassisted testing, symmetry reduction, canonical representation, neural networks.

I. I NTRODUCTION Unmanned Aerial Vehicles (UAVs) are increasingly employed in applications that require autonomous navigation in complex environments, such as indoor facilities [1], urban areas [2], and disaster sites [3]. In such environments, failures in obstacle avoidance can lead to collisions, making the safety and reliability of autonomous navigation systems a critical concern. To address these concerns, verification and validation methods, such as simulation-based testing, play an important role in establishing confidence in the trustworthiness of these systems. Simulation-based testing uses high-fidelity simulators to evaluate autonomous UAVs under a diverse range of environmental conditions while reducing the time, cost, and safety risks associated with physical flight testing. However, the input space makes exhaustive testing impractical. Searchbased testing offers a systematic approach to this challenge by guiding test generation towards scenarios that are more likely to reveal system failures [4]. This process, however,

requires evaluating numerous candidate scenarios, making the computational cost of high-fidelity simulations a significant bottleneck. Surrogate-assisted testing offers an approach to mitigating this bottleneck by using computationally inexpensive surrogate models to estimate test outcomes without executing the highfidelity simulator [5]. These models learn an approximation of the relationship between test inputs and simulation outcomes using data from previously executed high-fidelity simulations. Since evaluating a surrogate model is generally much less expensive than executing a high-fidelity simulation, a large number of candidate test scenarios can be assessed before selecting a smaller subset for simulation. In UAV obstacle-avoidance testing, test scenarios can be constructed by placing obstacles along a flight mission, with the objective of identifying configurations that cause the UAV to approach obstacles closely or exhibit unsafe behaviour. A flight mission may consist of multiple segments with different positions and orientations, each requiring its own obstacle configurations. Developing an accurate surrogate model for such scenarios requires training data representative of the configurations encountered across these segments. However, obtaining sufficient training data through high-fidelity simulations can be prohibitively expensive, especially when separate training data must be collected for different flight segments or missions. An opportunity to address this challenge arises from the geometric symmetries between flight segments. Obstacle configurations associated with different flight segments may have identical relative geometries despite differing in their global positions and orientations. Such geometrically equivalent configurations can be represented relative to a reference flight segment in a canonical coordinate frame, which we refer to as the canonical flight frame. In this work, we investigate a surrogate-assisted testgeneration approach that exploits these geometric symmetries to reuse a neural-network surrogate across different flight missions. During training, the surrogate is trained to predict the minimum clearance distance between a UAV and surrounding obstacles for a reference flight mission. This reference flight mission defines the canonical flight frame. During test generation, candidate obstacle configurations for a target flight

mission are transformed into the canonical flight frame and evaluated using the trained surrogate. Promising configurations are then transformed back into the target mission’s coordinate system and evaluated using the high-fidelity simulator. Geometric equivalence does not necessarily imply equivalent UAV behaviour. The response of an autonomous UAV may also depend on factors such as its incoming velocity, heading, controller state, preceding flight segments, and the surrounding environment. Consequently, the extent to which a surrogate trained in a canonical frame can provide accurate predictions for different missions must be established empirically. This work investigates the efficiency and effectiveness of canonical surrogate reuse for simulation-based UAV testing, as well as its limitations. Specifically, we address the following research questions: • RQ1: How does surrogate prediction accuracy vary across different flight missions? • RQ2: To what extent does an NN-based surrogate improve the efficiency of test generation? • RQ3: How does filtering based on a canonical surrogate affect test-suite diversity? To answer these questions and evaluate our approach, we apply it to efficiently create effective test cases for PX4Avoidance, a vision-based autonomous obstacle avoidance system developed on top of PX4-Autopilot [6]. We benchmark our approach against the test case generators that participated in the UAV Testing Competition 2024, 2025 and 2026 versions [7]–[11]. By addressing these research questions, this work makes the following main contributions: • A neural network surrogate model that approximates the PX-4 obstacle avoidance system and estimates the minimum clearance distance between a UAV and an obstacle. • An extensive empirical evaluation comparing the efficiency and effectiveness of the surrogate-guided testcase generator with those of state-of-the-art test-case generators. • The limitations of canonical surrogate reuse for simulation-based UAV testing. • A dataset of simulation results for over 2500 different obstacle configurations for a flight mission, which can be used to develop other types of surrogate models. The remainder of this paper is organized as follows. Section II reviews related work and Section III introduces the UAV case study. Section IV describes the symmetry-aware testing approach, while Section V reports the empirical evaluation. Section VI discusses the findings, assumptions, and limitations. Finally, Section VII concludes the paper and outlines future work. II. R ELATED W ORK The UAV testing competition has motivated several strategies for exploring the test-input space. In particular, DeepHyperion-UAV [12] uses illumination search,

CAMBA [13] employs cost-aware mutation, and TUMB [14] uses Monte Carlo Tree Search to generate obstacle configurations. These approaches employ different search strategies to identify critical test cases, but evaluating candidate configurations through high-fidelity simulation remains computationally expensive. To address the cost of simulation-based testing, researchers have explored alternative approaches to test evaluation and generation. Shrinah and Eder [15] employed a manually constructed low-fidelity path simulator to approximate UAV behaviour at a lower computational cost. Other approaches have investigated learning-based techniques. Zhu et al. [16] proposed SaFliTe, which uses a large language model to predict test interestingness from textual safety objectives and the current system state. Peltomäki and Porres [17] proposed WOGAN, which iteratively trains a Wasserstein generative adversarial network to learn a distribution over counterexamples from which failure-inducing inputs can be sampled. The use of surrogate models to reduce the cost of searchbased testing is well established. Nejati et al. [4] provide a taxonomy of surrogate-assisted search-based testing. Beglerovic et al. [5] combined surrogate modelling with stochastic optimisation to search for fault-revealing regions when testing autonomous vehicles. Similarly, Haq et al. [18] employed surrogate-assisted many-objective optimisation to improve the efficiency of online testing of DNN-enabled systems. These approaches establish that learned approximations can reduce the cost of evaluating candidate tests during search. In contrast, our work is concerned with the representation used to train such an approximation: rather than learning separately over mission-dependent spatial coordinates, geometrically equivalent local flight configurations are mapped to a common reference frame. Menghi et al. [19] proposed ARIsTEO for testing computeintensive cyber-physical models. ARIsTEO constructs a cheaper surrogate of the model under test, searches this approximation for failure-revealing inputs, validates candidate failures on the original model, and refines the approximation when a candidate is spurious. This approximation–refinement strategy addresses errors in the surrogate dynamically. Neural-network-based guidance has also been investigated in evolutionary testing. Zhong et al. [20] proposed AutoFuzz, which combines neural-network guidance with evolutionary fuzzing to identify traffic-violation scenarios for autonomous vehicles. Similarly, Zohdinasab et al. [21] proposed DeepAtash-LR, which integrates a surrogate model into an evolutionary focused-testing process for autonomous driving, directing test generation towards specified combinations of structural and behavioural features. While these approaches demonstrate different ways of using learned models to guide test generation, they do not investigate the reuse of a single surrogate across geometrically related test scenarios through canonical representation. Geometric symmetry has been widely investigated as a means of reducing redundant variation in machine learning.

We use the UAV Testing Competition as a case study for evaluating our symmetry-aware surrogate-assisted testing approach. The competition was introduced at SBFT in 2024 and subsequently organized at both SBFT and ICST in 2025 and 2026. The competition provides a benchmark for comparing automated test-case generation techniques for autonomous UAVs. The system under test is PX4-Avoidance, an obstacleavoidance system built on PX4-Autopilot. Test cases specify the geometry and placement of cuboid obstacles for a given flight mission. According to the competition constraints, valid test cases must have non-overlapping obstacles that lie entirely within the designated obstacle-placement area, defined as a 70, m × 40, m rectangle. Test cases are used to expose unsafe behaviours by executing the given flight mission in the presence of the obstacles defined by the test case using Aerialist, a simulation-based UAV testing framework for PX4-Autopilot [26]. The competition scores test generators based on the criticality and diversity of their test suites. Criticality is measured by the minimum clearance distance between the UAV and the obstacles, while diversity is assessed by comparing test-case

IV. S YMMETRY-AWARE S URROGATE -A SSISTED T ESTING A. Overview Our approach generates a set of tests that are predicted to cause a UAV to fly close to an obstacle for a given flight mission. Each test case consists of two obstacles, each parameterised by five values: width, height, centre coordinates, and rotation. Together, the two obstacles form a ten-element test vector. The approach uses a neural network surrogate model to estimate the minimum clearance distance between a UAV and an obstacle during a mission. The model is trained using randomly generated tests from a single canonical mission, which serves as the reference mission during training and inference. A transformation is found between any arbitrary mission and the canonical mission. This transformation enables tests to be generated within the surrogate model’s training domain, where prediction accuracy is expected to be highest. During test generation, batches of tests are first produced using a constrained-random generator. The surrogate model is then used in a filtering mechanism to select tests that are both critical and diverse. Critical tests are defined as those for which the minimum clearance distance is below 1.5m. Diverse tests are defined as those that increase the structural variability of non-overlapping geometries in the test suite’s input space (Equation 9). Finally, the selected test cases are executed in the high-fidelity simulator to obtain their actual minimum clearance distances and verify the surrogate’s predictions. B. Canonical Representation of Flight Segments To ensure the surrogate model is always used within its training domain, each arbitrary mission is transformed into a canonical mission. The canonical mission is a single vertical flight path, centred horizontally within the designated obstacleplacement area, as illustrated in Figure 1. 55 50

(a) Arbitrary Mission

55 50

(b) Canonical Mission WP2

WP2 40

Y / East (m)

III. C ASE S TUDY

similarities within the suite. We use the competition missions and evaluation criteria to assess the efficiency and effectiveness of our approach.

Y / East (m)

Cohen and Welling [22] introduced group-equivariant convolutional networks, which incorporate transformations such as rotations and translations into the network architecture. Bietti et al. [23] studied the sample-complexity benefits of learning under invariance. These works motivate the exploitation of geometric symmetry, but do not address surrogate reuse in simulation-based testing. Kaba et al. [24] proposed learning canonicalisation functions that transform inputs into a canonical representation before prediction. Our approach follows the same general principle, but determines the transformation directly from the geometry of a UAV flight segment rather than learning it. The objective is to reuse simulation data collected for one reference configuration when testing different mission segments, while ensuring that the generated obstacle configurations remain feasible. Wang et al. [25] investigated approximately equivariant networks for dynamical systems in which geometric symmetry is imperfect. This limitation is also relevant to our work: geometrically equivalent obstacle configurations may produce different UAV behaviour because of controller state, preceding flight segments, or environmental constraints. We therefore evaluate cross-mission surrogate accuracy empirically rather than assuming that geometric equivalence guarantees identical simulation outcomes. Existing research has established surrogate-assisted testing as a means of reducing expensive simulations [4], [5], [19], [21] and canonicalisation as a means of exploiting geometric symmetry [22]–[24]. Our work combines these established ideas to investigate whether a safety surrogate trained on a single canonical UAV flight configuration can be reused across different mission segments without retraining.

30 20 10

WP1

0 −5 −60 −50 −40 −30 −20 −10

0

10

20

30

40

50

40 30 20 10 0 −5 −60 −50 −40 −30 −20 −10

X / North (m) Flight path

Designated obstacle-placement area

WP1 0

10

20

30

40

50

X / North (m) Segment obstacle zone

Fig. 1: A segment of an arbitrary mission path is transformed to the canonical mission representation. The segmentplacement zone is chosen in the arbitrary mission frame such that when transformed between representations, it always remains inside the designated obstacle-placement area.

The transformation consists of three steps: 1) For each flying segment between two consecutive waypoints, determine a segment-placement zone by constructing the largest rectangle, symmetric about the segment, that lies entirely within the flight-testing arena. 2) Select the mission segment whose segment-placement zone has the greatest area. 3) Apply a two-dimensional translation and rotation to map both the mission segment and its associated segmentplacement zone into the canonical representation. The transformed segment-placement zone defines the geometric constraints used during random obstacle generation. Depending on the mission geometry, this zone may occupy only a subregion of the obstacle placement zone used to train the surrogate model. Consequently, all generated obstacles remain within the surrogate model’s training domain. After test generation, the inverse transformation is applied, mapping the generated obstacles from the canonical representation back to the original mission frame. Tests expressed in the original mission frame are then evaluated using the Aerialist simulator.

This sampling process was observed to produce highly skewed distributions for the obstacle widths and the x coordinate of obstacle one. Hence, to reduce the effect of skewness, a log transformation was applied to these features prior to standardisation. All ten geometry features were then normalised using feature-wise mean and standard deviation statistics calculated from the training set. Each obstacle configuration was evaluated using Aerialist to obtain the groundtruth minimum clearance distance used in training. The network was trained for 200 epochs using the Adam optimiser, a batch size of 128, and mean absolute error (MAE) as the loss function. D. Feasibility-Constrained Obstacle Generation The trained surrogate model is used with a constrainedrandom generator that generates obstacle geometries (tests). The constraints for this random generator are derived from the segment-placement zone for each mission, transformed into the canonical representation. These constraints are applied to the generation of the ten obstacle parameters to ensure that each obstacle pair do not overlap and remain entirely within the placement zone.

C. Surrogate Model Training

E. Surrogate-Guided Test Selection

The surrogate model is a neural-network regression model that predicts the minimum clearance distance between the UAV and any obstacle during the mission. The model takes the geometry of two obstacles as input and outputs an estimated minimum clearance. Its architecture, depicted in Figure 2, comprises two parallel subnetworks, one for each obstacle, with sizes of 5 input, 10 hidden and 20 output layer neurons. The outputs of the subnetworks are concatenated and passed to a fully connected network with 40 input, 20 hidden and 1 output neuron. ReLU activations are used for all hidden layers, while the output layer uses a linear activation. This architecture was selected empirically using validation performance. The surrogate model’s dataset is created from 2716 random test cases of two obstacles within the designated obstacleplacement area. Eighty percent of the random test cases were used for training, and the remaining twenty percent were used for validation. The dataset was sampled from the space of all possible obstacle configurations by a specialist constrained-random generator that employs semantic sampling rather than direct geometric sampling. Scenario features such as obstacle intrusion into the flight path, size, aspect ratio, rotation, relative neighbour direction, and passage width were discretised into a set of categories. For example, obstacle rotation was classified into low (0◦ to 45◦ ) and high (45◦ to 90◦ ) categories, while relative obstacle placement was classified into one of eight neighbouring sectors. Data generation first uniformly sampled a combination of these categories, one from each feature, and subsequently randomly sampled geometric values from the corresponding parameter ranges. This approach ensured balanced coverage of distinct scenario types and avoided overrepresentation of geometrically similar configurations.

Test generation is split into two phases. In both phases, a fixed-size leaderboard is maintained to store the most critical and diverse tests discovered so far. Phase one focuses on identifying tests with low predicted obstacle clearances. A batch of tests is repeatedly generated by the constrained-random generator. For each test, the surrogate model predicts the minimum obstacle clearance distance. The leaderboard is then updated to retain the tests with the smallest predicted clearances seen so far, while tests with larger predicted clearances are discarded. This process of generation and selection is repeated 100 times. At the end of phase one, the leaderboard contains tests with small clearances, but potentially poor diversity. Phase two shifts the emphasis to improving leaderboard diversity while retaining only critical tests. The constrainedrandom generator generates new batches of tests to improve test suite diversity, and the surrogate model filters out any noncritical tests (predicted minimum clearance distance > 1.5m), leaving a set of candidate tests. For each candidate set, a series of alternative leaderboards are created by replacing one existing test with a candidate test, and evaluating the diversity of the resulting leaderboard using Equation 9, described in Section V. The leaderboard with the highest diversity is retained for the next iteration. The process of candidate generation, selection and diversity evaluation is repeated for 100 rounds, producing a leaderboard that balances both test criticality and diversity. Phase three verifies the predicted criticality of test cases in the leaderboard. Each test in the final leaderboard is evaluated using the high-fidelity simulator to obtain a more accurate estimate of minimum clearance. Our approach balances computational efficiency and prediction accuracy. In the first two phases, the approach rapidly explores a large space of

Two-obstacle surrogate neural network Obstacle 1 branch

5 input features L, W, X, Y, R

Dense (10) ReLU

Dense (20) ReLU

Concatenate 40 features

Obstacle 2 branch

5 input features L, W, X, Y, R

Dense (10) ReLU

Dense (20) ReLU

Dense (20) ReLU

Dense (1) Linear output

Predicted minimum distance

Fig. 2: The architecture of the surrogate model.

potential tests, while the computationally expensive simulator is reserved for the relatively small set of tests selected for the final leaderboard in phase three. Consequently, simulation effort is concentrated on the most promising tests, improving confidence in the final ranking without requiring exhaustive simulation of all considered tests. V. E MPIRICAL E VALUATION A. Experimental Setup The performance of our approach was benchmarked against submissions from the 2024, 2025 and 2026 editions of the ICST/SBFT UAV testing competitions. The benchmark comprises the flight mission set, simulation budget and scoring criteria. We aligned our evaluation with these components to enable a direct comparison between our proposed approach and the prior competition entrants. We also recalculated the scores of the benchmarks using the publicly available simulation artifacts and scoring criteria. All experiments were conducted on a machine equipped with an Intel Core i710870H processor (8 cores, 16 threads) and 32 GB of RAM, running Ubuntu 20.04.6 LTS. UAV simulations were executed in Docker containers using Aerialist version 0.2.02 (Git revision f44ea6a) with Ubuntu 18.04.5 LTS as the container operating system. The datasets, source code, and experimental artifacts supporting this study are available on Zenodo [27]. Each component of the benchmark is summarised in this section, with complete details of the format and scoring for each competition year found in [7]–[11]. 1) flight mission: The three flight missions common to all competitions, namely CS2, CS4 and CS5, were selected for evaluating our approach. Each mission varies in the number and layout of waypoints, and the extent to which the flightpath segments overlap the valid obstacle-placement area. See Figure 3. 2) Simulation Budget: The simulation budget defines the maximum number of test cases that can be evaluated using Aerialist during test generation, prior to submission and scoring. In accordance with the competition requirements, we use simulation budgets of 200 for 2024 and 100 for 2025 and 2026. Consequently, we execute our approach twice: once

with a budget of 200 to generate a test suite for comparison against the 2024 entrants, and once with a budget of 100 for comparison against the 2025 and 2026 entrants. In both runs, the leaderboard size is set equal to the corresponding simulation budget. 3) Scoring Criteria: The scoring is based on simulations of the submitted test suite using Aerialist. Each test case is simulated three times, and averages are calculated for the minimum clearance distance, flight path, and execution time. Each simulation run was limited to 500 seconds. Runs that exceeded this limit before completing the required flight mission were terminated and recorded as timed out. Different scoring criteria is used for each year, reflecting changes made by the competition organisers intended to improve the assessment criteria. 2024: The submitted test suite (S) is split into two groups. One group, Ss is scored directly, and the other group, Sr is scored based on an average score from the first group. The competition organisers chose this approach to reduce the computational time needed to score an entire test suite. The final score integrates the test scores for simulated and unsimulated tests and a test similarity penalty: suite score(s) = (test score(Ss ) + rest score(Sr )) × similarity penalty(Ss )

(1)

For the first group, Ss , 20 tests are selected at random from the submitted test suite. Each of these test cases was checked for validity. Obstacles must not overlap or extend outside the valid obstacle-placement area. Valid test cases are then simulated three times with Aerialist, and the average path, execution time and minimum obstacle clearance distance are recorded. Runs of the simulated tests were awarded 5 points if the minimum clearance distance is less than 0.25m, 2 points if the distance is between 0.25 and 1.00m, 1 point if the distance is between 1.00m and 1.5m and zero otherwise. Tests with minimum clearance distances less than 1.5m are denoted critical tests. The points from the three simulations for each test case are averaged.

(a) CS2

(c) CS5 WP3

60 Y / East (m)

(b) CS4

WP2

WP3 WP2

40

WP2 20 0

WP3 −60

−30

WP4

WP1

0 X / North (m) Flight path

30

60

−60

Selected segment

−30

WP4

WP1 0 X / North (m)

30

60

−60

Designated obstacle-placement area

−30

WP1

0 X / North (m)

30

60

Segment obstacle zone

Fig. 3: The flight missions, path segments, and segment-placement zones for transformation to the canonical mission.

The test score for each test case is then calculated using its average point (avg point), the number of obstacles used (obst) and the average mission flight time (avg time) in minutes. 10 × avg point(t) (2) test score(t) = #obst(t)2 × avg time(t)

scores. The ranks for each category were summed across missions to determine the category standings. The overall ranking gave failure detection (test criticality) performance three times the weight of test diversity:

The test score of the set of selected tests test score(Ss ) is the sum of the test scores of the individual tests test score(t). The remaining unsimulated tests are scored based on an average from the simulated set. P test score(t) rest score(Sr ) = |Sr | × t (3) 2|Ss |

Diversity is scored as:

If less than 20 tests are submitted, then |Sr | = 0 and no score is given for unsimulated tests. Test diversity is calculated using the Dynamic Time Warping distance between the trajectories of any two simulated tests. A test is counted as similar if the DTW distance is less than 75. This value was chosen for compatibility with the 2024 competition artifacts. The similarity penalty is calculated as the complement of the ratio of similar test cases from the total number of simulated test cases: #similar tests(Ss ) similarity penalty(Ss ) = 1 − (4) |Ss | 2025: The scoring method for 2025 was updated as follows. The set of 20 tests for simulation (Ss ) were not selected randomly. Instead, the tests for simulation are determined by the entrant’s test generator. For our approach, these are the 20 tests with the maximum test score(t) (The formula for calculating test score(t) did not change in 2025.), where the test score is calculated using the results of the simulations performed during phase three using the simulation budget. Furthermore, tests not selected for simulation are not awarded any score. In contrast to 2024, the 2025 competition did not assign an overall score to each test suite. Instead, participants were ranked separately for each mission based on their test suite score, which is called failure score in 2025, and diversity

overall rank(Ss ) =0.75 × f ailure rank(Ss )

(5)

+0.25 × diversity rank(Ss )

diversity score(Ss ) = 1 −

1 |Ss |

X

similarity(ti , tj )

ti ,tj ∈Ss

(6) where: similarity(ti , tj ) =

|area(ti ) ∩ area(tj )| |area(ti ) ∪ area(tj )|

(7)

2026: The diversity score is split into two components: input diversity, which measures variation in test geometries, and output diversity, which measures variation in the resulting UAV flight paths. The overall ranks of the test suites are calculated as follows: overall rank(Ss ) =0.50 × f ailure rank(Ss ) +0.25 × input diversity rank(Ss ) +0.25 × output diversity rank(Ss ) (8) The selection of the simulation set, Ss , the calculation of test score, and ranking approach follow the same method used in 2025. Input diversity is calculated as: input diversity = Dunion (1 + CV ) Where: Dunion =

A(∪ni=1 Gi) Pn 1/n i=1 A(Gi )

(9) (10)

and the structural variability, CV , is defined as: q Pn 1 2 n i=1 (A(Gi ) − Ā) n−1 1X CV = , whereĀ = A(Gi ) n i=1 Ā (11)

and A(Gi ) is the area of a single obstacle and n is the number of obstacles in Ss . The output diversity of a test suite is calculated from the UAV’s Aerialist simulated flight trajectories in the presence of obstacle placements. For each test, j, and its average simulated flight trajectory, τ¯j , the dynamic time warping (DTW) distance, δj is calculated as the minimum pairwise distance to any other test’s average trajectory τ¯k : m

δj =

min DT W (τ¯j , τ¯k )

k=1,k̸=j

(12)

Output diversity is then defined as the minimum DTW distance, averaged across all tests in the submitted test suite. m

ouput diversity(Ss ) =

1 X δj m j=1

(13)

VI. R ESULTS AND D ISCUSSION The proposed approach demonstrated strong performance across the three competition editions as shown in Table III, Table IV, and Table V. Under the 2024 evaluation criteria, it achieved the highest score for all three missions. In 2025, it obtained the best overall ranking, achieving the highest failure scores for CS2 and CS5 and the second-highest for CS4. In 2026, the approach ranked second overall. A noteworthy observation is that our approach achieved the highest precision in all three benchmark evaluations. We define precision as the proportion of test cases identified as critical by a generator that are subsequently confirmed as critical by the high-fidelity simulator. To account for non-determinism in the simulation environment, criticality is confirmed by taking the average minimum clearance distance from three high-fidelity simulations. In 2024, our approach submitted 228 test cases across the three missions. Of the 60 cases selected for evaluation, 53 were confirmed critical, yielding a precision of 88.3%. This exceeded the 79.2% precision achieved by the next-best competitor in terms of precision, TAIist, which submitted 24 test cases, 19 of which were confirmed critical. In 2025, our approach submitted 146 test cases and achieved a precision of 96.7% (58 out of 60 evaluated cases), compared with 66.7% (40 out of 60) for PALM, which submitted 216 test cases. In 2026, the same 146 test cases and their evaluation from 2025 were reused for benchmarking the precision of our approach. Thus, the 96.7% (58/60) precision from the total of 146 submitted test cases exceeded the 76.8% precision (43 out of 56 evaluated cases) achieved by Intelligent-UAV-Gen, which submitted 159 test cases. Overall, the results show that our approach consistently achieved higher precision than the strongest competing generators across all three benchmarks. Its high precision, together with the large number of generated test cases, indicates that our approach has high repeatability. Another particularly interesting observation concerns the relationship between input and output diversity. In 2026, our

approach achieved the highest input-diversity score for all three missions. This is consistent with the design of the second test-generation phase, which explicitly maximises input diversity. With regard to output diversity, our approach ranked third for CS2 and fourth for CS4 and CS5. The relatively low output diversity of our approach is explained in the discussion of RQ3 in Section VI-C. These contrasting results indicate that generating geometrically diverse obstacle configurations does not necessarily produce diverse UAV flight trajectories. Another observation is that output diversity varies across missions. In 2026, three of the four approaches achieved their lowest output-diversity scores on CS4. One potential explanation is that among the three missions, CS4 has the largest proportion of its planned flight path outside the obstacle placement zone (56.84%, compared with 44.51% for CS2 and 51.84% for CS5). Consequently, a smaller proportion of its route lies within the region where obstacles can be placed, potentially limiting opportunities for obstacle-induced behavioural variation. A. RQ1: How does the Surrogate Prediction Accuracy vary for different flight missions? Prediction accuracy is quantified using the root mean squared error (RMSE) between the NN surrogate’s predicted minimum clearance distance and the ground-truth clearance obtained from Aerialist. The assessment is made using 300 tests, which are the combined set of tests generated by the proposed approach for the 2024 and 2025/2026 benchmarks. None of these tests were included in either the training or validation sets, ensuring the evaluation measures the model’s performance on previously unseen test cases. To provide a reference point, the trained surrogate model was also evaluated on its validation dataset, which was generated by the constrained-random generator for the canonical mission. Comparing these results enables the accuracy of the surrogate model on generated tests to be contrasted with its accuracy on tests drawn from the same distribution as the training process. Mission CS2 CS4 CS5 Validation Set

RMSE 1.35m 0.71m 1.46m 0.50m

TABLE I: Surrogate Model Prediction Error Since missions two, four and five are transformed into the canonical representation prior to prediction, it might be expected that their RMSE values would be similar to that observed for the validation set. However, the RMSE is higher for all three missions than for the validation set. This difference can be explained by how the ground truth clearances are obtained. The validation data were generated for the canonical mission, which contains only a single flight-path segment. However, the benchmark missions may feature multiple flight path segments that pass through the obstacle placement zone.

These additional segments can interact with the obstacles and influence the minimum observed clearance during simulation. Such interactions are not represented in the surrogate model’s training data and therefore are a source of error for its predictions. The characteristics of the three benchmark missions support this explanation. Of the three missions, only mission four has a single flight path segment through the obstacle placement zone (See Figure 3-b). In contrast, mission two crosses its respective obstacle placement zone twice and mission five passes through it three times. As the number of flight-path crossings through the obstacle placement zone increases, so does the opportunity for obstacle interactions that are not represented in the canonical mission training data. B. RQ2: Test-Generation Efficiency The efficiency of a test generation method can be measured by how many critical test cases are generated from a given testing budget. We call this ratio a hit rate. We compare the hit rate of our approach with the constrained-random generator used to create the surrogate model’s training and validation datasets. During dataset generation, the constrained-random generator produced 2716 test cases for the canonical flight mission of which 265 are critical, i.e. it has approximately a 10% hit rate. This dataset does not exclude tests that cause the UAV to exceed the allowed time limit of 500 seconds as specified in the competition scoring criteria. Consequently, the constrained-random generator’s hit rate is an upper limit. In contrast, the surrogate-assisted pseudo-random generator used a total simulation budget of 900 to identify 375 critical test cases across the benchmark missions, corresponding to a hit rate of approximately 42%. The overall hit rate of our approach is calculated as the number of critical test cases identified by our approach divided by the sum of the simulation budgets for all missions for 2024 and 2025/26 benchmarks. The surrogate-assisted approach achieved a critical-test-case hit rate approximately four times that of the constrainedrandom generator. It is also noticeable from Table II that the hit rate for our approach varied by mission. Taking the average hit rate across the 300 simulations for each mission (200 from 2024 and 100 from 2025/26), produced hit rates of 34%, 49% and 42% for missions CS2, CS4 and CS5, respectively. Each of these average hit rates is higher than the hit rate achieved by the constrained-random generator. Note that we compared the efficiency of our approach against a constrained-random generator rather than a purely random generator, as the latter would have produced far fewer critical test cases. The constrained-random generator is more efficient than a purely random generator that does not embed domain knowledge into the generation process. C. RQ3: How Does Canonical Surrogate-Based Filtering Affect Test-Suite Diversity? This research question investigates whether surrogate-based filtering restricts the behavioural diversity of resulting test

suites, given that the model is trained on pseudo-randomly generated test cases for a single canonical flight mission. As discussed in RQ1, the surrogate may produce more accurate predictions for flight missions whose characteristics resemble those of the canonical mission used during training. However, differences in prediction accuracy across missions do not necessarily imply that the filtering process systematically favours particular types of test cases. Prediction accuracy and selection bias are distinct concerns. Nevertheless, our approach achieved one of the lowest behavioural diversity scores among the participants in the 2026 competition. One possible explanation for the low behavioural diversity is the restricted range of obstacle configurations produced by the constrained-random generator in the first phase of our approach. To remain within the surrogate’s training domain, obstacle configurations are restricted to a predefined region relative to the flight path. These constraints limit the variety of configurations that can be generated and may reduce interactions with other flight segments, potentially resulting in less diverse UAV trajectories. Another possible explanation concerns obstacle size. Our approach generated comparatively small obstacles, which may cause relatively minor deviations from the intended flight path and reduce the likelihood of interactions with other flight segments. Overall, the relatively low behavioural diversity achieved by our approach may be associated more strongly with the restrictions imposed during initial test-case generation than with surrogate-based filtering. However, confirming the effect of these restrictions on diversity would require comparing the output diversity of two test suites, one generated by a purely random test generator and the other using the constrainedrandom generator under otherwise identical conditions. VII. C ONCLUSION AND F UTURE W ORK This paper investigated a symmetry-aware surrogateassisted approach for simulation-based testing of autonomous UAVs. By transforming obstacle configurations into a canonical flight frame, the approach enables a neural-network surrogate trained on a single reference mission to be reused across different flight missions without retraining. The surrogate estimates the minimum clearance distance between the UAV and surrounding obstacles, allowing potentially critical test cases to be identified before executing computationally expensive high-fidelity simulations. The empirical evaluation demonstrated the effectiveness of the approach in generating critical test cases. Compared with the constrained-random generator used to produce the training data, the surrogate-assisted approach increased the critical test-case hit rate from approximately 10% to 42%. It also achieved competitive results against participants in the 2024–2026 UAV Testing Competitions, ranking first overall in 2024 and 2025 and second in 2026. However, the results also revealed limitations in transferring surrogate predictions across missions with different flight-path characteristics. Furthermore, although the approach achieved high input diversity,

TABLE II: Test case outcomes and hit rates across benchmark years and flight missions. Benchmark Year

Simulation Budget

Flying Mission

Number of Test Cases Timed Out Non-Critical Critical

Hit Rate (%)

2024

200 200 200

CS2 CS4 CS5

22 1 15

116 111 106

62 88 79

31 44 40

2025/2026

100 100 100

CS2 CS4 CS5

17 5 13

43 36 40

40 59 47

40 59 47

TABLE III: Benchmark results for the 2024 evaluation. Each entry reports the number of test cases (critical test cases) or the score (score rank).

CS2

CS4

CS5

Overall

Tool

Tests (critical)

Score (rank)

Tests (critical)

Score (rank)

Tests (critical)

Score (rank)

Score sum (rank sum)

Our approach WOGAN-UAV CAMBA TUMB DeepHyperion AmbieGen TAIist Surrealist

62 (19) 61 (14) 36 (7) 69 (2) 2 (2) 30 (12) 7 (7) 10 (?)

18.27 (1) 12.55 (2) 11.84 (3) 0.12 (8) 1.22 (6) 2.82 (5) 0.81 (7) 5.03 (4)

88 (18) 81 (4) 33 (0) 135 (16) 10 (8) 36 (4) 6 (4) 10 (?)

20.32 (1) 2.35 (4) 0.00 (7) 7.12 (3) 7.74 (2) 0.86 (5) 0.43 (6) 0.00 (7)

78 (16) 39 (12) 11 (1) 114 (7) 22 (4) 65 (5) 11 (8) 1 (?)

19.21 (1) 8.40 (2) 3.16 (3) 2.73 (4) 1.08 (7) 1.22 (6) 1.57 (5) 0.75 (8)

57.80 (3) 23.30 (8) 15.00 (13) 9.97 (15) 10.04 (15) 4.90 (16) 2.81 (18) 5.78 (19)

Note: “?” indicates that the number of critical test cases was not available in the 2024 competition artifacts.

TABLE IV: Benchmark results for the 2025 evaluation. Each entry reports test cases (critical test cases) or a failure/diversity score (rank).

Tool

Tests (critical)

CS2

CS4

CS5

Failure (rank)

Diversity Tests Failure Diversity Tests (rank) (critical) (rank) (rank) (critical)

Failure (rank)

Diversity (rank)

Rank value

47 (19) 10.89 (1) 17 (5) 2.71 (3) 37 (6) 2.93 (2) 6 (0) 0.00 (6) 76 (19) 0.80 (4) 9 (1) 0.11 (5) 2 (0) 0.00 (6)

0.93 (1) 0.82 (5) 0.85 (4) 0.89 (2) 0.46 (6) 0.18 (7) 0.89 (2)

4.75 8.5 10.75 13.00 15.00 15.00 16.00

Our approach 40 (20) 11.90 (1) Pseudo-Random 31 (7) 6.54 (3) Evolv-1 57 (12) 9.02 (2) TGen-UQ 5 (1) 0.50 (5) PALM 58 (1) 0.14 (7) Surrealist 52 (11) 4.16 (4) OptObstacles 3 (1) 0.44 (6)

0.94 (2) 0.86 (5) 0.88 (4) 1.00 (1) 0.57 (6) 0.21 (7) 0.93 (3)

59 (19) 66 (12) 22 (2) 6 (1) 82 (20) 9 (3) 7 (0)

this did not consistently translate into high diversity in the resulting UAV trajectories. Future work will investigate integrating the trained surrogate into search-based and evolutionary test-generation techniques. Since the surrogate can estimate obstacle clearance distance at a fraction of the computational cost of high-fidelity simulation, it offers the potential to evaluate large populations of candidate test cases efficiently and guide the search towards critical scenarios. Another direction is to investigate neural-network-based generative approaches, including generative adversarial networks, for learning to produce critical

9.14 (2) 9.30 (1) 1.08 (6) 1.44 (5) 3.58 (3) 1.93 (4) 0.00 (7)

0.85 (4) 0.87 (3) 0.82 (5) 0.94 (1) 0.42 (6) 0.20 (7) 0.92 (2)

Overall

obstacle configurations. The trained surrogate could serve as a computationally inexpensive evaluator to guide these generators towards failure-revealing scenarios. Finally, we plan to explore combining large language models with the surrogate to generate and evaluate critical test cases, exploiting the complementary capabilities of language models for scenario generation and neural-network surrogates for rapid prediction of test outcomes. Future work will also investigate improving surrogate generalisation across missions by incorporating flight-context information and accounting for interactions with multiple flight-path segments.

TABLE V: Benchmark results for the 2026 evaluation. Each entry reports test cases (critical test cases) or a failure/diversity score (rank). CS2 D in (rank)

CS4 D out (rank)

Tests Failure (critical) (rank)

D out (rank)

Rank value

Intelligent-UAV-Gen Our approach TLQD Surrealist

66 (20) 19.23 (1) 10.74 (2) 4.13 (2) 16 (4) 3.12 (2) 6.89 (3) 3.07 (2) 77 (19) 25.33 (1) 8.64 (2) 3.11 (3) 40 (20) 11.90 (3) 39.11 (1) 2.72 (3) 59 (19) 9.14 (1) 16.85 (1) 1.37 (4) 47 (19) 10.89 (2) 32.76 (1) 2.41 (4) 100 (7) 6.66 (4) 9.79 (3) 23.31 (1) 100 (4) 1.11 (4) 16.74 (2) 14.44 (1) 100 (10) 9.13 (3) 4.74 (3) 39.62 (1) 80 (17) 18.25 (2) 2.80 (4) 1.00 (4) 13 (4) 2.44 (3) 2.68 (4) 2.67 (3) 79 (0) 0.00 (4) 3.00 (4) 4.26 (2)

5.50 6.50 8.25 9.75

R EFERENCES [1] M. Lam, J. Herrera, S. S. Afzal, K. Zhou, and F. Adib, “MiNav: Autonomous drone navigation indoors using millimeter-waves,” Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, vol. 9, no. 3, pp. 97:1–97:32, 2025. [2] X. Lin, C. Wang, K. Wang, M. Li, and X. Yu, “Trajectory planning for unmanned aerial vehicles in complicated urban environments: A control network approach,” Transportation Research Part C: Emerging Technologies, vol. 128, p. 103120, 2021. [3] D. Oh and J. Han, “Smart search system of autonomous flight UAVs for disaster rescue,” Sensors, vol. 21, no. 20, p. 6810, 2021. [4] S. Nejati, L. Sorokin, D. Safin, F. Formica, M. M. Mahboob, and C. Menghi, “Reflections on surrogate-assisted search-based testing: A taxonomy and two replication studies based on industrial ADAS and Simulink models,” Information and Software Technology, vol. 163, p. 107286, 2023. [5] H. Beglerovic, M. Stolz, and M. Horn, “Testing of autonomous vehicles using surrogate models and stochastic optimization,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2017, pp. 1–6. [6] PX4 Autopilot Project, “PX4-Avoidance: Obstacle detection and avoidance,” GitHub repository, 2024. [Online]. Available: https: //github.com/PX4/PX4-Avoidance [7] S. Khatiri, P. Saurabh, T. Zimmermann, C. Munasinghe, C. Birchler, and S. Panichella, “SBFT tool competition 2024: CPS-UAV test case generation track,” in Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing (SBFT 2024). ACM, 2024, pp. 29–32. [8] S. Khatiri, T. Zohdinasab, P. Saurabh, D. Humeniuk, and S. Panichella, “SBFT tool competition 2025 - UAV testing track,” in 2025 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT), 2025, pp. 5–8. [9] ——, “ICST tool competition 2025 - UAV testing track,” in 2025 IEEE Conference on Software Testing, Verification and Validation (ICST), 2025, pp. 815–818. [10] R. E. Uysal, A. Javadi, P. Aryan, A. A. Babikian, D. Humeniuk, S. Khatiri, and S. Panichella, “SBFT tool competition 2026 - UAV testing track,” in Proceedings of the 19th International Workshop on Search-Based and Fuzz Testing, 2026, pp. 6–10. [11] R. E. Uysal, G. Loubet-Bonino, A. Javadi, P. Aryan, A. A. Babikian, D. Humeniuk, S. Khatiri, and S. Panichella, “ICST tool competition 2026 - UAV testing track,” in 2026 IEEE International Conference on Software Testing, Verification and Validation (ICST), 2026, pp. 513–516. [12] T. Zohdinasab and A. Doreste, “DeepHyperion-UAV at the SBFT tool competition 2024 – CPS-UAV test case generation track,” in Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing (SBFT ’24). ACM, 2024, pp. 49–50. [13] M. De Liso and Z. W. Soi, “CAMBA CPS-UAV at the SBFT tool competition 2024: CAMBA: Cost-aware mutation-based test case generation for unmanned aerial vehicles,” in Proceedings of the 17th ACM/IEEE International Workshop on Search-Based and Fuzz Testing (SBFT ’24). ACM, 2024, pp. 47–48. [14] S. Tang, Z. Zhang, A. Cetinkaya, and P. Arcaini, “TUMB at the SBFT 2024 tool competition – CPS-UAV test case generation track,” in Proceedings of the 17th ACM/IEEE International Workshop on SearchBased and Fuzz Testing (SBFT ’24). ACM, 2024, pp. 53–54. [15] A. Shrinah and K. I. Eder, “Pseudo-random UAV test generation using low-fidelity path simulator,” in 2025 IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT). IEEE, 2025, pp. 51–52.

D out (rank)

Tests Failure (critical) (rank)

D in (rank)

Overall

Tool

Tests Failure (critical) (rank)

D in (rank)

CS5

[16] T. Zhu, A. Skapars, F. Mackenzie, D. Kehoe, W. Newton, S. Embury, and Y. Sun, “SAFLITE: Fuzzing autonomous systems via large language models,” arXiv preprint arXiv:2412.18727, 2024. [17] J. Peltomäki and I. Porres, “Learning test generators for cyber-physical systems,” arXiv preprint arXiv:2410.03202, 2024. [18] F. U. Haq, D. Shin, and L. Briand, “Efficient online testing for DNNenabled systems using surrogate-assisted and many-objective optimization,” in Proceedings of the 44th International Conference on Software Engineering (ICSE). ACM, 2022, pp. 811–822. [19] C. Menghi, S. Nejati, L. C. Briand, and Y. Isasi Parache, “Approximation-refinement testing of compute-intensive cyber-physical models: An approach based on system identification,” arXiv preprint arXiv:1910.02837, 2019. [20] Z. Zhong, G. Kaiser, and B. Ray, “Neural network guided evolutionary fuzzing for finding traffic violations of autonomous vehicles,” arXiv preprint arXiv:2109.06126, 2021. [21] T. Zohdinasab, V. Riccio, and P. Tonella, “Focused test generation for autonomous driving systems,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 6, Jun. 2024. [22] T. Cohen and M. Welling, “Group equivariant convolutional networks,” in Proceedings of the 33rd International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 48. PMLR, 2016, pp. 2990–2999. [Online]. Available: https: //proceedings.mlr.press/v48/cohenc16.html [23] A. Bietti, L. Venturi, and J. Bruna, “On the sample complexity of learning under geometric stability,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 34, 2021. [Online]. Available: https://proceedings.neurips.cc/paper files/ paper/2021/hash/9ac5a6d86e8924182271bd820acbce0e-Abstract.html [24] S.-O. Kaba, A. K. Mondal, Y. Zhang, Y. Bengio, and S. Ravanbakhsh, “Equivariance with learned canonicalization functions,” in Proceedings of the 40th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 202. PMLR, 2023, pp. 15 546–15 566. [Online]. Available: https://proceedings.mlr. press/v202/kaba23a.html [25] R. Wang, R. Walters, and R. Yu, “Approximately equivariant networks for imperfectly symmetric dynamics,” in Proceedings of the 39th International Conference on Machine Learning (ICML), ser. Proceedings of Machine Learning Research, vol. 162. PMLR, 2022, pp. 23 078–23 091. [Online]. Available: https://proceedings.mlr.press/ v162/wang22aa.html [26] S. Khatiri, S. Panichella, and P. Tonella, “Simulation-based testing of unmanned aerial vehicles with Aerialist,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). ACM, 2024, pp. 134– 138. [27] A. Shrinah, C. Bennett, and K. Eder, “Symmetry-Aware SurrogateAssisted Testing of UAVs: Datasets and experimental artifacts,” Zenodo, 2026, research dataset and experimental artifacts. [Online]. Available: https://doi.org/10.5281/zenodo.23876845

Related documents

• SHIFT: Surface-aware High-speed Integration For TSDFs#974661
• PlannerForge: LLM Agents for Scenario-Based Testing of Motion Planners in Autonomous Driving#986584
• HumanHalo: Safe and Efficient 3D Navigation Among Humans via Minimally Conservative MPC#1129128
• Symmetry-Aware Steering of Equivariant Diffusion Policies: Benefits and Limits#1490420
• DR-VLA: Learning Human-Like Quadruped Navigation from Egocentric Web Videos via Domain-Invariant Scene Representations#1343884