arXiv CSopen access

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos Sayed Amir Mousavi and Xiaoyin Wang Department of Computer Science, University of Texas at San Antonio San Antonio, Texas, USA {seyedamir.mousavi,xiaoyin.wang}@utsa.edu

arXiv:2605.16544v1 [cs.SE] 15 May 2026

Abstract As Augmented Reality (AR) becomes more and more embedded in daily life, ensuring the quality, safety, and reliability of AR applications is increasingly important. However, AR apps present unique challenges for automated testing. Unlike static GUI layouts in traditional mobile apps, AR apps acquire their interaction interface from the surrounding environment, which is volatile and non-deterministic. Recent advancements like ARCore Playback and ARKit Replay allow developers to reuse real-world scenarios by recording and playing back enriched videos, enabling more feasible automated AR testing. However, using playback videos introduces two major challenges: test inputs must be timed precisely, and interactive areas in the video are dynamic, irregular, and difficult to identify. To address these challenges, we propose TARIPlay, a framework that analyzes playback videos to detect, track, and filter proper interactive areas over time for automated testing. In particular, TARIPlay identifies viable test opportunities based on criteria like stability and visibility, then feeds this information to an automated testing engine to simulate user interactions. We perform an experiment with four open-source AR apps and nine playback videos. Evaluation results show that TARIPlay significantly outperforms the existing tool Monkey in test coverage (55.8% over 41.98% on branch coverage) of AR-related code, and can also be used to assess the quality of playback videos for testing suitability.

CCS Concepts • Software and its engineering → Software testing and debugging; • Social and professional topics → Software maintenance; • Human-centered computing → Mixed / augmented reality.

Keywords Augmented Reality, Automated Testing, PlayBack Videos

1

Introduction

Augmented Reality (AR) overlays digital content onto real-world views in real-time for immersive user experiences. Applications such as Pokémon Go [31], Amazon [2], IKEA Place [21], and Google Lens [13] demonstrate AR’s transformative potential across gaming, education, healthcare, and retail [34]. As AR apps get more and more involved in daily activities, their quality, safety, and reliability attract more and more attention. Due to AR apps’ real-time interactions with complex environmental contexts in the real world, their bugs may cause more severe consequences, even immediate safety Preprint. Accepted at ICSE 2026 – 2026 IEEE/ACM 48th International Conference on Software Engineering, April 12–18, 2026, Rio de Janeiro, Brazil. DOI: 10.1145/3744916.3787817

risks. For instance, in AR navigation applications, when imprecise content placement or improper occlusion handling are performed in the code, virtual elements may occlude critical aspects of the physical environment, leading to compromised user decisions and immediate risk for traffic accidents [35]. Similarly, a flaw in an AR-supported surgery system may lead to severe harm to patients if giving inaccurate guidance [22]. These real-world implications underscore the importance of robust testing and quality assurance practices in the development of AR apps. Despite the demand of high software quality, existing testing support for AR apps is still at its early stage. Traditional automated GUI testing frameworks fall short due to their inability to handle AR apps’ interactions with real-world scenes [28]. As a result, AR developers often have to rely on manually constructed testing scenes and manual test execution [28], which are laborintensive and inefficient. Simulated environments such as Unity Mars [48] have been recently developed, but it not yet clear how effective virtual reality scenes may simulate real-world scenes in AR testing, because they lack the noises and imperfections of the reality (e.g., coarse surfaces, diffuse reflections). Recently, ARCore [11] and ARKit [4] both introduced recording and playback capabilities (i.e., ARCore Playback [12] and ARKit Replay [3]) to enable scenario reuse. These new features allow mobile phone users to record a video (enriched with sensor information) of the real world and run AR applications directly on the video. With these new features, the automatic execution of test cases for AR apps finally become feasible, and datasets of playback videos (e.g., ARTBank [56]) have been constructed to facilitate AR app testing. However, testing with playback videos is a setting much different from GUI testing [17, 27, 38], raising two novel challenges for automating the testing process. First, in automatic GUI testing, the testing process is fully controlled by the driver and we only need to feed a sequence of GUI events in order without worrying much about the time gap between them. But playback videos are strictly time-constraint, so test opportunities only occur at certain time period of a video. For example, a detected plane may show up at the 10𝑡ℎ second of the video and disappear at the 21𝑠𝑡 second of the video. So an input event must be fed within the that time period to trigger a related app behavior. Second, unlike traditional GUI with predefined UI elements (e.g., buttons, checkboxes) and layouts, interactive areas (e.g., planes, human faces, collectively called trackables) in a playback video are dynamically detected by the underlying AR framework so both their locations and bounding boxes are rapidly changing over time. Even worse, some interactive areas in the video may be of irregular shapes (e.g., a very narrow strip), moving fast on the screen, or disappearing quickly, so they may

Amir et al.

• An analysis and summarization of major challenges in playback-video-based AR app testing. • Novel techniques to identify test opportunities from playback videos by tracking the size, location stability, and life span of interactive areas. • An evaluation of our framework TARIPlay in the setting of automatic AR App testing, validating the strength of extracted test opportunities on enhancing code coverage (especially on AR interaction code) and evaluating playback videos for testing suitability. Figure 1: Dynamics of Planes in 2 seconds not be proper interactive areas for testing (and human users typically do not interact with them). To illustrate the two challenges, in Figure 1, we show the dynamics of a detected plane over 2 seconds in a playback video. In the figure, the table top plane moves and disappears quickly and may not be a good interactive area to target in testing. Therefore, without knowing the time period, location, and size information about proper interactive areas in playback videos, it would be very difficult to automate AR testing with them. VR testing tools (e.g., VRTest [60], VRGuide [53]) cannot be applied to testing with playback videos because they rely on the flexible control of camera movement and angle to explore the virtual scene, and their goal is to create the most efficient exploration paths of the camera. However, for playback videos, once they are created, the camera’s exploration path becomes fixed, and the goal of testing tools becomes maximizing the interaction frequency and variety on the fixed exploration path. In this paper, we present TARIPlay, a Test framework for AR applications based on Interactive area tracking in Playback videos. In particular, TARIPlay addresses the special challenge of playbackvideo-based AR testing (compared with virtual-scene-based AR testing) by extracting test interfaces from videos. Given a Android ARCore playback video, our framework first uses the API of underlying ARCore session to extract all interactive areas within each frame of the video. Then, it uses a novel algorithm to combine interactive areas across frames and identify interactive areas that appear over a certain time period. After that, the framework filters the interactive areas with a number of criteria based on visibility, stability, and life span, to finally identify proper interactive areas as test opportunities. Information about these test opportunities are then fed to an automatic test engine to trigger UI events accordingly. Note that although our framework is mainly designed for automatic testing and we evaluate it accordingly, the information can also benefit test code developers so that they can tell when and where to trigger a UI event in their scripts. We evaluated TARIPlay on four open source AR apps and nine playback videos which form the benchmark video set of ARTBank [56]. We compared TARIPlay with Monkey, which is the only existing tool applicable to AR app testing. Our evaluation shows that TARIPlay achieves average branch coverage of 55.80%, compared with 41.98% of Monkey. Our experiment also shows that, the number and quality of test opportunities extracted from a playback video are correlated with the test coverage on it, indicating that our extracted test opportunities can be used as a quality metrics for playback videos and filter out those of low quality and not suitable for testing. In particular, our paper makes the following major contributions.

2

Background

In this section, we will introduce some background knowledge about the loading of AR playback videos into Android ARCore sessions, and the polygon transformation algorithms which we use for detecting interactive areas from video frames. We expect the information will help with better understanding of our approach.

2.1

Playback Videos in ARCore Sessions

On an Android phone with ARCore support, playback videos can be recorded within an ARCore session using API methods startRecording() and stopRecording() of class ARCoreSession. The API methods will create an enriched MP4 video stream file which stores various data types, including the camera’s video stream, phone sensor data, and analyzed environment information. The recorded playback videos can be loaded back into an ARCore session through API method setPlaybackDatasetUri(). After the loading, the underlying ARCore system will treat the videos as runtime camera inputs and any Android AR apps can be executed upon them. When a playback video is loaded into the ARCore session, we can use various API methods provided by ARCore to extract all detected trackables (interactive areas in the real world space, such as a plane or a human face) from the current frame of the video. A detected trackable is typically presented as a polygon, with the coordinates (in the real-world space with the camera as the origin) of its geometric center and boundary points available. TARIPlay is built upon these extracted data with follow-up data processing techniques to identify test opportunities.

2.2

Polygon Geometric Algorithms

Polygon geometric algorithms are a set of computer geometry algorithms used for solving questions about polygons. In our approach, we need to use a 3D vertex projection algorithm to map a polygon(i.e., a detected interactive area from the 3D space of the ARCore session) in real-world space to a 2D polygon on the phone screen plane. Since the mapped 2D polygon may go outside the boundary of the screen, we further need to use a polygon clipping algorithm to acquire the part of the polygon visible on the phone screen. The 3D vertex projection algorithm [20] maps 3D points from a virtual scene onto a 2D display. The algorithm rely on a camera projection matrix, which combines intrinsic parameters (like focal length and optical center, saved in the playback video) and extrinsic parameters (such as position and orientation) to transform 3D world coordinates into 2D screen coordinates. In our approach, we choose to use the 3D vertex project algorithm implemented in the ARCore framework to map polygon vertices to the phone screen.

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

Figure 2: Illustration of Steps in Sutherland-Hodgman Algorithm (Top Left to Bottom Right) The Sutherland–Hodgman algorithm [47] is a classic computer graphics algorithm used for polygon clipping, particularly to clip a polygon against a convex clipping window, such as a rectangle. For each edge of the clip boundary, the algorithm examines each pair of adjacent vertices in the polygon and determines whether the edge between them lies entirely inside, entirely outside, or partially inside the clip edge, and then generates new vertices accordingly. Figure 2 illustrates the algorithm on clipping a polygon “W” with a 5-sided polygon.

3

Approach

In this section, we introduce our TARIPlay framework in details. TARIPlay takes a playback video as its input and identify a set of test opportunities (proper interactive areas with large enough size, stable position, and long enough life span). The overview of TARIPlay is presented in Figure 3. From the figure, we can see that TARIPlay has four major steps: (1) using ARCore to split the playback video to a number of frames, (2) extracting trackables (interactive areas in the real-world space) from each frame, (3) performing visibility analysis to convert trackables to interactive areas on the phone screen, and (4) performing life span analysis to identify stable interactive areas on the phone screen and output them as test opportunities.

3.1

Video Loading and Frame Sampling

In the first step, we need to create an ARCore session to load the playback video. To achieve this, we create an empty Android AR app (as a part of TARIPlay whose only function is creating an ARCore session, loading a playback video and then extracting information about trackables from frames. TARIPlay also relies on ARCore’s playback video loader to normalize frames to a standard aspect ratio (1920 × 1080), ensuring consistency between different devices. The remaining analyses are performed offline after the playback video finishes. Since consecutive frames are very similar to each other, it is not necessary to analyze every frame. Therefore, TARIPlay samples frames at 10 frames per second (fps). We choose 10 fps as our sampling rate because ∼100 ms is a commonly reported lower bound for human visual recognition, so interactive areas with shorter lifetime are unlikely to be perceived [50] [49]. Our sampling reduces computational processing overhead and has an additional benefit to ignore some noises caused by erroneously rendered or missed single frame, yet we did not observe any loss of valid interactions in our experiments.

3.2

Trackable Extraction

From each sampled frame, TARIPlay extracts trackables available at the frame using ARCore API method

session.getAllTrackables(), and then we filter out trackables that are not tracked on the frame due to occlusions or tracking failures using below condition: plane.getTrackingState() == TrackingState.TRACKING. For each qualified trackable, TARIPlay collects the trackable ID, extents in all axes, and coordinates of the vertices and the center. Note that the trackable ID is used to link trackables across different frames. Based on the collected information, TARIPlay further filters out trackables not facing the camera so it is impossible to interact with them. Specifically, we filter out a trackable if the following condition is true: n · (c − p) < 0

(1)

, where n is the trackable’s normal vector, c is the camera’s position, and p is the trackable’s center. TARIPlay currently considers only planes and filters out other types of trackables, because those trackables are not available in playback videos. Human faces are the second most commonly used trackables second to planes [51], but due to privacy concerns, few people are sharing playback videos with human face trackables, and we do not find such videos from the ARTBank [56] dataset. That being said, since human faces and other trackables such as pictures can also be modeled as polygons, our approach automatically applies to them as long as we do not filter them out.

3.3

Visibility Analysis

After TARIPlay extracts all the trackables from a frame, it will perform visibility analysis to (1) calculate the projection of each trackable on the phone screen plane, (2) clip each projection area based on the phone screen scope and other overlapping projection areas, and (3) regularize the clipped projections from polygons to their inscribed rectangles. Using the inscribed rectangles as an approximation is an important design decision we make to simplify the technique. Although the inscribed rectangle may underestimate the interactive area of a trackable projection [26], this is a conservative approximation because UI events performed on the inscribed rectangle will be definitely received by the trackable. Furthermore, since testing tools typically send UI events to the center of an interactive area, the approximation will have only minimal influence on the testing process. On the other hand, using inscribed rectangle has large benefit as it allows easy and efficient calculation of area intersections, which is essential for the following life span analysis. The three steps in our visibility analysis are detailed below. • Trackable Projection: To perform the projection, we first extract all polygon vertices from a trackable’s representation, where each vertex vlocal = (𝑥, 𝑦 = 0, 𝑧, 𝑤 = 1)⊤ is defined in the plane’s local coordinate system. Then, we project vertices to phone screen plane using the ModelView-Projection matrix chain: pscreen = P · V · T · vlocal , where P is the projection matrix, V is the view matrix, and T is the plane’s pose transformation matrix. Note that we can acquire all the matrices from ARCore API methods. Finally, we convert 𝑝 screen to screen coordinates with

Amir et al.

Figure 3: Overview of TARIPlay visible polygon.

Android Y-axis correction:

𝑥 raw = {min(xvisible ), max(xvisible )}

(𝑥 ndc + 1) ·𝑊  2  𝑦𝑛𝑑𝑐 + 1 𝑦screen = 1 − ·𝐻 2 𝑥 screen =

𝑦raw = {min(yvisible ), max(yvisible )} After that, for each vertex of the bounding box, we iteratively apply conservative shrinkage as below to shrink the bounding box to an inscribed rectangle. Note that a bound (e.g., (𝑥 min ) is moved until both vertices on its side (e.g., the two left vertices for 𝑥 min ) go inside the polygon.

where 𝑊 and 𝐻 are screen dimensions, and (𝑥 ndc, 𝑦ndc ) = (𝑥 screen /𝑤 screen, 𝑦screen /𝑤 screen ). After all vertices of a trackable polygon are projected to the phone screen plane, we can connect them to form the projection area. • Polygon Clipping of Projection Areas: after projected areas are calculated, TARIPlay further performs polygon clipping (using Sutherland-Hodgman algorithm) to acquire the visible parts (i.e., areas that are within the scope of the phone screen and not occluded by other areas) of projected areas, which we refer to as visible polygons later in the text. For clipping into the screen rectangle, we use the vertices vector {(0, 𝐻 ), (𝑊 , 𝐻 ), (𝑊 , 0), (0, 0)} in clockwise order. For each clipping edge, TARIPlay computes line intersections using parametric equations: 𝑡=

𝑥 min = max(0, 𝑥 raw,min + 0.025 · Δ𝑥) 𝑥 max = min(𝑊 , 𝑥 raw,max − 0.025 · Δ𝑥) 𝑦min = max(0, 𝑦raw,min + 0.025 · Δ𝑦) 𝑦max = min(𝐻, 𝑦raw,max − 0.025 · Δ𝑦) where Δ𝑥 = 𝑥 raw,max − 𝑥 raw,min and Δ𝑦 = 𝑦raw,max − 𝑦raw,min . After this process, we acquire an inscribed rectangle for each visible polygon, and we refer to these inscribed rectangles as visible boxes later in the paper. The choice of 0.025 for bound reduction step is a tradeoff between performance and accuracy. We rely on Google ARCore to identify trackables and our on-the-fly analysis needs to keep up with the speed of video processing. Smaller steps improve accuracy but lead to more computation and sometimes video lags / crashes, while larger steps cause more inaccuracy of the calculated bounding box.

(𝑥 1 − 𝑥 3 )(𝑦3 − 𝑦4 ) − (𝑦1 − 𝑦3 )(𝑥 3 − 𝑥 4 ) (𝑥 1 − 𝑥 2 )(𝑦3 − 𝑦4 ) − (𝑦1 − 𝑦2 )(𝑥 3 − 𝑥 4 )

, where (𝑥 1, 𝑦1 ) and (𝑥 2, 𝑦2 ) define the polygon edge, and (𝑥 3, 𝑦3 ) and (𝑥 4, 𝑦4 ) define the clipping edge. To handle the occlusion, we use the same clipping approach. In particular, TARIPlay keeps track of distance of vertices to the camera, and ordered them from the closest to the farthest. For a given projection area, TARIPlay first identifies polygon edges (from all projection areas) that have at least one end point between the projection area’s original plane and the phone, and then uses these edges to perform clipping on the projection area. Finally, the clipped area are removed and the resulted potentially concave area (if still existing) is partitioned to convex partitions. For efficiency purpose, we process the projection areas from the closest (to the phone) to the farthest. • Calculation of the Inscribed Rectangle: After all the visible polygons are acquired, TARIPlay further calculates their inscribed rectangles to facilitate later analyses. To achieve this goal, for a given visible polygon, it first extracts axis-aligned bounds from the polygon vertices as below, where (𝑥 visible , 𝑦visible ) is the coordinates of a vertex of the

After TARIPlay completes the visibility analysis and acquires all the visible boxes for a frame, it will filter out all the visible boxes that are not large enough. We define Visibility Ratio as below, and filter out all the visible boxes with Visibility Ratio lower than 10%. This default threshold is set according to the general guideline [32] for mobile UI design about the size of interactive areas which reliably support multi-finger interactions such as pinch and swipe. The movement and deformation of detected trackables in AR apps also call for larger interactive areas for stable interaction. 𝐴box 𝐴screen , where 𝐴screen is the screen area in pixels, and 𝐴box is the area of the visible box in pixels. Visibility Ratio =

3.4

Life Span Analysis

While the size of visible boxes are important for effective testing, their life span is also important. If a visible box appears on the

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

Algorithm 1 Algorithm for Life Span Analysis Input: VisibleBox, AllFrames Output: LifeSpan, StableBox 1: Let 𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥 = 𝐹𝑢𝑙𝑙𝑆𝑐𝑟𝑒𝑒𝑛 2: Let 𝐿𝑖 𝑓 𝑒𝑆𝑝𝑎𝑛 = ∅ 3: Let 𝑆𝑡𝑎𝑟𝑡𝑒𝑑 = 𝐹𝑎𝑙𝑠𝑒 4: for 𝐹𝑟𝑎𝑚𝑒 ∈ 𝐴𝑙𝑙𝐹𝑟𝑎𝑚𝑒𝑠 do 5: if 𝑣𝑖𝑠𝑖𝑏𝑙𝑒(𝑉 𝑖𝑠𝑖𝑏𝑙𝑒𝐵𝑜𝑥, 𝐹𝑟𝑎𝑚𝑒) then 6: Let 𝑆𝑡𝑎𝑟𝑡𝑒𝑑 = 𝑇𝑟𝑢𝑒 7: Let 𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥 = 𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥 ∩ 𝑉 𝑖𝑠𝑖𝑏𝑙𝑒𝐵𝑜𝑥 8: if !𝑣𝑖𝑠𝑖𝑏𝑙𝑒(𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥, 𝐹𝑟𝑎𝑚𝑒) then 9: return 𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥, 𝐿𝑖 𝑓 𝑒𝑆𝑝𝑎𝑛 10: end if 11: Add 𝐹𝑟𝑎𝑚𝑒 to 𝐿𝑖 𝑓 𝑒𝑆𝑝𝑎𝑛 12: else if 𝑆𝑡𝑎𝑟𝑡𝑒𝑑 then 13: break 14: end if 15: end for 16: return 𝑆𝑡𝑎𝑏𝑙𝑒𝐵𝑜𝑥, 𝐿𝑖 𝑓 𝑒𝑆𝑝𝑎𝑛

Figure 5: A Exemplar Gantt Chart Output of TARIPlay shorter than two seconds and report the remaining as test opportunities. We choose two seconds as the default threshold because it takes about 0.6 seconds for a human being to reflect on a visual stimulus and give hand response [10] [37], so two is the fewest number of whole second that allows two consecutive UI events to be triggered naturally. In addition, the sudden appearance of AR interactive areas and multi-finger gestures (gestures involving multiple finger movements within a single stable window) both require longer reflection time. Although we use automatic testing and coverage to evaluate TARIPlay, we believe that it is also important that the generated test cases simulate how typical users would interact with the app. Our experiments further studied the effectiveness of interactive area detection with different visibility area and life span thresholds.

3.5

Figure 4: Visible Box Intersection in Life Span Analysis screen, and then disappears (due to lost tracking of the corresponding trackable) or moves outside the phone screen (due to movement of the phone during the video duration) in a short amount of time, it cannot be well leveraged in testing, because a UI event may not be successfully received due to computation delay, for multiple UI event sequences (e.g., tapping to create an object and then pinching to resize), visible boxes with short life span will cause the following event to be infeasible (i.e., the created object is no longer in the screen). Therefore, TARIPlay further calculate the life span of visible boxes and reports only those with long enough life span as test opportunities. The procedure of life span analysis is presented in Algorithm 1, and an illustration is shown in Figure 4. From the algorithm, we can see that, the life span analysis of a visible box starts at the first frame it appears in the playback video. Then, during following consecutive frames, the analysis continuously update the stable visible area and the last-visible frame. If visibility is no longer true (i.e., the stable visible area drops below 10% of the screen) or the playback video ends, the algorithm will end and report all the frames between the starting and ending frames. From Figure 4, we can see that the stable visible area of a visible box shrinks to the box in the middle (marked with diagonal lines) as we intersecting the visible boxes at frames 1, 2, and 3, and that stable visible area is the safest screen area to send UI events during testing. After we acquired life span of all visible boxes, we use the threshold of two seconds to filter out the visible boxes with life spans

Visualization and Interaction Scheduling

To better present the test opportunities reported by TARIPlay, we can further visualize the visible boxes and their life spans in Gantt charts. For the example in Figure 5, we can see that visible boxes are presented as blocks along the time line of the video. If used for manual test code development, developers can utilize the chart to easily decide when to tap on a box and place a 3D object so that they have enough time later to interact with it. They may also easily identify the video period (e.g., second 45 to 60 in the figure) when multiple visible boxes exist so they can test cross-plane actions such as moving an object from one plane to another. It should be noted that, since Google ARCore’s computer vision algorithms bring in randomness, the test opportunities created by TARIPlay may change when executed multiple times. Therefore, TARIPlay will analyze a playback video for three times and calculate the intersection of all test opportunities as its final output.

3.6

Automated AR App Testing with TARIPlay

While TARIPlay can be used to support human test developers to write tests more easily, it also provides a basis for automatic AR apps testing because it allows the test driver to always send UI events to interactive areas on the screen and thus makes the testing more efficient. It may also potentially enable the adaptation of traditional GUI testing strategies [43] to AR because the test opportunities provided by TARIPlay are comparable to the UI layouts provided by UIAutomator [15] (though the time constraints of test opportunities add more complexity and needs additional techniques to handle). In this paper, we will leave the above research opportunities for future and focus on the evaluation of the generated test opportunities themselves for automatic testing. Since Monkey [16] is the only automatic testing tool applicable to AR apps, for fair comparison, we implemented an automatic testing tool also with random strategy but on top of TARIPlay output and use it in our evaluation. In

Amir et al.

particular, our tool generates random UI events with the same distribution as Monkey. The difference is that, at a specific time point, we send the UI events only to coordinates within the visible boxes existing at the time point, according to the output of TARIPlay. For multi-coordinate gestures such as pinch, TARIPlay randomly chooses multiple coordinates within the identified interactive area.

4

Evaluation

We conducted an experiment study to evaluate the effectiveness of our random testing tool based on TARIPlay framework against Monkey, the existing automatic testing tool directly applicable to AR apps.

4.1

Research Questions

The primary goal of our experiment is to find out whether our TARIPlay framework can enhance the effectiveness of automatic testing on AR apps. Therefore, we try to answer the following three research questions. • RQ1: What is the effectiveness of TARIPlay on detecting test opportunities in AR playback videos, with different thresholds of visibility area and life span? • RQ2: How much does TARIPlay help to boost code coverage over Monkey on automatic AR app testing? • RQ3: How does app complexity (e.g., gesture diversity, hybrid views) impact effectiveness of TARIPlay and Monkey? • RQ4: How environmental factors and video qualities may affect achieved code coverage?

4.2

Experimental Setup

Configurations of Compared Approaches. Our experimental design employs a controlled comparison methodology using three distinct testing configurations: (1) the Standard Monkey configuration which executes Monkey testing with 50,000 completely random events and no throttling between actions, (2) the Enhanced Monkey which applies 50,000 events with 100ms throttling and restricts events to taps, swipes, and pinches (parameters: –anyevent=0 –pct-majornav=0, –pct-appswitch=0, –pct-syskeys=0) to approximate more realistic human interaction patterns and prevent excessive navigation events that would exit AR scenes prematurely, and (3) our proposed approach, which employs output of TARIPlay from playback videos, and strategically confines UI events to the detected visible boxes only during their life span. For UI events, we issue random sequence of tapping and gestures similar to Monkey but limit gestures to detected test opportunities. Hardware Environment. The experimental environment utilized a Xiaomi 11T Pro device running Android 14 with testCoverageEnabled = true and ProGuard/R8 disabled to ensure consistent hardware conditions and accurate instrumentation across all test runs. We acquire test coverage through instrumentation based on JaCoCo version 0.8.11 integrated with Gradle 7.3 and Android Debug Bridge version 34.0. Our analysis pipeline leveraged Python 3.11 scripts for statistical computation and visualization generation. Subjects. Our evaluation dataset comprises four open-source AR apps from Github and nine ARCore playback videos from the ARTBank benchmark folder [56], forming 4 × 9 = 36 test settings.

The subject open-source AR apps represent a diverse set of programming language and framework configurations (i.e., Java/Kotlin/C# and ARCore/Sceneform/SceneView). Each app exhibits distinct interaction paradigms and complexity characteristics. This selection encompasses applications ranging from singleview AR experiences to hybrid interfaces combining traditional UI elements with AR components, supporting varying gesture vocabularies from simple tap interactions to complex multi-finger manipulations including rotation and scaling operations. The ARTBank benchmark of nine playback videos represents diverse real-world testing conditions with balanced factor coverage: 3 indoor × 3 outdoor environments across 3 lighting conditions (low, medium, bright) with varying camera movement patterns from static positioning to high-motion scenarios. This stratified dataset allows us to assess our approach’s robustness across heterogeneous deployment conditions commonly encountered in AR applications.

4.3

Metrics and Analysis Framework

Our evaluation employs multiple complementary metrics to comprehensively assess testing effectiveness. Code coverage analysis utilizes five JaCoCo metrics: instruction coverage measuring the percentage of executed bytecode instructions, branch coverage capturing decision path exploration, line coverage indicating source code reach, complexity coverage based on cyclomatic complexity paths, and method coverage tracking invoked functionality. Some AR apps also often have a GUI component to lead users to the AR sessions. Since our focus is on testing AR-related code and ARspecific behavior, we configure Jacoco to not considering Android framework boilerplate and restrict the coverage analysis to only application-specific packages containing AR-related code. Beyond traditional coverage metrics, we introduce the Gesture Success Rate (GSR) as a novel metric specifically designed for AR testing evaluation. GSR quantifies the percentage of attempted gestures (UI input) that successfully trigger intended application responses, computed as the ratio of successful gesture completions to total gesture attempts on all trackables. Success is determined via deterministic application callbacks and logcat analysis that capture recognized gestures through temporal matching between triggered events and application logs. This metric proves particularly valuable for assessing multi-finger interaction reliability, where traditional random testing often fails due to spatial and temporal coordination requirements. Because coverage distributions are non-Gaussian and we have repeated measurements of the same (app, video) scenarios across three methodologies, we use non-parametric repeated-measures procedures throughout. For each coverage metric (Line / Branch / Method), we run a Friedman test (blocks = (app, video) × run; treatments = {Standard Monkey, Enhanced Monkey, TARIPlay}). For pairwise contrasts we use Wilcoxon signed-rank tests with Holm family-wise adjustment across the three methodology pairs. We report Cliff’s 𝛿 (equivalently, paired rank-biserial) as the effect size with bias-corrected and accelerated (BCa) bootstrap 95% CIs (10 000 resamples).

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

Table 1: TARIPlay’s effectiveness on recognizing Test Opportunities Threshold Setting Prec.(L) Rec.(L) Prec.(S) Rec.(S) 2s+10% 79.7% 92.6% 69.1% 96.3% 1s+10% 69.6% 95.4% 58.7% 96.3% 3s+10% 86.6% 78.7% 76.3% 82.0% 2s+5% 66.6% 95.4% 55.8% 96.3% 2s+20% 79.9% 81.0% 69.7% 83.6%

4.4

Results and Analysis

4.4.1 RQ1: Detection of Interactive Areas. To assess the quality of test opportunities detected by TARIPlay , we conducted a manual study on 9 playback videos. Two people (one of the authors and another student who is not an author) independently labeled test opportunities in the videos, so that the test opportunities detected by TARIPlay can be classified as opportunities (labeled as a test opportunity by both people), borderlines (labeled as a test opportunity by only one person), or non-opportunities (labeled as a test opportunity by neither). The study results with different thresholds are presented in Table 1. We keep one threshold at its default value (2s or 10%), and change the other threshold for larger (3s or 20%) or smaller values (1s or 5%). Columns 2-3 presents precision and recall when considering borderlines as opportunities (L for Loose), and Columns 4-5 presents precision and recall when considering borderlines as non-opportunities (S for Strict). The results show precision / recall of 79.7% / 92.6% (Loose) and 69.1% / 96.3% (Strict) using the default setting. Lowering duration/visibility thresholds yields marginal recall gains but noticeably reduces precision; increasing thresholds improves precision at the expense of recall (which is more important in testing), so the default setting (2s, 10%) provides a good balance point. In practice, we observed two common false positive cases: (i) interactive areas that visibly shake due to camera/scene motion, and (ii) very narrow areas that are technically clickable but hard to interact with. 4.4.2 RQ2: Comparison of Coverage. Table 2 presents detailed code coverage comparison across all four applications, comparing Monkey baselines with our TARIPlay approach. We repeated each configuration three times and report the mean coverage in Table 2. To quantify stochasticity, we also calculate across-execution variability over the three runs. For Monkey-S, the variance of line, branch, and method coverages are ±4.21%, ±4.10%, and ±5.10%, respectively. For Monkey-E, the variance of line, branch, and method coverages are ±8.71%, ±8.58%, and ±9.58%, respectively. For TARIPlay, the variance of line, branch, and method coverages are ±0.54%, ±1.06%, and ±2.00%, respectively. The lower variance for TARIPlay is consistent with its use of stable test opportunities. TARIPlay achieves notable improvements across all metrics, with branch coverage showing substantial gains across all applications (+15.72 pp average over Standard Monkey, +13.82 pp over Enhanced Monkey). Method coverage demonstrates consistent improvements with an average of +12.25 pp over Standard Monkey and +10.62 pp over Enhanced Monkey, while line coverage shows meaningful improvement averaging +9.33 over vs Standard Monkey and +10.85 pp over Enhanced Monkey. Statistical Analysis. Our dataset comprises 4 Apps × 9 Videos × 3 Executions = 108 observations (per approach variant per metric).

Omnibus Friedman tests show strong differences among methodologies for all three coverage metrics (𝑝 < 10−13 for Line, Branch, and Method). For our primary contrast (TARIPlay vs. Enhanced Monkey), Wilcoxon signed-rank yields 𝑝 = 2.91 × 10−11 . Corresponding Cliff’s 𝛿 values indicate very large effects: 𝛿 = 0.890 (Line), 0.849 (Branch), 0.836 (Method), all closing to 1.0. While we configure Jacoco to rule out non-AR-related code, there are also AR-related code which is executed all the time, including initialization of AR sessions, permission handling for storage and camera, resource loading, etc. Compared with such code, it is more important to compare the coverage on code that can be triggered only with interactions in AR sessions. For each app, we identify always-executed code by running the app through each playback video without triggering any UI events. When Jacoco provides code coverage at different code granularity levels, we remove the intersections between evaluated runs (Monkey variants and TARIPlay) and no-input runs. The remaining code is considered interaction code which is harder to cover and more important on comparing effectiveness of testing techniques. Figure 6 further compares code coverage on the interaction code. Figure 6 is a group of four box plot (one plot for each app) where each box describes the distribution of interaction code coverage over different playback videos in the corresponding setting. From the figure, we have two observations. First, for each app, our approach outperforms both Monkey variants on almost all videos, showing consistent enhancement also over different videos. Second, our approach achieves a larger coverage improvement on interaction code, show over 20 percentage point gain in all apps except for AR Ecommerce. The figure also shows variance on absolute code coverage and distributions across different apps and coverage criterion, which we detail below. Hello AR: This app supports placing and removing of multiple objects. TARIPlay performed well consistently on the app. While Monkey variants sometimes can achieve acceptable coverage (on videos with abundant test opportunities), the variance is very large as they cannot stably trigger AR interactions. AR Ecommerce: This application allows single object placement (model.setParent(anchorNode)) and maintains object selection for following user interactions with the placed object. The variance is low for all approaches because just catching one test opportunity will allow successful placement of objects. TARIPlay performed still better because it restricted the following interactions within the interactive area, and thus boosted their success rate. AR Builder: This application supports tap, move, and scaling gestures but not rotation. It provides semi-AR functionality including rescaling, object removal, screenshots, color/type selection through popups, and input fields for object dimensions. Monkey performs better in ARBuilder than other apps (still with high variance due to its high randomness) because it supports more Semi-AR (traditional GUI inside AR) features. AR Simulator: This app features placement of a single object and supports dragging through AR gestures for precise movements, plus rotation capabilities similar to AR Builder but limited to one placed object per session. Our approach performs much better than Monkey because it limits gestures into visible boxes so more complicated gesture sequences in the same area are more likely to

Amir et al.

Table 2: Comprehensive Coverage Metrics Comparison Application AR Builder

AR Ecommerce

AR Simulator

HelloAR

Overall

Metric Branch Method Line Branch Method Line Branch Method Line Branch Method Line Branch Method Line

Monkey S 42.44% 71.25% 66.53% 45.00% 71.43% 67.57% 36.96% 58.62% 53.27% 35.92% 70.51% 59.32% 40.08% 67.95% 64.17%

Monkey E 40.90% 70.02% 65.68% 46.67% 72.43% 65.03% 35.99% 58.62% 53.04% 44.35% 77.25% 66.83% 41.98% 69.58% 62.65%

TARIPlay 58.18% 80.42% 77.78% 49.63% 81.07% 69.37% 58.45% 70.50% 68.34% 57.10% 88.76% 78.31% 55.80% 80.20% 73.50%

Δ (pp) +15.74 +9.17 +11.25 +2.96 +8.64 +1.80 +21.49 +11.88 +15.07 +12.75 +11.51 +11.48 +13.82 +10.62 +9.33

Improvement +37.1% +12.9% +16.9% +6.3% +11.9% +2.7% +58.2% +20.3% +28.3% +28.7% +14.9% +17.2% +33.0% +15.1% +14.5%

Figure 6: Coverage Comparison on Interaction Code happen. Our performance varies across videos because some videos do not have enough test opportunities for complicated gestures (e.g., dragging for precise movements) supported in the app. 4.4.3 RQ2: Coverage vs Complexity. Results in Table 2 and Figure 6 show that our approach makes larger improvements on the two

more complicated apps AR Builder and AR Simulator. Statistical analysis reveals relations between application gesture complexity. AR Builder, supporting four distinct gesture types including complex multi-finger operations, shows the largest overall coverage improvements. Hello AR, with its simplified interaction model focused primarily on tap gestures, demonstrates consistent but more

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

Table 3: Gesture Success Rates by Type and Application Type

AR Builder

Tap Drag Rotation Scaling

96% 41% 32% 28%

Tap Drag Rotation Scaling

100% 95% 91% 93%

AR Ecomm. Monkey 98% 45% 39% 34% TARIPlay 100% 92% 88% 89%

AR Simul.

HelloAR

99% 48% – –

97% 42% 35% 29%

100% 96% – –

100% 94% 86% 92%

modest gains. This pattern suggests that our approach provides greatest value for applications implementing sophisticated interaction paradigms that challenge existing testing tools. To better understand the performance different techniques on more complicated gestures, we performed gesture success rate analysis and compare the gesture success rate of different techniques on the four apps. The results are presented in Table 3. From the table, we can see substantial improvements in multifinger interaction reliability when comparing TARIPlay against Monkey. While baseline Monkey achieves acceptable success rates for simple tap gestures at 94.3%, performance degrades substantially for complex interactions. In particular, for Monkey, Drag gesture success rates reach only 39.7%, rotation gestures succeed in merely 34.2% of attempts, and scaling operations achieve only 28.9% success rates. Analysis of Monkey’s limitations reveals three key challenges: (1) Spatial Ignorance—Monkey touches are completely random across the screen without knowledge of placed object locations or anchor positions, failing to target specific anchors for manipulation and lacking understanding of AR spatial relationships; (2) Chaotic Movement Patterns—Monkey generates unpredictable ACTION_MOVE sequences that may not maintain proper singlefinger contact or satisfy hit-testing requirements; (3) Validation Criteria Failures—Even when near anchors, Monkey’s erratic movement patterns may not meet the validation criteria needed for gesture recognition. The challenges are compounded when applications deselect placed objects after placement (Hello AR vs AR Simulator), as 3D raycasting constants (3D bounding spheres or 2D constants) may degrade gesture support if bounding spheres are insufficiently large for normal interaction requirements. TARIPlay addresses these limitations by precisely aligning multifinger events with confirmed stable plane regions and appropriate temporal windows. In particular, drag gesture success rates improve to 92.4%, while rotation and scaling operations achieve 86.8% and 89.3% success rates respectively. Overall gesture success rate increases from 67% with Monkey to 96% with our approach. 4.4.4 RQ3: Video Factor Impact. Figure 7 shows the correlation between achieved code coverage and various factors from the playback videos. Because environmental variables are ordinal, we use Spearman’s rank correlation (𝜌). For each cell we compute the p-value as shown in the brackets. Exogenous variables (to the left of the dashed line in figure 7) are environment factors related to general video features, whereas variables on the right are TARIPlay-derived metrics. Among the endogenous variables, average plane duration (average lifespan of visible boxes) has the largest observed positive

Figure 7: Correlation Between Coverage and Video Factors, *p<0.05, **p<0.01 correlation with branch coverage (𝜌 ≈ 0.73, p ≈ 0.03), suggesting that videos providing longer stable interaction opportunities enable more comprehensive testing. Mutual stability and mean overlaping areas show similar positive trends. Since Google ARCore’s computer vision algorithms bring in randomness, the test opportunities created by TARIPlay may change when executed multiple times. After three executions, we calculate the result similarity as mutual stability, and mean overlapping areas as the average overlapped test opportunities across executions. We can see that when the extracted test opportunities are more stable across executions, the more likely we can achieve a higher code coverage. However, none of these correlations are statistically significant at 𝑝 = 0.05, so we consider these relationships as descriptive patterns rather than definitive correlations. For exogenous environmental factors, we convert lighting conditions (low = 1, hight = 3), environment sizes (small = 1, large = 3), and indoor / outdoor (indoor = 1, outdoor = 2) to numbers and performed the correlation analysis (the labels are provided by ARTBank). From the figure, we can see that lighting condition is slightly correlated with code coverage, but both environment sizes and indoor / outdoor settings have consistently negative 𝜌 values, showing that it is more difficult to achieve high coverage in playback videos of larger spaces and outdoor settings. 4.4.5 UI Events Statistics. TARIPlay demonstrates improvements on the number of UI events triggered compared to Monkey approaches while maintaining superior coverage effectiveness. We set the total number of events (per app per video per execution) to 50,000 to maximize Monkey’s test coverage potential, and 50,000 was never reached before the video stops. Baseline Monkey with no throttle triggers 18,171 events per execution on average . Enhanced baseline with 100ms delays between events triggers 2,182 events per execution on average, while TARIPlay triggers only 198 event per execution on average. Please note that although all executions take exactly the same time (video length), triggering fewer UI events indicates that TARIPlay tests are more meaningful and may be adapted to human tests more easily.

4.5

Threats to Validity

The major threat to construction validity is the coverage-quality assumption that higher code coverage always leads to improved fault detection capability, and it may not be the case [23]. In AR apps, a same UI events on different trackables may trigger exact the same code, but have different visual effects, so sending more UI events to cover more trackables is also important. To mitigate this threat,

Amir et al.

we supplemented coverage analysis with gesture success rates to provide multiple effectiveness perspectives. The major threat to internal validity is the instrumentation bias where JaCoCo coverage measurement may miss dynamically-loaded methods or reflectionbased code execution. To mitigate this threat, we manually verified the class loader runtime logs and confirmed that all the loaded classes are from the original source code package the corresponding executions are recorded in the coverage reports. Another threat to internal validity is the randomness in evaluation results caused by non-determinism in TARIPlay. To mitigate this threat, we run three executions for both the test opportunity detection and the app testing. For further mitigation, we plan to run more executions and investigate how many executions are sufficient for the evaluation of AR testing referring to the guidelines for randomized algorithm in testing [5]. The major threat to the external validity is that our evaluation is limited to the Android ARCore framework, the Android ARCore Apps / playback videos , or just to the subject apps / playback videos / mobile device used in the experiment. Therefore, our findings may not be generalized to other apps / videos and Apple ARKit and its corresponding playback videos. To mitigate this threat, we used four apps that cover multiple AR domains and interaction styles. We include single-view, pure-AR experiences alongside AR with extensive UI components. Interaction styles range from taponly object placement to persistent multi-finger object engagement. The apps span retailing, education, and simulation domains. The nine videos are the benchmark provided in ARTBank and were chosen to cover different scene sizes, indoor/outdoor settings, and lighting conditions. Correlation and effect-size claims on a relatively small dataset (4 apps × 9 videos) risk low power and inflated Type-I error under multiple comparisons. For coverage comparisons we use non-parametric repeated-measures tests with Holm correction and report distribution-free effect sizes with uncertainty. In the future, we plan to mitigate this threat by performing experiment on more apps / playback videos and ARKit apps / playback videos.

4.6

Replication Package

To support reproducibility and facilitate adoption by the AR testing community, we provide a comprehensive replication package1 including all experimental artifacts and analysis infrastructure. Our package contains TARIPlay’s source code and subject apps / playback videos. It also contains complete Python analysis scripts for statistical computation and visualization generation, raw JaCoCo coverage reports and gesture success logs from all experimental runs, and detailed reproduction instructions with parameter configuration files. The replication package includes complete video corpus with extracted metadata in structured JSON format, enabling validation of our mutual interval computation approach and exploration of alternative scheduling strategies.

they do not allow dynamic configuration of the testing scene (e.g., adjusting position of physical objects in the scene or changing lighting conditions) because the videos can hardly be revised after their generation. We believe this limitation can be largely alleviated when large sets of playback videos are accumulated and shared on the Internet, allowing AR testers to choose a subset of videos that cover the usage scenarios of their apps. Note that the required cost and expertise of creating playback videos are very low and even a non-technical person can create a lot of them with a recorder app. An alternate approach is to use a simulated physical environment (i.e., a VR scene) as the testing scene, which allows more flexible dynamic scene configuration. It will also allow conditional test cases (e.g., moving the camera based on intermediate test output). However, more studies are required to understand how the imprecision of the simulated environment may affect AR testing. For example, the mathematically generated surface may hide bugs related to imprecise plane detection, which is common for rough surface in the real world. Unnatural lighting conditions and reflections may also cause rendering difference of virtual objects, which may either miss rendering errors or cause false positives in testing. As for now, simulated environments are also supported in mainstream AR frameworks such as in Unity Mars [48] and Google ARCore Virtual Scene [11]. As they complement each other, we envision that both playback videos and VR test scenes will play important roles in automated AR testing.

5.2

After TARIPlay identifies test opportunities from playback videos, it can provide a UI layout at each time point similar to UIAutomator [15], so it becomes possible to apply existing Android GUI testing strategies to test AR apps. For example, we can use model-based testing [6] to trigger events within a visible box or across multiple visible boxes based on state transitions. We may also use patternbased testing [8] to trigger a UI event sequence pattern on a visible box, hybrid analysis [63] to take advantage of static information and combinatorial testing [33] to exhaust different interaction combinations. However, even with support of TARIPlay, the still-existing unique challenges of AR testing are time constraints (events must be finished within time limits) and the lack of back button (one can never go back to a previous state because the playback video always goes forward). So more advanced scheduling algorithm is required to better order UI events and distribute them into different time slots.

6

Playback videos enable AR user interaction (e.g., object placement, object manipulation) with a pre-recorded video of a physical environment. One limitation of testing with playback videos is that 1 https://anonymous.4open.science/r/TARIPlay-53F4

Related Works

Our research is mainly related to the testing of AR/VR applications, and the temporal tracking of planes.

6.1 5 Discussion 5.1 Playback Videos vs. Simulated Scenes

Adapting Existing GUI Testing Strategies

Testing and Analysis of AR and VR Apps

Two recent works PredART [39] and VOPA [55] facilitates AR app testing, but they focus on predicting test oracles to automatically detect AR bugs in screenshots or videos recorded during testing. In another related area of autonomous driving, Stocco et al. developed SelfOracle [46], a self-assessment framework that monitors deep neural network confidence at runtime to predict unsafe or unsupported driving scenarios. These techniques can

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

complement TARIPlay by checking its output videos after test interfaces are detected and interactions are performed. On VR testing, AutoWalker [1] is similar to a Monkey for VR and randomly guides the player camera in the VR scene. Later, Wang et al. developed VRTest [60] and VRGuide [53] to more efficiently explore VR spaces by detecting shorter paths in the space to interact with virtual objects. AutoWalker, VRTest, and VRGuide perform testing by controlling camera movement within interactive virtual scenes, which is not possible for fixed-camera playback videos. More recently, Li et al. developed a series of techniques [24, 25] to study and automatically assess cybersickness in virtual reality applications. On VR game testing, Gil et al. [44] and Souza el al. [7] proposed approaches to model VR applications and cover the model nodes and edges using automatic test cases. Zhao et al. [64] developed an testing approach to learn from human player behaviors. Rzig et al. [42] studied the characteristics of unit tests in VR applications and found they were of lower quality than their counterparts in other applications. It should be noted that testing of AR apps has very different challenges than the testing of VR apps. First, AR apps need to be tested in real-world scenes which are difficult to set up and reproduce, so playback videos and TARIPlay become necessary. In contrast, VR apps can be directly tested within the VR scene. Second, VR apps have all interactive objects predefined in the code, but AR apps read interactive objects from a changing environment so it always face issues of instable irregular interaction interface. Vision-based testing [54, 58] is another promising approach that may be applicable to AR apps, but existing techniques focus on static screen images instead of videos. Note that TARIPlay is already built upon the output of ARCore, leveraging the powerful computer vision techniques from Google. Several empirical studies have investigated VR and video game software. Murphy-Hill et al. [30] examined video game developers to identify the unique challenges they face compared to traditional software development. Washburn et al. [18] analyzed failed game projects to uncover common pitfalls, while Lin et al. explored update patterns on the Steam platform to determine the priorities of game updates. Rodriguez and Wang [41] studied the characters and trends of open source XR projects. Pascarella et al. [36] looked into open-source video game projects to highlight their characteristics and how they differ from non-game software projects. Harms [19] introduced a set of guidelines for evaluating the usability of AR applications and classified various usability issues. Nusrat et al. [59] examined performance repair logs of VR applications to pinpoint the main causes of performance degradation. Zhang et al. [61] studied the privacy risks of AP apps in mobile computing context [45, 52, 62]. Molina et al. [29] developed a code dependency analysis to detect dependencies across code and VR assets.

6.2

Temporal Tracking Methods based on Stability

Since our work tries to identify stable areas in a space over time, it is related to the series of work in AR to identify trackables based on their stability. Temporal tracking in AR involves keeping consistent identification of the same plane across multiple frames, and it is supported in ARCore. ARCore provides unique plane trackables (with persistent hash IDs within a session) that can be used to

follow a plane over time [14]. Our approach also leverages these trackable IDs to avoid expensive re-detection. There are also works on assessing plane stability over time. Some recent research [57] measure how consistent the plane’s position and shape remain across the selected frames. One method [26] is to compute the overlapping area of the plane’s footprint between successive frames – for example, using an Intersection over Union (IoU) metric as is done in . A high overlap ratio over time suggests the plane region is stable [40]. Frame skipping [9] is a technique to improve processing speed, with the trade-off of potentially missing rapid changes. Although these works also try to measure stability of planes in the 3D space, our approach works on a different setting (playback videos) and has a different purpose (focusing on AR app testing), so we developed visibility and life span analyses based on projection, and created different criteria (trackables are considered stable as long as their stable visible area is larger enough).

7

Future Works

In the future, we plan to work on the following research directions. First, we will extend our experiment to include more apps and playback videos, studying the effectiveness of our approach in more experiment settings. Second, we will extend our framework to support additional AR platforms such as ARKit and Unity AR Foundation and study whether our approach can be also applicable to a different AR framework. Third, we plan to study how much the test opportunities detected by TARIPlay can support manual test script development and evaluate the usefulness of the produced Gantt charts for AR testers. In particular, we can ask human subjects to write AR test code for playback videos with and without TARIPlay, and compare their working efficiency. Fourth, TARIPlay currently does not consider the occlusion among placed virtual objects and trackables, which may lead to unusable test opportunities. We plan to develop techniques to filter occluded interactive areas. Finally, we will work on the adaptation of existing GUI testing strategies to AR apps based on TARIPlay output, developing new techniques with consideration of time constraint and optimized event scheduling.

8

Conclusion

This work addressed the fundamental challenge of effective automated testing in Augmented Reality applications, where traditional random testing approaches fail to account for the spatial and temporal constraints inherent in AR interactions. We developed TARIPlay, a metadata-guided testing framework that leverages stable plane detection to schedule UI interactions within optimal spatio-temporal windows. Our empirical evaluation across four AR applications and nine representative testing scenarios demonstrates substantial improvements over existing approaches. TARIPlay achieves an average branch coverage improvement of 13.82 percentage points (from 41.98% to 55.80%) and gesture success rate improvements of 41.7 percentage points (from 51.8% to 93.5%). These gains stem from our method’s spatial awareness and temporal coordination capabilities, which enable systematic exploration of AR-specific code paths that random testing frequently misses. This work is supported in part by NSF grants CCF-1846467, CCF-2007718, CNS-2221843, and CCF-2418093. We also would like to thank Sahan Kalutarage for helping to label ground-truth test opportunities in playback videos.

Amir et al.

References [1] 2024. AutoWalk: Automated Accessibility Testing. https://github.com/. Accessed December 2024. [2] Amazon. 2024. Amazon Shopping. https://www.amazon.com/. Accessed December 2024. [3] Apple. 2024. ARKit: Recording and Replaying AR Session Data. https://developer. apple.com/documentation/arkit/recording-and-replaying-ar-session-data. Accessed December 2024. [4] Apple. 2025. ARKit Recording and Replaying. https://developer.apple.com/ documentation/arkit/arsession/recording_and_replaying_ar_session_data Accessed March 2025. [5] Andrea Arcuri and Lionel Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering (Waikiki, Honolulu, HI, USA) (ICSE ’11). Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.org/10.1145/1985793.1985795 [6] Young-Min Baek and Doo-Hwan Bae. 2016. Automated model-based Android GUI testing using multi-level GUI comparison criteria. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering. 238–249. [7] Diego Correa, María Cecilia Bastarrica, and Renzo Angles. 2018. Automated GUI Testing of Android Apps: A Systematic Mapping Study. In Proceedings of the 2018 IEEE International Conference on Software Quality, Reliability and Security (QRS 2018). IEEE, 358–365. [8] Pedro Costa, Ana CR Paiva, and Miguel Nabuco. 2014. Pattern based GUI testing for mobile applications. In 2014 9th International Conference on the Quality of Information and Communications Technology. IEEE, 66–74. [9] Zhicheng Ding, Zhixin Lai, Siyang Li, Panfeng Li, Qikai Yang, and Edward Wong. 2024. Confidence Trigger Detection: Accelerating Real-Time Trackingby-Detection Systems. In 2024 5th International Conference on Electronic Communication and Artificial Intelligence (ICECAI). 587–592. https://doi.org/10.1109/ ICECAI62591.2024.10674884 [10] Alastair G Gale. 1997. Human response to visual stimuli. In The perception of visual information. Springer, 127–147. [11] Google. 2024. ARCore: Google Play Services for AR. https://developers.google. com/ar. Accessed October 2024. [12] Google. 2024. Google ARCore Playback. https://developers.google.com/ar/ develop/recording-and-playback. Accessed December 2024. [13] Google. 2024. Google Lens. https://lens.google/. Accessed December 2024. [14] Google. 2025. Plane. https://developers.google.com/ar/reference/java/com/ google/ar/core/Plane Accessed March 2025. [15] Google. 2025. UI Automator. https://developer.android.com/training/testing/uiautomator.html. Accessed May 2025. [16] Google. 2025. UI/Application Exerciser Monkey. https://developer.android.com/ studio/test/other-testing-tools/monkey. Accessed June 2025. [17] Tianxiao Gu, Chengnian Sun, Xiaoxing Ma, Chun Cao, Chang Xu, Yuan Yao, Qirun Zhang, Jian Lu, and Zhendong Su. 2019. Practical GUI testing of Android applications via model abstraction and refinement. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 269–280. [18] Lin Gui, Chunrong Fang, Zhihong Zhao, and Qingkai Shi. 2022. An Empirical Study of Automated Testing for Mobile Games. In Proceedings of the 44th International Conference on Software Engineering (ICSE 2022). ACM, 1–12. [19] Patrick Harms. 2019. Automated Usability Evaluation of Virtual Reality Applications. In Proceedings of the 11th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS 2019). ACM, 1–12. [20] Donald Hearn. 1997. Computer graphics, C version. Pearson Education India. [21] IKEA. 2024. IKEA. https://www.ikea.com/. Accessed December 2024. [22] Youngjun Kim, Hannah Kim, and Yong Oock Kim. 2017. Virtual reality and augmented reality in plastic surgery: a review. Archives of plastic surgery 44, 03 (2017), 179–187. [23] Pavneet Singh Kochhar, Ferdian Thung, and David Lo. 2015. Code coverage and test suite effectiveness: Empirical study with real bugs in large systems. In 2015 IEEE 22nd international conference on software analysis, evolution, and reengineering (SANER). IEEE, 560–564. [24] Shuqing Li, Cuiyun Gao, Jianping Zhang, Yujia Zhang, Yepang Liu, Jiazhen Gu, Yun Peng, and Michael R Lyu. 2024. Less cybersickness, please: Demystifying and detecting stereoscopic visual inconsistencies in virtual reality apps. Proceedings of the ACM on Software Engineering 1, FSE (2024), 2167–2189. [25] Shuqing Li, Qisheng Zheng, Cuiyun Gao, Jia Feng, and Michael R Lyu. 2025. Extended Reality Cybersickness Assessment via User Review Analysis. Proceedings of the ACM on Software Engineering 2, ISSTA (2025), 1303–1325. [26] Chen Liu, Kihwan Kim, Jinwei Gu, Yasutaka Furukawa, and Jan Kautz. 2019. PlaneRCNN: 3d plane detection and reconstruction from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 4450–4459. [27] Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024. Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions.

In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13. [28] Sascha Minor, Vix Kemanji Ketoma, and Gerrit Meixner. 2023. Test automation for augmented reality applications: a development process model and case study. i-com (2023). https://doi.org/10.1515/icom-2023-0029 [29] Jacinto Molina, Xue Qin, and Xiaoyin Wang. 2021. Automatic extraction of code dependency in virtual reality software. In 2021 IEEE/ACM 29th International Conference on Program Comprehension (ICPC). IEEE, 381–385. [30] Gail C. Murphy, Eleni Stroulia, and Paul Sorenson. 2014. Cowboys, Ankle Sprains, and Keepers of Quality: How Is Video Game Development Different from Software Development?. In Proceedings of the 36th International Conference on Software Engineering (ICSE 2014). ACM, 1–11. [31] Niantic. 2024. Pokémon GO. https://pokemongolive.com/. Accessed December 2024. [32] Erik G Nilsson. 2009. Design patterns for user interface for mobile applications. Advances in engineering software 40, 12 (2009), 1318–1328. [33] Xintao Niu, Changhai Nie, Hareton Leung, Yu Lei, Xiaoyin Wang, Jiaxi Xu, and Yan Wang. 2018. An interleaving approach to combinatorial testing and failureinducing interaction identification. IEEE Transactions on Software Engineering 46, 6 (2018), 584–615. [34] Pranav Parekh, Shireen Patel, Nivedita Patel, and Manan Shah. 2020. Systematic review and meta-analysis of augmented reality in medicine, retail, and games. Visual Computing for Industry, Biomedicine, and Art 3, 1 (2020), 21. https://doi. org/10.1186/s42492-020-00057-7 [35] Nikolai Pärsch, Clemens Harnischmacher, Martin Baumann, Arnd Engeln, and Lutz Krauß. 2019. Designing Augmented Reality Navigation Visualizations for the Vehicle: A Question of Real World Object Coverage?. In HCI in Mobility, Transport, and Automotive Systems: First International Conference, MobiTAS 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26-31, 2019, Proceedings (Orlando, FL, USA). Springer-Verlag, Berlin, Heidelberg, 161–175. https://doi.org/10.1007/978-3-030-22666-4_12 [36] Luca Pascarella, Franz Schwerfeger, Fabio Palomba, and Alberto Bacchelli. 2018. Video-based Reproducing of User Interaction for Android Apps. In Proceedings of the 5th IEEE/ACM International Conference on Mobile Software Engineering and Systems (MOBILESoft 2018). ACM, 9–19. [37] Marcel Pfister, Jaw-Chyng L Lue, Francisco R Stefanini, Paulo Falabella, Laurie Dustin, Michael J Koss, and Mark S Humayun. 2014. Comparison of reaction response time between hand and foot controlled devices in simulated microsurgical testing. BioMed research international 2014, 1 (2014), 769296. [38] Xue Qin, Hao Zhong, and Xiaoyin Wang. 2019. TestMig: Migrating GUI Test Cases from iOS to Android. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2019). ACM, 270–281. [39] Tahmid Rafi, Xueling Zhang, and Xiaoyin Wang. 2022. PreDART: Towards automatic oracle prediction of object placements in augmented reality testing. In Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–13. [40] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 2019. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 658–666. [41] Irving Rodriguez and Xiaoyin Wang. 2017. An empirical study of open source virtual reality software projects. In 2017 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 474–475. [42] Dhia Eddine Rzig, Foyzul Hassan, and Chakkrit Tantithamthavorn. 2023. VRGuide: Efficient Testing of VR Scenes via Dynamic Cut Edges. In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE 2023). IEEE, 1–13. [43] Kabir S Said, Liming Nie, Adekunle A Ajibode, and Xueyi Zhou. 2020. GUI testing for mobile applications: objectives, approaches and challenges. In Proceedings of the 12th Asia-Pacific Symposium on Internetware. 51–60. [44] Juliana Saraiva, Eduardo Aranha, and Eduardo de Almeida. 2020. Automated Functional Testing for Mobile Applications: A Systematic Mapping Study. In Proceedings of the 28th International Conference on Program Comprehension (ICPC 2020). ACM, 283–294. [45] Rocky Slavin, Xiaoyin Wang, Mitra Bokaei Hosseini, James Hester, Ram Krishnan, Jaspreet Bhatia, Travis D Breaux, and Jianwei Niu. 2016. Toward a framework for detecting privacy policy violations in android application code. In Proceedings of the 38th International conference on software engineering. 25–36. [46] Andrea Stocco, Michael Weiss, Marco Calzana, and Paolo Tonella. 2020. Misbehaviour Prediction for Autonomous Driving Systems . In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE Computer Society, Los Alamitos, CA, USA, 359–371. https://doi.org/10.1145/3377811.3380353 [47] Ivan E Sutherland and Gary W Hodgman. 1974. Reentrant polygon clipping. Commun. ACM 17, 1 (1974), 32–42. [48] Unity Technologies. Accessed in June 2025. Unity MARS. https://unity.com/ products/unity-mars. [49] Simon Thorpe, Denis Fize, and Catherine Marlot. 1996. Speed of processing in the human visual system. Nature 381 (1996), 520–522.

TARIPlay: A Test Framework for AR Applications based on Interactive Area Tracking in Playback Videos

[50] Rufin VanRullen and Simon J. Thorpe. 2001. The time course of visual processing: from early perception to decision-making. Journal of Cognitive Neuroscience 13, 4 (2001), 454–461. [51] Arihant Singh Verma, Aditya Singh Verma, Sourabh Singh Verma, and Harish Sharma. 2023. 3 A comprehensive study for recent trends of AR/VR technology in real world scenarios. Handbook of Augmented and Virtual Reality 1 (2023), 31. [52] Xiaoyin Wang, Xue Qin, Mitra Bokaei Hosseini, Rocky Slavin, Travis D Breaux, and Jianwei Niu. 2018. Guileak: Tracing privacy policy claims on user input data for android applications. In Proceedings of the 40th International Conference on Software Engineering. 37–47. [53] Xiaoyin Wang, Tahmid Rafi, and Na Meng. 2023. Vrguide: Efficient testing of virtual reality scenes via dynamic cut coverage. In 2023 38th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 951–962. [54] Xusheng Xiao, Xiaoyin Wang, Zhihao Cao, Hanlin Wang, and Peng Gao. 2019. Iconintent: automatic identification of sensitive ui widgets based on icon classification for android apps. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 257–268. [55] Xiaoyi Yang, Yuxing Wang, Tahmid Rafi, Dongfang Liu, Xiaoyin Wang, and Xueling Zhang. 2024. Towards automatic oracle prediction for ar testing: Assessing virtual object placement quality under real-world scenes. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 717–729. [56] Xiaoyi Yang, Xueling Zhang, Tahmid Rafi, and Xiaoyin Wang. 2023. Augmented Reality Testing Bank. Microsoft Corp.. Available: https://github.com/ ARTBankManager/ARTBank, [Accessed: May 10, 2025]. [57] Weicai Ye, Hai Li, Tianxiang Zhang, Xiaowei Zhou, Hujun Bao, and Guofeng Zhang. 2021. SuperPlane: 3D Plane Detection and Description from a Single

Image. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 207–215. https: //doi.org/10.1109/VR50410.2021.00042 [58] Shengcheng Yu, Chunrong Fang, Ziyuan Tuo, Quanjun Zhang, Chunyang Chen, Zhenyu Chen, and Zhendong Su. 2023. Vision-based mobile app gui testing: A survey. arXiv preprint arXiv:2310.13518 (2023). [59] Nusrat Zahan, Thomas Zimmermann, Patrice Godefroid, Brendan Murphy, Chaiyong Ragkhitwetsagul, and Titus Barik. 2022. What are Weak Links in the npm Supply Chain?. In Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP 2022). ACM, 1–10. [60] Wenjie Zhang, Bei Li, Zhengwei Qi, and Hao Zhong. 2022. VRTest: An Extensible Framework for Automatic Testing of Virtual Reality Scenes. In Proceedings of the 44th International Conference on Software Engineering: Companion Proceedings (ICSE 2022). ACM, 158–162. [61] Xueling Zhang, Rocky Slavin, Xiaoyin Wang, and Jianwei Niu. 2019. Privacy assurance for android augmented reality apps. In 2019 IEEE 24th Pacific Rim International Symposium on Dependable Computing (PRDC). IEEE, 114–1141. [62] Xueling Zhang, Xiaoyin Wang, Rocky Slavin, Travis Breaux, and Jianwei Niu. 2020. How does misconfiguration of analytic services compromise mobile privacy?. In Proceedings of the ACM/IEEE 42nd international conference on software engineering. 1572–1583. [63] Xueling Zhang, Xiaoyin Wang, Rocky Slavin, and Jianwei Niu. 2021. Condysta: Context-aware dynamic supplement to static taint analysis. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE, 796–812. [64] Yan Zhao, Enyi Tang, Haipeng Cai, Xi Guo, Xiaoyin Wang, and Na Meng. 2022. A lightweight approach of human-like playtest for android apps. In 2022 IEEE international conference on software analysis, evolution and reengineering (SANER). IEEE, 309–320.