GUITestScape: Towards Open-set Evaluation on Exploratory GUI Testing Xiaoyi Chen1 , Yifei Gao1 , Yang Xu1 , Xingxing Song1 , Yi Zhang2 , Jitao Sang1* , 1 Beijing Jiaotong University, 2 Independent Researcher Corresponding author: [email protected]
arXiv:2605.29532v1 [cs.SE] 28 May 2026
Abstract
tion (Gao et al., 2026a). Yet current evaluation falls short of measuring these capabilities faithfully. We identify two fundamental challenges: (1) Display-defect blindness. Current benchmarks (Gao et al., 2026a; Ahmed et al., 2026) focus almost exclusively on interaction defects, overlooking display defects such as rendering errors, missing elements, and layout misalignment (Liu et al., 2020, 2022). These defects often have the most immediate impact on user experience, yet detecting them demands open-ended visual anomaly detection, a capability that interaction-oriented evaluation never probes. (2) Closed-set evaluation. Existing protocols verify agent reports against predefined defect annotations, which introduces two limitations. First, the evaluation cannot scale to unlabeled defects, since any anomaly outside the annotated set has no reference for verification. Second, scoring collapses the entire testing process into a single end-state judgment (He et al., 2024), conflating qualitatively distinct failure modes. An agent may never reach the defect scenario, may reach it but fail to trigger the defect, or may trigger the anomaly yet fail to recognize it. Consequently, agents with very different capability profiles are flattened to indistinguishable scores. Figure 1 illustrates these challenges with two examples: a realistic interaction case where display defects appear repeatedly during testing, and an exploratory testing process where agents exhibit different failure modes before successful defect detection. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types. Each defect is instantiated into a structured evaluation case along three dimensions: a resettable scenario that ensures run-to-run consistency, a state-level test basis that captures app-specific trigger criteria, and multiple navigation tasks that simulate how testers might approach the same defect through dif-
Exploratory GUI testing is a particularly demanding setting for MLLM agents: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interaction. However, current evaluation falls short on two fronts. First, existing benchmarks focus almost exclusively on interaction defects, leaving display defects outside the evaluation frame. Second, evaluation protocols are bound to predefined defect annotations, collapsing the testing process into a single end-state judgment that conflates qualitatively distinct failure modes. To address these challenges, we present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects spanning interaction and display types, and introduce GUIJudge, an open-set evaluator that decomposes an agent’s testing trajectory into independently diagnosable capabilities. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation beyond predefined annotations, substantially outperforming all baselines. Benchmarking on GUITestScape further reveals that detection remains the critical bottleneck for existing models across both defect types, and that integrating GUIJudge’s verifiers into existing agents significantly boosts their detection performance without retraining.
1
Introduction
Multimodal Large Language Models have substantially strengthened GUI agents, extending their reach from single-step grounding to long-horizon tasks (Qin et al., 2025; Rawles et al.; Xie et al., 2024a) and practical applications such as software quality assurance (Zhao et al., 2024). Among these, exploratory GUI testing stands out as particularly demanding: without predefined test scripts, an agent must autonomously navigate an application and discover defects through its own interac1
Navigation Task: Verify if the latest software release information is accessible and fully up to date.
d. Successfully Detect
a. Fail to Reach
GUI Defect [Display Defect] Current page has overlapping headers. [Interaction Defect] Clicking on "UI Inference" jump to "aggregated stats".
Garbled text displayed Outcome:
Outcome:
Inconsistent icon sizes
(a) Display defects in one test case.
No Defect
The agent navigated to the defect context but bypassed the defect, failing to trigger it.
The agent deviates from the task intent early on, leaving no opportunity to trigger the defect.
b. Fail to Trigger
Outcome: Following defect exposure, the agent failed to identify and report the anomaly.
c. Fail to Detect
(b) Different outcomes in exploratory GUI testing.
Figure 1: Motivating examples for GUITestScape and GUIJudge. (a) A realistic testing process exposes multiple display defects, such as garbled text and inconsistent icon sizes. (b) The same exploratory testing process may lead to different outcomes, including failures at Reach, Trigger, and Detect, as well as successful defect detection.
ferent interaction patterns. We further introduce GUIJudge, an open-set evaluator that decomposes an agent’s testing trajectory into three independently diagnosable capabilities: Reaching, Triggering, and Detection, disentangling failure modes that prior end-state scoring conflates. Experimental results demonstrate that GUIJudge achieves reliable process-aware evaluation, outperforming all baseline evaluators with 97.12% Recall at the Detect stage. GUITestScape further reveals that detection remains the critical bottleneck for existing models, with the best-performing model reaching only 40.55% Recall. This bottleneck can be effectively addressed by integrating GUIJudge’s verifiers into existing agents without retraining. Our contributions include:
cation interfaces, but their testing logic is mainly driven by event generation and UI structure rather than app intent or defect semantics. Recent work has introduced MLLM-based agents into GUI testing. ScenGen (Yu et al., 2026), VisionDroid (Liu et al., 2024), GUITester (Gao et al., 2026b), and SpecOps (Ahmed et al., 2026) move testing toward scenario-guided, vision-driven, or defect-oriented exploration. GUITestScape follows this direction by providing a unified benchmark for evaluating GUI agents in exploratory GUI testing scenarios. 2.2
Existing GUI benchmarks such as AndroidWorld (Rawles et al., 2025) and OSWorld (Xie et al., 2024b) mainly evaluate task completion. GUI testing benchmarks instead target GUI testing capabilities: GUI Testing Arena (Zhao et al., 2024) structures evaluation around test intent generation, task execution, and defect detection; GUITester (Gao et al., 2026a) introduces GUITestBench for exploratory defect discovery; and ScenGen (Yu et al., 2026) emphasizes scenario-based test construction. Compared with these benchmarks, GUITestScape provides a more comprehensive benchmark that covers both interaction and display defects.
• We present GUITestScape, an interactive benchmark covering 61 real-world Android applications and 508 preset defects across both interaction and display types. • We propose a three-stage capability decomposition that disentangles exploratory GUI testing into Reaching, Triggering, and Detection as independently diagnosable dimensions. • We introduce GUIJudge, an open-set evaluator that enables process-aware evaluation beyond predefined defect annotations.
2
Related Works
2.1
Automated GUI Testing
Benchmarks for GUI Testing
2.3
GUI Trajectory Evaluation
Evaluating exploratory GUI trajectories is difficult because valid trigger paths are not unique and defect evidence may appear only in short trajectory intervals. Rule-based methods, including those used in GUITester (Gao et al., 2026b) and WebArena (Zhou et al., 2024), rely on pre-
Automated GUI testing has long relied on random, heuristic, or UI-guided exploration. Tools such as Monkey (Choudhary et al., 2015) and DroidBot (Li et al., 2017) can efficiently traverse appli2
defined checks or golden-path matching, making them reliable in closed-set settings but difficult to generalize. LLM-as-Judge methods have been studied in related agent evaluation settings such as Mind2Web (Deng et al., 2023), AgentRewardBench (Lù et al., 2025), and Beyond Binary Rewards (Damani et al., 2025). However, holistic judging over long trajectories provides limited process diagnosis. GUIJudge instead localizes defectrelevant segments and verifies defect evidence with verifiers designed for different evidence forms, enabling process-aware evaluation of the testing process.
3
the resulting interaction trajectory rather than visual evidence alone. We adopt three fault modes from GUITester (Gao et al., 2026a). (1) Operation No Response (ONR), where an interaction yields no observable feedback. (2) Unexpected Task Result (UTR), where the outcome of a multi-step operation deviates from its intended effect. (3) Navigation Logic Error (NLE), where flawed transitions lead to incorrect navigation flow. 3.2
A benchmark requires a dynamic case in which an agent can actually encounter, trigger, and judge a defect. We instantiate each defect into such a case along three dimensions, each addressing a distinct requirement of evaluation: (1) a resettable scenario (§3.2.1) that restores the same starting environment before every run; (2) test basis (§3.2.2) that provides the criteria for judging whether a defect is genuinely triggered; and (3) multiple navigation tasks (§3.2.3) that simulate the diverse interaction patterns of real testers
GUITestScape
This section presents GUITestScape, an interactive benchmark for evaluating agents in exploratory GUI testing. Figure 2 provides an overview of its construction pipeline. We first describe the collection and categorization of real-world defects, covering both display and interaction types (§3.1). We then detail the per-defect composition that constitutes each evaluation case, comprising a resettable scenario, a state-level test basis and multiple navigation tasks (§3.2). Finally, we report the scale and distribution of GUITestScape (§3.3). 3.1
Benchmark Composition
3.2.1
Resettable Scenario
Whether a defect remains reproducible depends on the application’s runtime context (Romano et al., 2021), and uncontrolled drift in this context makes evaluation unstable across runs. We therefore wrap each defect in a resettable evaluation scenario that fixes its prerequisites for triggering. Beyond the target application, we specify the defect-relevant initial conditions under which testing begins, including pre-populated application data and required system-side settings. These conditions are restored before every evaluation, so that different agents are tested from an identical starting point when probing the same defect. Consequently, evaluation results reflect the agent’s testing competence rather than environmental contingency.
Defect Collection and Categorization
We collect GUI defects from public issue trackers and bug reports of real-world Android applications. Building on the defect taxonomy of GTArena (Zhao et al., 2024), we organize the collected defects into two categories, display defects and interaction defects. Display Defects are visual abnormalities embedded in the rendered interface itself, present regardless of whether the user interacts with the screen. Detecting them requires open-ended visual anomaly perception, since no behavioral cue signals their presence and the agent must judge the screen on visual evidence alone. We further divide them into two fault modes. (1) Content Rendering defects (CR) concern the abnormal presentation of textual or visual content, such as broken icons or garbled text. (2) Element Layout defects (EL) concern abnormalities in spatial organization, such as overlapping elements or alignment errors. Interaction Defects are anomalies that surface from the interplay between an agent’s actions and the application’s responses, and detecting them requires reasoning about the logical consistency of
3.2.2
State-level Test Basis
Exploratory GUI testing conducted by human testers does not rely solely on generic interaction experience. Instead, testers interpret interaction results according to a test basis (Schieferdecker, 2025), which captures the design specifications and business logic of the target application. For example, when a feedback form returns the user to the home screen after submission, a tester familiar with the app’s design rationale recognizes this as the intended streamlined flow, whereas the same transition in an app that should keep the user on 3
Evaluation Case
Raw Defect
Repository: OpenTracks v4.13.0 Issue #213: Aggregated stats page ······ Expected behavior:The “UI Inference” ······
Construction pipeline of GUITestScape
Resettable Scenario APK
Preset data
Permission & System state
Resettable Scenario: ensure fair agent comparison State-level Test Basis: provide app-specific criteria Multi-Entry Task Design: diverse pattern evaluation
State-level Test Basis
Multi-Entry Task Design
Testing intent: Check settings hierarchy navigation. App business logic : UI Inference links to a configuration subpage.
Entry A from track detail Entry B from Home
···
Entry N from Search
Precondition State shows main settings page. Trigger
States show irrelevant config.
Evidence
Retry still reaches unrelated state.
Task set: (1) Enter the UI Interface from the main menu. (2) Check the UI settings from a track detail page. (3) Search for UI Inference ····
Figure 2: Construction pipeline of GUITestScape. Each real-world defect is instantiated into an evaluation case with a resettable scenario, a state-level test basis, and a multi-entry task set.
the editable page would be flagged as a navigation logic error. Without access to such app-specific semantics, an agent has no consistent ground for judging across applications. For each defect in GUITestScape, we provide state-level test basis based on the target application’s business logic and testing intent. It specifies which states demonstrate that the necessary preconditions have been established, which states indicate that the defect has been triggered, and which states provide verifiable evidence of the resulting anomaly. Rather than prescribing a unique action trajectory, the state-level test basis keeps navigation paths open and only fixes the state-level checkpoints that any valid trigger process must traverse. This design provides the app-specific test basis that generic interaction priors cannot cover, while remaining permissive enough to accommodate diverse interaction patterns. 3.2.3
Figure 3: Hierarchical distribution of preset defects across defect types, fault modes, and application categories in GUITestScape.
ing to it becomes a controlled source of variation, exposing whether an agent’s testing competence generalizes across interaction patterns rather than being tied to any single predefined route.
Multi-Entry Task Design
3.3
Human testers exploring an application rarely approach a defect through a single fixed path; they typically vary entry points and interaction sequences to rule out coincidental triggers and to expose the defect under different operational conditions. To preserve the exploratory nature of testing, GUITestScape pairs each defect with a set of navigation tasks derived from its state-level test basis. Specifically, we leverage the entry conditions, result-relevant states, and inspection points encoded in the test basis to instantiate tasks that approach the same defect-relevant context from different functional entry points and under varying levels of task guidance. The defect therefore remains the evaluation target, while the path lead-
Benchmark Statistics
GUITestScape collects 508 preset defects from 61 real-world Android applications across 5 major categories. We consolidate these defects into 82 resettable scenarios, annotate each with state-level test basis, and expand them into 384 navigation tasks. Figure 3 shows the distribution across defect types and application categories, with defects nearly evenly split between interaction (53%) and display (47%) types.
4
GUIJudge
Building on the evaluation cases provided by GUITestScape, we introduce GUIJudge, our evaluator 4
Resettable Scenario
Process-aware Evaluation Outcome
Response
Action
Trigger: Pass/Fail
Reach: Pass/Fail
GUI Agent Execute
4.2
2 Defect-specific Verifiers
Thought :.. Action: Click GUI_Defect: Interaction
Thought :.. Action: Click GUI_Defect: Dislplay
Detect
Detect
Trajectory : Screenshots
···
Interaction
···
History Step 1:
Step 2:
Thought : I need to open the setting pase by ….. Action: Click () GUI_Defect: None Thought : I have reched setting then find ….. Action: Scroll () GUI_Defect: None
····· Step N:
Evaluating an agent’s testing trajectory faces two practical obstacles. First, exploratory trajectories interleave defect-relevant moments with extensive irrelevant navigation, making end-to-end judgment prone to evidence dilution. Second, display and interaction defects manifest through fundamentally different forms of evidence, precluding a single unified verification procedure. These two challenges motivate the design of the trajectory retriever (§4.2.1) and the defect-specific verifiers (§4.2.2).
GUIJudge
Task : Enter the UI Interface setting from the main menu.
Thought : Navigate up adn retry click UI ….. Action: Click() GUI_Defect: None
Overlapping headers.
Navigates incorrectly to Aggregated stats
Trigger
Display
1 Trajectory Retriever Defect-relevant segment
·· Retry ··
···
Retriever Precondition
Trigger
Evidence
Test Basis : Precondition State shows main settings page. Trigger
States show irrelevant config.
Evidence
Retry still reaches unrelated state.
4.2.1 Reach
Trajectory Retriever
To reduce trajectory noise caused by defectirrelevant steps, GUIJudge first retrieves defectrelevant segments from the full trajectory. The retriever performs the Reach judgment based on the state-level test basis of the corresponding evaluation case. Specifically, it localizes segments corresponding to the precondition states required for defect exposure. If such segments are identified, GUIJudge considers the agent to have reached the target context; otherwise, the defect discovery process fails before exposure. The retrieved segments are not treated as final defect judgments. Instead, they are passed to defect-specific verifiers, which further judge whether the defect is triggered and whether the agent reports the anomaly correctly.
Figure 4: GUIJudge Workflow. Given an agent trajectory and the state-level test basis of the corresponding evaluation case, GUIJudge first retrieves defect-relevant segments and then applies two defect-specific verifiers to produce process-aware judgments across three dimensions: Reach, Trigger, and Detect.
for exploratory GUI testing. We first define three evaluation dimensions that decompose defect discovery into diagnosable capabilities (§4.1), then detail the evaluation pipeline of GUIJudge, including a trajectory retriever and two defect-specific verifiers (§4.2). Figure 4 provides an overview of the GUIJudge workflow. 4.1
Two-Stage Evaluation Pipeline
Reach: Pass/Fail
4.2.2
Evaluation Targets
Defect-specific Verifiers
Since display and interaction defects rely on different evidence forms, GUIJudge applies defectspecific verifiers to the retrieved defect-relevant segments. For display defects, the verifier examines GUI states for visual anomalies, such as garbled text, broken images, overlapping elements, or alignment errors. For interaction defects, the verifier examines action-state transitions and checks whether the observed response is consistent with the task expectation and the app-specific business logic. The verifiers judge Trigger by determining whether the target defect is actually exposed, and judge Detect by checking whether the agent’s report is consistent with the verified defect evidence.
Defect discovery unfolds in three stages: an agent must first reach the defect’s location, then trigger the defect through interaction, and finally recognize the resulting anomaly. A breakdown at any stage halts this progression and leaves the defect unreported. Yet existing end-state evaluation collapses the entire progression into a single verdict, masking the capability gap that actually caused the failure. GUIJudge therefore evaluates each stage as an independent dimension. • Reaching evaluates whether an agent correctly interprets the testing task and navigates to the precondition state required for defect exposure. • Triggering evaluates whether an agent, having reached the defect-relevant context, performs the interaction that surfaces the defect. This dimension applies only to interaction defects. • Detection evaluates whether an agent, once the defect is exposed, recognizes the anomaly and reports it as a GUI defect.
5
Experiments
We conduct comprehensive experiments to answer the following research questions: • RQ1: Does GUIJudge yield valid and reliable judgments of GUI testing outcomes? 5
Display Defects Method
Stage
CR
Interaction Defects
EL
NLE
ONR
Overall UTR
Overall
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
GUIJudge (Process-Aware Performance) GUIJudge Reach 100.00 GUIJudge Trigger 100.00 GUIJudge Detect 100.00
94.34 94.34 89.36
100.00 100.00 96.00
100.00 100.00 92.31
100.00 100.00 100.00
98.25 96.77 96.77
100.00 100.00 100.00
100.00 100.00 100.00
100.00 100.00 92.31
97.03 94.92 94.12
100.00 100.00 97.12
97.87 97.12 93.95
Ablation of GUIJudge Design w/o Retrieval Detect w/o Typed Verifiers Detect
85.71 95.24
72.39 90.91
70.67 84.00
66.69 89.37
75.56 80.00
78.02 85.71
94.12 82.35
90.39 90.32
65.38 65.38
69.58 79.07
76.92 80.77
74.23 87.05
Baseline Evaluators (Detect Only) GUI-Tester Detect – Rule-based Detect – Claude-4.6 Detect 66.67 GPT-5.4 Detect 53.97 Gemini-3.1 Flash Detect 84.13
– – 71.67 61.24 80.33
– – 14.67 24.12 45.33
– – 21.94 35.91 55.73
86.67 86.67 86.67 77.78 82.22
89.66 92.68 78.79 76.91 70.51
62.51 68.75 92.16 60.78 82.35
76.92 81.48 83.15 67.91 74.42
72.58 60.13 62.82 64.13 56.41
78.26 71.43 62.41 66.51 58.62
73.21 69.64 60.26 53.85 67.31
81.19 80.41 64.14 61.88 67.53
Table 1: Comparison of evaluation performance on the manually annotated evaluation set. The table reports GUIJudge’s process-aware performance, two ablations without trajectory retriever and without type-specific verifiers, and comparisons with baseline evaluators that only support final-outcome detect evaluation. In the Detect block, bold marks the best result in each column, and underline marks the second-best. "–" indicates that the corresponding evaluator does not support that defect type.
• RQ2: What is the performance landscape of existing models on the exploratory GUI testing? • RQ3: Can GUIJudge’s defect verifiers improve the agent testing capability?
evaluator. The evaluated models span two categories: general-purpose vision-language models (General Models), including GPT-5.4, Gemini-3.1Pro, Qwen-3-VL-Plus, and Seed-1.8; and GUIspecialized agent models (Agent Models), including UI-TARS-1.5-7B, GUI-Owl-1.5 (8B/32B), and MAI-UI-8B. This setup enables a systematic comparison of exploratory GUI testing capability across different fault modes and model categories. RQ3: Verifier Integration Analysis. We integrate GUIJudge’s defect-specific verifiers into the workflows of UI-TARS-1.5-7B, Seed-1.8, and Gemini3.1-Pro. This allows the agents to retain their existing navigation capabilities while receiving additional support for defect recognition during execution. The resulting performance is then evaluated on GUITestScape to assess the practical benefit of verifier integration. Evaluation Metrics. Across all experiments, we adopt Recall and F1 as the primary metrics. Recall directly reflects a model’s defect discovery capability in exploratory GUI testing, while F1 guards against overestimation arising from overly aggressive anomaly reporting.
Accordingly, we first describe the experimental setup (§5.1), and then organize the experiments into three parts: GUIJudge validation (§5.2), model benchmarking on GUITestScape (§5.3), and verifier integration analysis (§5.4). 5.1
Experiment Setup
In this section, we describe the experimental settings and evaluation metrics used throughout our experiments. RQ1: GUIJudge Validation. To quantitatively assess the effectiveness of GUIJudge’s design and the reliability of its judgments, we construct a manually annotated evaluation set covering 91.57% of the defect scenarios in the benchmark. We compare GUIJudge against several representative evaluator baselines, including the hybrid evaluator from GUITester (Gao et al., 2026a), a rule-based evaluator, and three LLM-as-Judge models: GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Flash. We further conduct two ablation studies to validate the design choices in GUIJudge’s pipeline, examining the respective contributions of trajectory retriever and type-specific verifiers. RQ2: Model Benchmarking on GUITestScape. We benchmark a diverse set of existing models on GUITestScape using GUIJudge as a unified
5.2
RQ1
Overall Performance. As shown in Table 1, GUIJudge achieves 97.87% and 97.12% F1 scores at both the Reach and Trigger dimensions. At the Detect dimension, GUIJudge improves overall Recall from 73.21% to 97.12% (+23.91%) and overall 6
Display Defects Model
Stage
CR
Interaction Defects
EL
NLE
ONR
Overall UTR
Overall
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
General Models GPT-5.4 Gemini-3.1-Pro Seed-1.8 Qwen-3-VL-Plus
Detect Detect Detect Detect
23.91 46.74 53.26 28.26
33.85 59.72 65.33 42.62
5.23 30.23 44.19 22.09
8.29 42.98 56.30 33.63
19.67 47.54 40.98 27.87
24.74 55.24 48.08 37.36
17.33 41.33 34.67 28.00
21.49 46.62 45.22 33.87
16.67 27.78 27.78 14.81
20.34 33.89 35.71 20.51
15.75 36.42 40.55 23.23
21.48 46.19 51.05 32.82
Agent Models UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
Detect Detect Detect Detect
5.43 1.09 9.78 1.09
9.71 2.13 17.65 2.15
1.74 1.74 3.49 1.16
3.28 3.39 6.63 2.29
29.51 1.64 4.92 8.20
37.89 2.99 8.70 14.29
8.00 0.00 2.67 9.33
10.34 0.00 4.82 15.38
5.56 0.93 0.00 3.39
7.74 1.75 0.00 6.06
7.48 1.18 3.94 4.13
11.66 2.26 7.30 7.62
Table 2: Pass@1 Detect-stage performance of general models and specialized agent models on GUITestScape. Bold marks the best result in each column, and underline marks the second best.
F1 from 81.19% to 93.95% (+12.76%) over the strongest baseline evaluator. These results demonstrate that GUIJudge not only reliably retrieves defect-relevant trajectory segments, but also produces accurate judgments on top of them, effectively reformulating final-only evaluation into a staged, process-aware capability assessment. Effect of Retrieval. Removing trajectory retrieval reduces overall Detect Recall from 97.12% to 76.92%. The performance drop is smallest on ONR but largest on UTR. This is because ONR defects are typically triggered by a single action under relatively fixed verification conditions, making their evidence straightforward to localize; UTR defects, by contrast, often require cross-state comparison and delayed outcome verification, resulting in more dispersed and harder-to-localize evidence. These results indicate that evaluation reliability is fundamentally constrained by evidence localization quality, confirming that trajectory retrieval is essential for stable evaluation and constitutes a key source of GUIJudge’s performance gains. Analysis Across Defect Types. At the Detect stage, GUIJudge achieves the best performance across all defect types. By contrast, GUITester and the rule-based evaluator cannot handle display defects, while LLM-as-Judge methods exhibit unstable performance across defect categories. The ablation further reveals that replacing type-specific verifiers with a unified verifier reduces overall F1 from 93.95% to 87.05%, while also increasing performance variance across defect types. This degradation stems from the fundamentally different evidence requirements of the two defect categories: display defects are assessed through visual anomalies in interface states, whereas interaction defects
are assessed through the consistency between action outcomes and task expectations. By tailoring each verifier to its corresponding evidence form, GUIJudge achieves more stable and reliable evaluation across diverse defect types. ANSWER TO RQ1
GUIJudge enables reliable process-aware evaluation, and stable evaluation requires both trajectory retrieval and type-specific verification.
5.3
RQ2
Overall Performance. Figure 5 shows that the evaluated models are generally able to trigger defects under task guidance. In particular, Seed achieves overall F1 scores of 93.93% and 89.18% at the Reach and Trigger stages, respectively. However, performance drops sharply for most models from Trigger to Detect. For example, the F1 score of UI-TARS decreases from 72.45% to 28.08%. This suggests that, in exploratory testing scenarios, the core bottleneck arises after the defect is exposed, when models fail to accurately recognize anomaly evidence and make judgments accordingly. Therefore, future research on GUI Test Agents should focus on further improving anomaly awareness, outcome verification, and defect attribution. General Models vs. Agent Models. As shown in Figure 5 and Table 2, general models exhibit considerably smaller performance degradation from Trigger to Detect than agent models. For instance, Seed-1.8’s F1 score drops from 89.18% to 64.33%, whereas UI-TARS declines more sharply from 72.45% to 28.08%. This contrast suggests that the primary gap between the two groups lies in how 7
(a) Overall Recall, Pass@1
(b) Overall F1, Pass@1
(c) Overall Recall, Pass@3
(d) Overall F1, Pass@3
Figure 5: Overall Recall and F1 trends across Reach, Trigger, and Detect for general models and agent models on GUITestScape under Pass@1 and Pass@3. Detailed results are provided in Tables 4 and 5 in the appendix.
they interpret abnormal states exposed during task execution. Agent models are largely optimized for task-intent following, which enables strong Reach and Trigger performance even at smaller model scales; however, once an anomaly is exposed, they tend to overlook it rather than treating it as evidence for further judgment. General models, by contrast, are more oriented toward holistic interface understanding, which makes them better equipped for state interpretation and anomaly judgment after defect exposure.
Overall
Agent Setting w/ GUIJudge Verifier UI-TARS-1.5-7B Gemini-3.1-Pro Seed-1.8
Recall↑
F1↑
57.02 (+49.54) 61.27 (+24.85) 70.97 (+30.42)
69.07 (+57.41) 71.07 (+24.88) 79.48 (+28.43)
Table 3: Detect stage overall performance after integrating GUIJudge Verifier into each agent’s workflow. Numbers in parentheses denote absolute gains over the corresponding unassisted baseline. Bold marks the best result, and underline marks the second-best.
ANSWER TO RQ2
(1) General models outperform specialized GUI agent models in exploratory GUI testing, with Seed-1.8 achieving the best overall Detect performance among all evaluated models. (2) Current models struggle with anomaly recognition after defect exposure, especially when the judgment requires structural layout expectations or app specific outcome verification.
its Recall from 7.48% to 57.02% and its F1 from 11.66% to 69.07%. Since the verifiers are incorporated without retraining the underlying interaction policy, these gains indicate that a dedicated defect verification module can effectively compensate for agent models’ inherent weakness in anomaly recognition and defect judgment after defect exposure.
Analysis Across Fault Modes. Nearly all models perform better on CR than on EL, and also better on NLE and ONR than on UTR. CR is usually supported by directly visible evidence, such as garbled text, whereas EL further depends on whether the model can infer a reasonable page layout, which may vary substantially across apps. Similarly, NLE and ONR can often be judged from a single state transition, while UTR requires checking across states whether the expected outcome occurs under the app specific business logic. These results suggest that current models mainly lack two capabilities: (1) judging interface structural plausibility and (2) verifying delayed task outcomes under app specific business logic. 5.4
ANSWER TO RQ3
Integrating GUIJudge’s verifiers consistently improves Detect performance across all agents, confirming that defect recognition can be effectively decoupled from navigation capability and addressed through modular integration.
6
Conclusion
This paper identifies two core challenges in exploratory GUI testing evaluation: insufficient coverage of display defects and closed-set final-outcome evaluation. We introduce GUITestScape, the first interactive benchmark that jointly covers display and interaction defects, and propose GUIJudge, which enables open-set, process-aware evaluation. Experiments demonstrate the reliability of GUIJudge and further reveal that the main bottlenecks of current models in exploratory GUI testing lie in judging interface structural plausibility and verifying delayed outcomes.
RQ3
Table 3 shows that integrating GUIJudge’s defectspecific verifiers substantially improves exploratory GUI testing performance across all evaluated agents. Most notably, UI-TARS-1.5-7B improves 8
Limitations
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Preprint, arXiv:2306.06070.
(1) Extraction of App-specific Test Basis. Compared with closed-set evaluation, GUITestScape substantially reduces construction costs. However, reliable defect judgment requires app-specific standards about expected behavior and interface structure. In our benchmark, these standards are provided through resettable scenarios and state-level test bases, which support reproducible and judgeable evaluation, but their construction still partly relies on human effort. (2) Boundary in Evaluating Test Design. GUITestScape evaluates the execution stage of exploratory GUI testing: given a navigation task, whether an agent can reach, trigger, and detect defects during interaction. It does not yet evaluate upstream test design capabilities, such as generating testing tasks, designing boundary conditions, or planning exploration strategies for broader coverage. (3) Broader Platform and Workflow Coverage. GUITestScape currently focuses on mobile Android applications. Although it covers both display and interaction defects, it does not yet cover web, desktop, or more complex cross-application workflows. These settings may involve different interaction patterns, environmental dependencies, and defect manifestations. Future work will focus on these directions: (1) automatically extracting app-specific test bases from application documents, historical interaction traces, and normal page styles, (2) extending evaluation from test execution to test design, including testing task generation and boundary condition design, and (3) expanding GUITestScape to broader GUI platforms and more complex workflows.
Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, and Jitao Sang. 2026a. Guitester: Enabling gui agents for exploratory defect discovery. Preprint, arXiv:2601.04500. Yifei Gao, Jiang Wu, Xiaoyi Chen, Yifan Yang, Zhe Cui, Tianyi Ma, Jiaming Zhang, and Jitao Sang. 2026b. Guitester: Enabling gui agents for exploratory defect discovery. arXiv preprint arXiv:2601.04500. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-toend web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6864–6890. Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In Proceedings of the 39th International Conference on Software Engineering Companion, ICSE-C ’17, page 23–26. IEEE Press. Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2020. Owl eyes: Spotting ui display issues via visual understanding. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering, pages 398–409. Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Nighthawk: Fully automated localizing ui display issues via visual understanding. IEEE Transactions on Software Engineering, 49(1):403–418. Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, and Qing Wang. 2024. Seeing is believing: Visiondriven non-crash functional bug detection for mobile apps. Preprint, arXiv:2407.03037.
References Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy. 2025. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. Preprint, arXiv:2504.08942.
Syed Yusuf Ahmed, Shiwei Feng, Chanwoo Bae, and Calix Barrus Xiangyu Zhang. 2026. Specops: A fully automated ai agent testing framework in real-world gui environments. arXiv preprint arXiv:2603.10268. Shauvik Roy Choudhary, Alessandra Gorla, and Alessandro Orso. 2015. Automated test input generation for android: Are we there yet? (e). In 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 429–440.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Ui-tars: Pioneering automated gui interaction with native agents, 2025. URL https://arxiv. org/abs/2501.12326.
Mehul Damani, Isha Puri, Stewart Slocum, Idan Shenfeld, Leshem Choshen, Yoon Kim, and Jacob Andreas. 2025. Beyond binary rewards: Training lms to reason about their uncertainty. Preprint, arXiv:2507.16806.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice
9
Li, William Bishop, Wei Li, Folawiyo CampbellAjala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. 2025. Androidworld: A dynamic benchmarking environment for autonomous agents. Preprint, arXiv:2405.14573.
A
GUIJudge
A.1
Workflow of GUIJudge
Algorithm 1 presents the workflow of GUIJudge for evaluating a single agent trajectory. GUIJudge takes as input the agent trajectory τ , the agent defect report ρ, the navigation task g, and the corresponding state-level test basis B. It proceeds in two stages: a Trajectory Retriever that determines Reach and extracts defect-relevant segments, followed by parallel defect verification, where the Display Defect Verifier scans the retrieved segments for display defects and the Interaction Defect Verifier scans the same segments for interaction defects. The workflow finally returns the verdicts Reach, Trigger, and Detect for the trajectory. Trajectory Retriever scans the full trajectory τ to identify defect-relevant segments by matching trajectory steps against the precondition description Bprecondition and the evidence description Bevidence in the state-level test basis B. Once a trajectory step matches Bprecondition , GUIJudge marks Reach as true. It then continues searching forward for a step matching Bevidence . When both are matched in order, the sub-trajectory from the precondition-matched step to the evidence-matched step is recorded as a defect-relevant segment. This process is repeated until the trajectory ends, thereby collecting all such segments for subsequent defect verification. Display Defect Verifier treats Trigger as not applicable and sets it to ⊥. If the retrieved defectrelevant segments are non-empty, the Display Defect Verifier scans each state within each segment as a standalone GUI observation. Once a state is verified to contain a display defect, GUIJudge compares the verified defect with the agent defect report ρ. If the reported step and defect description are consistent with the verified result, Detect is marked as true. Interaction Defect Verifier verifies the retrieved defect-relevant segments at the level of state transitions. For each segment, it first resets the interaction history H and then traverses all transitions in that segment in temporal order. After each transition is added to the history, the verifier judges whether the current transition exposes an interaction defect under the navigation task g, the statelevel test basis B, and the accumulated history H. Once an interaction defect is verified, Trigger is marked as true. GUIJudge then compares the verified defect with the agent defect report ρ. If they
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo CampbellAjala, and 1 others. Androidworld: A dynamic benchmarking environment for autonomous agents, 2024. URL https://arxiv. org/abs/2405.14573. Alan Romano, Zihe Song, Sampath Grandhi, Wei Yang, and Weihang Wang. 2021. An empirical analysis of ui-based flaky tests. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 1585–1597. IEEE. Ina K. Schieferdecker. 2025. Navigating the growing field of research on ai for software testing – the taxonomy for ai-augmented software testing and an ontology-driven literature survey. Preprint, arXiv:2506.14640. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024a. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024b. Osworld: Benchmarking multimodal agents for openended tasks in real computer environments. Preprint, arXiv:2404.07972. Shengcheng Yu, Yuchen Ling, Chunrong Fang, Quan Zhou, Yi Zhao, Chunyang Chen, Shaomin Zhu, and Zhenyu Chen. 2026. Scenario-guided llm-based mobile app gui testing. Preprint, arXiv:2506.05079. Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, and Jianwei Yin. 2024. Gui testing arena: A unified benchmark for advancing autonomous gui testing agent. Preprint, arXiv:2412.18426. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606.
10
are consistent, Detect is marked as true. Together, the Trajectory Retriever and the two defect-specific verifiers produce the final Reach, Trigger, and Detect verdicts for the trajectory, which are then used by the downstream metrics in Section 5.
is shown in Table 6, the prompt used by the display verifier is shown in Table 7, and the prompt used by the interaction verifier is shown in Table 8.
Input: agent trajectory τ , agent defect report ρ, navigation task g, state-level test basis B Output: verdicts (reach, trigger, detect) Init: reach ← False; trigger ← False; detect ← False; segments ← ∅.
B.1
Process-Level Metric: Pass@1
B.2
// Stage 1: Trajectory Retriever 1 t ← 1; 2 while t ≤ |τ | do 3 if M ATCH(τ [t], Bprecondition ) then 4 reach ← True; 5 s ← t; 6 u ← t + 1; 7 while u ≤ |τ | do 8 if M ATCH(τ [u], Bevidence ) then 9 segments ← segments ∪ {τ [s : u]}; 10 t ← u; 11 break;
C
More Cases
C.1
Display Defect, Element Layout
Figure 6 shows an Element Layout defect on a music application. The task asks the agent to open the settings page through the navigation drawer. Once the settings list is reached, the icons for the Appearance Settings and Local music paths entries are noticeably larger than the icons of the other rows, breaking the uniform icon-size and text-start alignment that the rest of the list maintains. Because the anomaly is fully visible from a single rendered state, GUIJudge’s Display Verifier identifies the defect without any interaction-level reasoning.
t ← t + 1;
// Stage 2a: Display Defect Verifier if segments ̸= ∅ then 15 foreach segment σ ∈ segments do 16 foreach state o ∈ σ do 17 r ← fθdisp (o); 18 if r.has_defect then 19 display_detected ← True; 20 trigger ← ⊥; // not applicable for display defects 21 if C ONSISTENT(ρ, r) then 22 detect ← True;
14
C.2
C.3 return (reach, trigger, detect);
A.2
Display Defect, Content Rendering
Figure 7 shows a Content Rendering defect on a recipe application. The agent is asked to open the settings page and then tap Preferred category. The resulting dialog lists three options, but the third option renders as a row of garbled placeholder squares rather than legible text, indicating a font or character-encoding failure for that entry. The defect is exposed by a single GUI state, so the Display Verifier can identify the display defect based on a single screenshot.
// Stage 2b: Interaction Defect Verifier 23 if segments ̸= ∅ then 24 foreach segment σ ∈ segments do 25 H ← ∅; 26 ∆ ← T RANSITIONS(σ); 27 foreach transition (ot , at , ot+1 ) ∈ ∆ do 28 H ← H ∪ {(ot , at , ot+1 )}; 29 rt ← fθint (g, B, H, (ot , at , ot+1 )); 30 if rt .has_defect then 31 interaction_detected ← True; 32 trigger ← True; 33 if C ONSISTENT(ρ, rt ) then 34 detect ← True;
35
Process-Level Metric: Pass@3
Table 5 reports Pass@3 results on GUITestScape across the Reach, Trigger, and Detect stages and five fault modes for both general models and specialized agent models.
u ← u + 1;
13
Additional Experimental Results
Table 4 reports Pass@1 results on GUITestScape across the Reach, Trigger, and Detect stages and five fault modes for both general models and specialized agent models.
Algorithm 1: GUIJudge Workflow.
12
B
Interaction Defect, Operation No Response
Figure 8 shows an Operation No Response defect on a tasks application. The agent opens the search field, types “Exercise”, and taps the keyboard search button. The expected post-state is a task list filtered to entries matching the query, but the post-state is identical to the pre-state: the full
Prompt
Each module of GUIJudge is driven by a dedicated prompt. The prompt used by the trajectory retriever 11
Display Defects Model
Type
CR
Interaction Defects
EL
NLE
ONR
Overall UTR
Overall
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
General General General General Agent Agent Agent Agent
44.57 80.43 89.13 73.91 52.17 19.78 40.22 66.30
61.65 89.16 94.25 85.00 68.57 33.03 57.36 79.74
34.30 83.14 90.12 84.88 76.74 31.98 37.21 66.86
51.08 90.79 94.80 91.82 86.84 48.46 54.24 80.14
52.46 77.05 85.25 80.33 67.21 21.31 32.79 77.05
68.82 87.04 92.04 89.09 80.39 35.14 49.38 87.04
42.67 78.67 86.67 73.33 61.33 21.33 21.33 62.67
59.81 88.06 92.86 84.62 76.03 35.16 35.16 77.05
51.85 75.93 87.96 77.78 73.15 25.93 22.22 79.63
68.29 86.32 93.60 87.50 84.49 41.18 36.36 88.66
43.31 79.72 88.39 79.13 68.11 25.64 31.69 70.08
60.44 88.72 93.83 88.35 81.03 40.82 48.13 82.41
Trigger Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
General General General General Agent Agent Agent Agent
44.57 80.43 89.13 73.91 52.17 19.78 40.22 66.30
61.65 89.16 94.25 85.00 68.57 33.03 57.36 79.74
34.30 83.14 90.12 84.88 76.74 29.65 37.21 66.86
51.08 90.79 94.80 91.82 86.84 45.74 54.24 80.14
39.34 57.38 80.33 63.93 45.90 21.31 29.51 60.66
56.47 72.92 89.09 78.00 62.92 35.14 45.57 75.51
34.67 54.67 69.33 56.00 30.67 16.00 10.67 48.00
51.49 70.69 81.89 71.79 46.94 27.59 19.28 64.86
33.33 39.81 64.81 48.15 44.44 10.19 4.63 49.07
50.00 56.95 78.65 65.00 61.54 18.49 8.85 65.84
36.61 66.14 80.31 68.31 54.92 20.71 25.98 59.45
53.60 79.62 89.08 81.17 70.90 34.31 41.25 74.57
Detect Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
General General General General Agent Agent Agent Agent
23.91 46.74 53.26 28.26 5.43 1.09 9.78 1.09
33.85 59.72 65.33 42.62 9.71 2.13 17.65 2.15
5.23 30.23 44.19 22.09 1.74 1.74 3.49 1.16
8.29 42.98 56.30 33.63 3.28 3.39 6.63 2.29
19.67 47.54 40.98 27.87 29.51 1.64 4.92 8.20
24.74 55.24 48.08 37.36 37.89 2.99 8.70 14.29
17.33 41.33 34.67 28.00 8.00 0.00 2.67 9.33
21.49 46.62 45.22 33.87 10.34 0.00 4.82 15.38
16.67 27.78 27.78 14.81 5.56 0.93 0.00 3.39
20.34 33.89 35.71 20.51 7.74 1.75 0.00 6.06
15.75 36.42 40.55 23.23 7.48 1.18 3.94 4.13
21.48 46.19 51.05 32.82 11.66 2.26 7.30 7.62
Reach Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
Table 4: Stage-wise Pass@1 performance on GUITestScape across Reach, Trigger, and Detect, broken down by fault modes for general models and specialized agent models. Bold marks the best result in each column within each stage, and underline marks the second-best.
C.5
unfiltered list (Submit Status Report, Attend Staff Meeting, etc.) is still shown. Because the agent’s action is correct (“hit” on the search key is true) yet the interface produces no observable effect, the Interaction Verifier classifies the transition as an Operation No Response defect.
C.4
Interaction Defect, Unexpected Task Result
Figure 10 shows an Unexpected Task Result defect on a stock-information application. The agent is asked to switch the News tab into Chinese by tapping the language button at the top-right and selecting “Simplified Chinese”. After the selection, the language indicator visibly updates from EN to ZH, suggesting the preference was accepted, yet the actual interface content: news headlines, navigation tabs, and body text remains entirely in English. The post-state contradicts the task intent, so the Interaction Verifier classifies the case as an Unexpected Task Result.
Interaction Defect, Navigation Logic Error
Figure 9 shows a Navigation Logic Error on a social-image application. The task asks the agent to inspect the Report Pin reason list and then dismiss it by tapping the “X” button in the top-left corner. The expected behavior is to close only the report dialog and return to the previous pin-detail context; instead, the application returned to the main interface, performing a full navigation reset rather than a dialog dismissal. The Interaction Verifier flags this abnormal navigation path as a Navigation Logic Error. 12
Display Defects Model
Type
CR
Interaction Defects
EL
NLE
ONR
Overall UTR
Overall
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
Recall↑
F1↑
General General General General Agent Agent Agent Agent
58.70 86.96 92.39 89.13 61.96 32.61 55.43 77.17
73.98 93.02 96.05 94.25 76.51 49.18 71.83 87.12
47.67 86.63 93.02 91.86 83.72 45.93 51.16 76.16
64.54 92.83 96.39 95.76 91.14 62.96 67.69 86.47
64.75 83.61 88.52 85.25 73.77 36.07 45.90 81.97
78.57 91.07 93.91 92.04 84.91 53.01 62.92 90.09
63.89 82.67 90.67 88.00 66.67 34.67 34.67 69.33
77.99 90.51 95.10 93.62 80.00 51.49 51.49 81.89
56.00 80.56 95.37 91.67 85.19 40.74 37.04 87.04
68.19 89.23 97.63 95.65 92.00 57.89 54.05 93.07
55.91 84.45 92.52 89.96 76.38 39.17 47.24 78.35
71.72 91.57 96.11 94.72 86.61 56.02 64.16 87.86
Trigger Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
General General General General Agent Agent Agent Agent
58.70 86.96 92.39 89.13 61.96 32.61 55.43 77.17
73.98 93.02 96.05 94.25 76.51 49.18 71.83 87.12
47.67 86.63 93.02 91.86 83.72 43.02 51.16 76.16
64.54 92.83 96.39 95.76 91.14 60.33 67.69 86.47
50.82 67.21 80.33 73.77 50.82 31.15 39.34 62.30
67.39 80.39 89.09 84.91 67.39 47.46 56.47 76.77
45.33 64.00 77.33 73.33 40.00 26.67 18.67 52.00
62.39 78.05 87.22 84.62 57.14 42.11 31.46 68.42
43.52 50.00 76.85 66.67 55.56 18.52 13.89 59.26
60.65 66.67 86.91 80.00 71.43 31.25 24.39 74.42
47.64 73.23 85.63 81.10 63.39 32.68 39.76 67.52
64.52 84.55 92.26 89.57 77.59 49.26 56.92 80.61
Detect Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
General General General General Agent Agent Agent Agent
34.78 66.67 58.02 39.19 14.58 1.63 10.33 1.69
45.71 76.03 69.63 54.21 22.95 3.18 18.24 3.33
11.05 41.10 66.67 36.60 9.77 2.33 4.65 1.64
17.34 55.81 76.63 51.38 17.11 4.46 8.89 3.23
30.33 77.50 79.59 70.45 58.06 2.46 6.56 8.33
37.76 72.09 82.11 73.81 59.02 4.60 11.65 14.63
28.00 90.91 77.78 77.55 66.67 1.33 4.00 16.22
33.87 75.47 80.77 72.38 43.84 2.48 7.41 24.49
27.78 73.58 69.88 57.14 46.43 1.39 1.39 5.56
33.33 63.93 75.32 61.54 47.27 2.63 2.63 9.84
23.62 61.36 68.59 49.74 27.40 2.05 4.25 4.47
31.95 66.46 76.37 60.25 35.01 3.92 7.92 8.24
Reach Stage GPT-5.4 Gemini-3.1-pro Seed-1.8 Qwen3-VL-Plus UI-TARS-1.5-7B GUI-Owl-1.5-8B GUI-Owl-1.5-32B MAI-UI-8B
Table 5: Stage-wise Pass@3 performance on GUITestScape across Reach, Trigger, and Detect, broken down by fault modes for general models and specialized agent models. Bold marks the best result in each column within each stage, and underline marks the second-best. You are an advanced Android GUI trajectory retrieval expert. Your task is not to judge whether a defect has occurred, but to determine whether the current screenshot has already reached or presented the given key page state. Page Key Point: {current_kp} Judgment Principles: - Perform page state matching only: focus on whether the page, pop-ups, menus, lists, main titles, core controls, and visible content in the current screenshot are semantically consistent with the key point. - Exact text match is not required. As long as the page type, main UI structure, key controls, or key content basically match the description, judge matched=true. - If the key point describes a pop-up, menu, settings page, detail page, list page, or editing page, judge matched=true whenever that core interface is currently visible, even with minor occlusions, toasts, differing selection states, scroll positions, or language/format details. - If the screenshot is the same core page before or after this key state and the system is already within the relevant functional flow, also judge matched=true. - Only judge matched=false when the current screenshot clearly belongs to an unrelated app, unrelated page, error page, or splash screen, or is missing the core page, control, or content described in the key point. Output: Strict JSON only, must include matched and reason: { "matched": true/false, "reason": "..." }
Table 6: Trajectory Retriever Prompt
13
Task: Open the navigation drawer, then access the settings interface through the navigation bar to view all available setting options. Defect：The icons on the settings page are misaligned and fail to maintain consistent positioning with their corresponding text labels.
1.Open the drawer
2.Click “settings”
Elements Layout Error
Normal Elements Layout
Judgement: The settings menu exhibits distinct layout inconsistencies within its list-item components. The leading icons for each setting option vary significantly in size and aspect ratio (the "Music cache limit" icon appears horizontally stretched, while the "Appearance Settings" icon is much smaller). This irregular text alignment disrupts the uniform layout of the local module, resulting in a jagged and unpolished visual experience.
Figure 6: Example of element layout defect. The task requires opening the settings page via the navigation drawer. After reaching the settings list, the icons of the Appearance Settings and Local music paths entries appear noticeably larger than the icons of the other rows, breaking the uniform icon-size and text-start alignment that the rest of the list maintains and exposing an element-layout anomaly.
Task: Open the navigation drawer, go to the settings page, then click on 'Preferred category' to view the available category options. Defect：The third option in the selection dialog displays as garbled square characters, failing to render the intended text correctly.
1.Open the drawer
3.Click “Preferred category”
2.Click “settings”
Content Rendering Error
Judgement: The third option in the 'Preferred category' dialog displays two square placeholder blocks instead of readable text, indicating that the app's font does not support the required characters or there is an underlying text encoding issue affecting this specific list item.
Figure 7: Example of content rendering defect. The task requires opening the settings page and then tapping Preferred category. The resulting dialog lists three options, but the third option renders as a row of garbled placeholder squares instead of legible text, indicating a font or character-encoding failure for that entry.
14
Task: Click Search, type 'Exercise', then press Enter to start the search. Defect：After pressing search, the task list should filter to show only tasks matching 'Exercise'. Instead, the post-state shows the same unfiltered list
1.Open the search 2.Type “Exercise”
Operation No Response
Expected State
3.Clice “search”
Judgement:Agent correctly hit the keyboard search button (magnifying glass) with Hit=True. After pressing search, the task list should filter to show only tasks matching 'Exercise'. Instead, the post-state shows the exact same unfiltered list (Submit Status Report, Attend Staff Meeting, Credit Card Payment, Car Maintenance) with no change whatsoever. The search function failed to filter results even after explicitly triggering the search action, confirming the app's search functionality is broken.
Figure 8: Example of operation no response defect. The task requires opening the search field, typing “Exercise”, and tapping the keyboard search button to filter the list. After the search action, the task list remains identical to the pre-state with the full unfiltered list still shown, demonstrating that the search interaction produces no observable feedback or state change despite a correct hit on the search key.
Task: Confirm if there is an option for Japanese law on the 'Report Pin' page, and then help me click the 'X' in the top-left corner to exit." Defect：'Report Pin' Reason Selection Page Returned Abnormally, jumping back to the home feed instead of the previous screen.
3.Click “X”
2.Click “Report Pin”
1.Click “Picture”
Actual State
Expected State
Judgement:"Clicking the 'X' close button on the 'Report Pin' dialog should dismiss the dialog and return the user to the pin detail page or the previous context where the report was initiated. Instead, the app navigated all the way back to the main 'For you' feed home screen. This is an abnormal return path — closing a report dialog should not cause a full navigation reset to the home feed."
Figure 9: Example of navigation logic error defect. The task requires inspecting the Report Pin reason list and then dismissing the dialog by tapping the “X” button in the top-left corner. After tapping the close button, the app navigates all the way back to the main interface, performing an abnormal navigation reset rather than a dialog dismissal.
15
Task: I cannot understand English news; please help me switch to the 'News' tab and click the top right to change the language to Chinese. Defect：The language switch in the APP is not complete, leaving all news content and UI text still in English.
2.Click the language option 3.Click “Chinese”
1.Click “News”
Judgement:"The user clicked the 'Chinese (Simplified)' option and the selection was successful; the language button indeed changed from 'EN' to 'ZH', but the interface content did not switch to Chinese at all. All news headlines (such as 'Dow Jumps 1,300 Points', 'Stocks Rally As Oil Plunges'), navigation tabs (Market, Screener, News, etc.), and page text remained in English. This is a typical incomplete language-switching defect: the system recorded the language preference but did not actually apply it to the interface content, which does not meet the expected result of 'switching to Chinese' as required by the task."
Figure 10: Example of unexpected task result defect. The task requires switching the News tab into Chinese by tapping the language button at the top-right and selecting “Simplified Chinese”. After the selection, the language indicator updates from EN to ZH, yet the actual interface content remains entirely in English, contradicting the task intent of switching to Chinese.
16
You are a strict, evidence-driven Android GUI Display Defects Detector. You will receive one app screenshot and an XML summary of the page. Your task is to determine, based on visible content in the screenshot aided by the XML, whether display defects exist. Rules - Inspect only what is already visible in the screenshot, with the XML as auxiliary information. - Output has_defect=true only if there is a clear, describable, and locatable visible anomaly belonging to one of the two defect types below; otherwise output false. Defect Types 1. DD.ContentRendering - Images or icons that are missing, broken, blank, or failed to load. - Garbled text, abnormal characters, or placeholder values such as null / None / undefined. - Text that is occluded, clipped, or truncated, making it incomplete or unreadable. - The page structure clearly indicates that content should be present, but an obvious content gap or rendering failure is visible. 2. DD.ElementLayout - Similar elements are obviously missing, misaligned, abnormally sized, or abnormally positioned. - Elements overlap, drift, escape their container, or are only partially visible. - Within a repeating structure, an element is not aligned with its peers. - Duplicate or extraneous elements, or anomalous blank slots, disrupt the page structure. Evidence Requirements - evidence should contain only directly visible facts. - Prioritize describing the anomalous object and its comparison with similar objects or local boundaries. - location_hint provides an approximate location. Output: Strict JSON in the following format: { "has_defect": true/false, "defects": [{ "type": "None | DD.ContentRendering | DD.ElementLayout", "evidence": ["visible evidence from the screenshot"], "location_hint": "approximate location of anomaly", "reason": "one-sentence explanation of why it belongs to this type" }] }
Table 7: Display Defect Verifier Prompt
17
You are an Android GUI interaction defect verification expert. Your task is to judge, based on the candidate tasks, historical observations, the textual trace of the current step, and the pre/post screenshots of the current step, whether the result of the current step reflects an interaction defect. Inputs. You will receive: - Candidate tasks. - Actual observations of historical steps. - The current step to verify, including its thought, action, target, hit, pre-state, and post-state. - Two screenshots of the current step: pre and post. Defect Types 1. ID.OperationNoResponse — The user performed a reasonable interaction, but the interface did not give the expected feedback, or the obvious result that this interaction should have produced did not appear. 2. ID.NavigationLogicError — The current step caused the app to navigate to a wrong page, unrelated page, or incorrect navigation path, or the back/forward navigation logic is clearly abnormal. 3. ID.UnexpectedTaskResult — The current step produced an incorrect result, wrong state, wrong content, erroneous pop-up, missing result, unupdated state, incomplete setting toggle, or a name/text/file output that does not satisfy the task constraints; or a result that should not have been allowed (or that should have been allowed but was handled incorrectly). Strong Constraints 1. Judge only the current step. - Historical content serves as context only; old issues from previous steps must not be directly treated as defects of the current step. - A defect can be attributed to the current step only when this step clearly caused, propagated and exposed, or reconfirmed the issue. 2. Do not classify agent mistakes as app defects. - Clicking the wrong target, hit=false, input that does not satisfy the task requirements, missing required steps, misunderstanding the task, or never actually reaching the intended control is typically not an app interaction defect. - A defect can be assigned only when evidence shows that even though the current step legitimately hit and executed, the app’s result is still wrong. 3. Decide strictly based on evidence. - Use only the provided textual trace and the current step’s pre/post screenshots. - Do not speculate about hidden flows, background states, undisplayed toasts, or network results that were not provided. Step Number Rules - If a defect is identified, step must be the step number of the current step under verification. - If no defect is identified, step may be either the current step number or 0. Output: Strict JSON only, no Markdown, no code fences: { "has_defect": true/false, "defect": { "type": "None | ID.OperationNoResponse | ID.NavigationLogicError | ID.UnexpectedTaskResult", "step": <int>, "reason": "concise evidence-based reasoning explaining why this is or is not an app defect", "effect": "brief summary of what actually happened in the post-state; may be empty if no defect" } }
Table 8: Interaction Defect Verifier Prompt
18