arXiv:2604.19081v1 [cs.SE] 21 Apr 2026
Proactive Detection of GUI Defects in Multi-Window Scenarios via Multimodal Reasoning Xinyao Zhang, Rui Wang, Jinhao Cui, Haotian Huang Wei Xue School of Computer Science and Artificial Intelligence Dongfeng Motor Corporation Wuhan University of Technology Wuhan, China Wuhan, China [email protected] xinyaozhang, 339993, cuiwut0123, [email protected] Wenhua Hu, Jianwen Xiang, Rui Hao* School of Computer Science and Artificial Intelligence Engineering Research Center of Transportation Information and Safety (ERCTIS), MoE of China Wuhan University of Technology Wuhan, China whu10, jwxiang, [email protected] Abstract—Multi-window mobile scenarios, such as splitscreen and foldable modes, make GUI display defects more likely by forcing applications to adapt to changing window sizes and dynamic layout reflow. Existing detection techniques are limited in two ways: they are largely passive, analyzing screenshots only after problematic states have been reached, and they are mainly designed for conventional full-screen interfaces, making them less effective in multi-window settings. We propose an end-to-end framework for GUI display defect detection in multi-window mobile scenarios. The framework proactively triggers split-screen, foldable, and window-transition states during app exploration, uses Set-of-Mark (SoM) to align screenshots with widget-level interface elements, and leverages multimodal large language models with chain-of-thought prompting to detect, localize, and explain display defects. We also construct a benchmark of GUI display defects using 50 real-world Android applications. Experimental results show that multi-window settings substantially increase the exposure of layout-related defects, with text truncation increasing by 184% compared with conventional full-screen settings. At the application level, our method detects 40 defect-prone apps with a false positive rate of 10.00% and a false negative rate of 11.11%, outperforming OwlEye and YOLO-based baselines. At the finegrained level, it achieves the best F1 score of 87.2% for widget occlusion detection. Index Terms—GUI testing, display defect detection, multiwindow interfaces, multimodal large language models, visual grounding
I. I NTRODUCTION Mobile applications are increasingly used beyond the conventional full-screen smartphone setting. With the rapid adoption of foldable devices, tablets, split-screen multitasking, and freeform windows, modern apps must continuously adapt their interfaces to changing window sizes, aspect ratios, and display modes [1]–[3]. Such adaptations are error-prone: when the available display area shrinks or changes dynamically, defects such as text truncation, widget occlusion, layout overlap, and conflicts with system UI elements become more likely to occur and more visible to users. These defects directly degrade readability, operability, and overall user experience [4]–[6].
Existing GUI display defect detection techniques, however, are not designed for this new usage context. Prior approaches are largely passive: they take already-captured screenshots as input and determine whether a display defect exists [5], [6]. While effective for post hoc inspection, such methods cannot proactively steer app exploration toward defect-prone interface states. Moreover, existing studies, benchmarks, and detection models mainly focus on conventional full-screen smartphone UIs, with limited support for split-screen, foldable, and other dynamic multi-window scenarios [4]. Detecting display defects in multi-window environments is challenging for two reasons. First, triggering defects is difficult. Many adaptation-related defects manifest only when an app undergoes specific window transitions, such as entering split-screen mode, resizing to a smaller region, or switching display posture on foldable devices. Traditional exploration tools, which primarily traverse apps under a fixed full-screen configuration, cannot systematically cover such scenarios [7]. Second, identifying defects is difficult. Multi-window interfaces often contain visually ambiguous patterns: apparent overlap or partial occlusion may either indicate a true defect or reflect intentional design choices, such as floating action buttons, badge counters, or transient overlays. Therefore, methods based only on heuristic rules, layout metadata, or purely visual classification often lack the semantic understanding needed to distinguish acceptable UI compositions from genuine display defects [4]–[6]. In this paper, we propose an end-to-end framework for GUI display defect detection in split-screen, foldable, and dynamic window-transition scenarios. Our key idea is to integrate proactive defect-triggering exploration with multimodal defect reasoning. On the exploration side, we extend DroidBot [7] to actively induce split-screen, foldable, and window-resizing scenarios during app traversal, thereby increasing the likelihood of reaching defect-prone states. On the analysis side, we use Set-of-Mark (SoM) [8] to align screenshots with widgetlevel interface elements, and then employ multimodal large
However, these methods depend heavily on predefined rules and thresholds, making them less robust to cross-application variation, resolution differences, and dynamic layout changes in split-screen and foldable scenarios. They also lack explicit modeling of interface semantics, and thus struggle to distinguish acceptable occlusion (e.g., badges or floating action buttons) from true defects. B. Deep Learning-based GUI Testing Fig. 1. Examples of amplified display defects in split-screen and foldable scenarios.
language models with chain-of-thought prompting (CoT) [9] to reason about defect existence, localization, and explanation. In this way, the framework goes beyond passive screenshot inspection and supports active discovery of display defects in realistic multi-window usage settings. The main contributions of this paper are as follows: • We identify a new testing gap in GUI display defect detection: existing methods are predominantly reactive, screenshot-based, and centered on full-screen smartphone UIs, while modern multi-window mobile environments require both active defect triggering and context-aware defect interpretation. • We present an exploration framework that extends DroidBot to systematically trigger defect-prone split-screen, foldable, and window-transition scenarios during dynamic app traversal. • We develop a multimodal defect detection approach that combines SoM-based widget grounding with MLLM reasoning to detect, localize, and explain display defects in complex multi-window interfaces. • We construct, to the best of our knowledge, the first benchmark of GUI display defects in multi-window mobile scenarios using 50 Android apps, and use it to evaluate the proposed approach against existing baselines. II. R ELATED W ORK Research on mobile GUI quality assurance can be broadly grouped into three lines: (i) heuristic- or rule-based usability evaluation and interface inspection; (ii) deep learningbased GUI exploration and defect recognition; and (iii) LLM/MLLM-based script generation and interaction decisionmaking. Different from conventional full-screen settings, this work focuses on display and adaptation defects caused by window mode changes, especially split-screen and foldable states, where both dynamic scenario triggering and semanticlevel defect identification are required. A. Traditional GUI Testing Methods Traditional studies mainly rely on heuristic evaluation, remote usability testing, and interaction log analysis. Representative work includes Scholtz’s remote usability framework [10], mobile-oriented data collection platforms [11], mobile usability and gesture heuristics [12]–[14], and GUI trace visualization tools for reducing manual analysis effort [15].
Deep learning has improved GUI exploration and defect recognition. Deep GUI predicts touch targets by heatmap regression to improve exploration on sparse interfaces [16]. Humanoid and MUBot learn action strategies from human interaction traces [17], [18], while reinforcement learning has also been introduced to improve coverage and fault detection [19], [20]. For display defect detection, OwlEye applies convolutional neural network (CNN)-based modeling to identify multiple defect types from screenshots [6]. Nevertheless, defect annotations are scarce and cross-app distribution shifts are substantial, limiting generalization. More importantly, purely visual methods do not explicitly model widget semantics or design intent, and therefore remain prone to semantically dependent misclassification. C. LLM-based GUI Testing Large language models (LLMs) further extend GUI testing to script generation and interaction decision-making. GPTDroid formulates testing as a question-answering-based decision process [21]; InputBlaster generates anomalous inputs for text widgets to improve crash discovery [22]; and DroidBotGPT textualizes GUI states and candidate actions for LLMbased action selection [23]. However, existing studies mainly target functional testing rather than screenshot-level display defect detection. They usually rely on textualized GUI states, but lack explicit detection objectives and explainable oracles for visual defects. Although multimodal large language models (MLLMs) offer the potential to jointly use visual evidence and widget semantics, practical issues such as grounding stability, reproducibility, and reasoning cost remain. III. A PPROACH A. Method Overview As shown in Fig. 2, we propose an end-to-end framework for interface display defect detection in split-screen, foldable, and window-resizing scenarios. The framework addresses dynamic scenario triggering and semantic-aware defect identification in four stages. First, an enhanced DroidBot performs split-screen, foldable, and window-transition operations during exploration and collects screenshots, widget trees, and runtime context. Second, key widgets are filtered from the raw widget tree and serialized into structured user interface (UI) metadata. Third, a Set-of-Mark (SoM) representation is constructed to align screenshot regions with widget semantics. Finally, the marked screenshots and structured UI metadata are fed into a multimodal large language model with chain-of-thought prompting for defect diagnosis, localization, and explanation.
Fig. 2. Framework overview of our approach. Enhanced DroidBot collects screenshots and view hierarchies across multiple window states, and the resulting multimodal inputs are analyzed with CoT-guided reasoning for defect diagnosis, localization, and structured reporting.
B. Stage I: Multi-State Screenshot Generation and Evidence Collection DroidBot [7] is a model-based Android testing framework that extracts GUI information during exploration and builds a state model for event selection and path planning. It can also synchronously collect reproducible evidence, including GUI screenshots and the UI hierarchy. In our framework, we extend DroidBot with split-screen, foldable, and window-transition operations to support multi-window exploration and evidence collection. To expose defects that are difficult to trigger in conventional full-screen scenarios, we extend DroidBot to perform four concrete window operations during exploration, namely Split Test, Fold Test, Drag Up, and Drag Down. These operations cover three representative transition categories: entering split-screen mode and adjusting the split ratio, switching foldable states, and resizing the window by dragging its boundaries. For each stable interface state, we collect an aligned evidence triplet: ⟨𝐼, 𝑈, Γ⟩,
(1)
where 𝐼 is the raw screenshot, 𝑈 is the UI hierarchy (widgettree JSON), and Γ is the runtime context, including the app, Activity, window-mode parameters, and timestamp. This triplet is the unified input for subsequent filtering, alignment, and defect reasoning.1 C. Stage II: Key Widget Filtering and Ordering The widget tree often contains many decorative or container nodes. Directly using it as input introduces noise and increases reasoning cost. We therefore perform lightweight widget abstraction and filtering, and then derive an approximate reading order. 1) Widget Abstraction: From 𝑈, we parse a widget set 𝑛 , where each widget is represented as C = {𝑐 𝑖 }𝑖=1 𝑐 𝑖 = (𝑖𝑑𝑖 , 𝑡𝑦 𝑝𝑒 𝑖 , 𝑡𝑒𝑥𝑡𝑖 , 𝑏 𝑖 , 𝑐𝑙𝑖𝑐𝑘 𝑖 , 𝑟𝑖𝑑𝑖 ),
where 𝑏 𝑖 = (𝑥1𝑖 , 𝑦 1𝑖 , 𝑥2𝑖 , 𝑦 2𝑖 ) is the bounding box, 𝑐𝑙𝑖𝑐𝑘 𝑖 is runtime clickability, and 𝑟𝑖𝑑𝑖 is the resource identifier. 2) Lightweight Filtering: To remove nodes irrelevant to defect identification, we define a binary decision function 𝜑(·) that preserves only interactive or semantically meaningful widgets while filtering out extremely small elements, decorative backgrounds, and empty containers: h i 𝜑(𝑐 𝑖 ) = I area(𝑏 𝑖 ) ≥ 𝜏𝑎 ∧ ¬ Decor(𝑟𝑖𝑑𝑖 ) ∧ Sem(𝑐 𝑖 ) . (3) Here, Sem(𝑐 𝑖 ) indicates whether 𝑐 𝑖 is semantically relevant, i.e., clickable, text-bearing, or of a common interactive type. Specifically, 𝑐𝑙𝑖𝑐𝑘 𝑖 captures runtime clickability, whereas IsInteractive(·) encodes prior knowledge of interactive widget classes when runtime flags are unreliable. In addition, 𝜏𝑎 is the minimum area threshold; Decor(·) filters background and separator elements by decorative keywords or resource patterns; HasText(·) checks whether readable text is present; and IsInteractive(·) denotes common interactive widget types such as Button, EditText, and Switch. We further deduplicate text widgets representing different states of the same function using normalized keys. 3) Adjacency Relations and Ordering: To preserve consistency between the input sequence and spatial layout, we construct adjacency relations among retained widgets and derive an approximate reading order. For any two widgets 𝑐 𝑖 and 𝑐 𝑗 , the distance between their centers is 𝑑𝑖 𝑗 = 𝑐𝑒𝑛𝑡𝑒𝑟 (𝑏 𝑖 ) − 𝑐𝑒𝑛𝑡𝑒𝑟 (𝑏 𝑗 ) 2 .
If 𝑐 𝑖 and 𝑐 𝑗 overlap in their horizontal or vertical projections and the gap between their edges is below a threshold, we connect them to form an adjacency graph 𝐺 = (𝑉, 𝐸). Starting from the widget with the smallest (𝑦 1 , 𝑥1 ), we perform depthfirst search on 𝐺 and prioritize nearer neighbors to obtain an ordered sequence
(2) 𝜋 = [𝑐 𝜋 (1) , . . . , 𝑐 𝜋 (𝑚) ],
1 Detailed implementation of the enhanced multi-window testing framework
is provided in Appendix C.
(4)
which is used in Stages III and IV.
(5)
Based on 𝜋, we serialize retained widgets into structured UI metadata. Let 𝑢˜ 𝑖 = (𝑖𝑑𝑖 , 𝑡𝑦 𝑝𝑒 𝑖 , 𝑡𝑒𝑥𝑡 𝑖 , 𝑏 𝑖 , 𝑐𝑙𝑖𝑐𝑘 𝑖 , 𝑟𝑖𝑑𝑖 ),
(6)
denote the serialized representation of widget 𝑐 𝑖 . The structured UI metadata is then e = [ 𝑢˜ 𝜋 (𝑘 ) ] 𝑚 . 𝑈 𝑘=1
(7)
This metadata preserves the key attributes needed for subsequent marking and multimodal reasoning.2 D. Stage III: SoM Construction and Precise Grounding In multi-window scenarios, defect identification requires both visual evidence and structural semantics. When regions are described only in natural language, multimodal models are prone to ambiguous references and localization errors. Set-of-Mark (SoM) [8] mitigates this problem by overlaying symbolic markers on key widgets and linking them to widget attributes, enabling precise reference by marker ID. After obtaining the ordered key widgets, we construct a Set-of-Mark representation for precise grounding. Each retained widget is assigned a unique marker 𝑚 𝑖 , which is overlaid on the raw screenshot 𝐼 to generate a marked image 𝐼 SoM . We also generate a mapping table from markers to widget attributes: 𝑚 M = {(𝑚 𝑖 , 𝑡𝑦 𝑝𝑒 𝑖 , 𝑡𝑒𝑥𝑡𝑖 , 𝑏 𝑖 , 𝑐𝑙𝑖𝑐𝑘 𝑖 )}𝑖=1 .
(8)
Together, the raw screenshot, the marked image, and the mapping table form the mark set. SoM converts ambiguous visual references into referable symbolic IDs, enabling the model to localize defect-related regions through {𝑚 𝑖 } and improving explanation verifiability.3 E. Stage IV: CoT-Constrained Multimodal Reasoning and Structured Reporting We use the marked screenshot 𝐼 SoM as visual input, and the serialized marker mapping M together with runtime context Γ as textual input to a multimodal large language model. Formally, let 𝑆 = Serialize(M, Γ), (9) then the model input is ⟨𝐼 SoM , 𝑆⟩. CoT prompting constrains the model to reason in three steps: 1) structured interface understanding: identify highrisk regions and key elements under the current window mode and layout; 2) multimodal defect analysis: jointly use the marked screenshot and structured UI metadata to verify candidate defect types, such as text overlap/truncation, widget occlusion, missing images, null display, conflicts with system UI, and split-screen/foldable incompatibility; and 3) defect diagnosis and localization: output the final defect type, associated SoM markers, and concise evidence and explanation. 2 The
complete component extraction and filtering procedure, including adjacency thresholds and DFS-based ordering, is detailed in Appendix D. 3 An illustrative example of SoM construction is provided in Appendix E.
The final output is a structured report: R = {𝑡𝑦 𝑝𝑒, 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛, 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒, 𝑒𝑥 𝑝𝑙𝑎𝑛𝑎𝑡𝑖𝑜𝑛},
(10)
where 𝑡𝑦 𝑝𝑒 is the defect category, 𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 is the set of SoM marker indices, 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 contains verifiable observations such as overlapping text fragments, occlusion relations, or missing regions, and 𝑒𝑥 𝑝𝑙𝑎𝑛𝑎𝑡𝑖𝑜𝑛 is a concise cause analysis. This output supports both defect inspection and subsequent statistical analysis.4 IV. E XPERIMENTAL S ETUP Apps and scenarios. We evaluate the proposed method on 50 real-world Android applications from diverse categories, including social networking, entertainment, utilities, and education. Each application is tested under two settings to quantify the impact of window-mode changes on defect exposure and detection performance: (i) conventional full-screen and (ii) split-screen/foldable (abbreviated as split/fold). We evaluate the proposed method on 50 real-world Android applications from diverse categories, including social networking, entertainment, utilities, and education.5 Testing pipeline and baselines. For each application, we first perform automated exploration and screenshot collection, then conduct data preprocessing, including widget filtering and SoM construction, and finally perform defect reasoning and report generation. On average, each app yields about 1,800 screenshots, and the end-to-end process takes about 2 hours, for a total runtime of about 100 hours. We compare our method with OwlEye [6] and YOLO [24]. For fairness, all methods use the same screenshots and follow the same annotation criteria. Our method is built on Qwen2.5-VL-32B and adapted via LoRA-based fine-tuning, with rank 8, alpha 16, and dropout 0. We train the model with AdamW for 100 epochs using a learning rate of 5×10−5 , a per-device batch size of 4, an effective batch size of 8, 4-bit quantization, double quantization, and gradient accumulation of 8. Evaluation protocol and metrics. We evaluate from three perspectives: scenario contribution, application-level reliability, and fine-grained defect-type performance. Specifically, we compare the numbers of exposed defects under the conventional and split/fold settings, evaluate whether a method identifies at least one defective screenshot for each application, and report Precision, Recall, and F1 for major defect types. At the application level, reliability is measured by the false positive rate (FPR) and false negative rate (FNR). Let 𝑇 (𝑎) ∈ {0, 1} be the human-annotated ground-truth label for application 𝑎, and let 𝑃(𝑎) ∈ {0, 1} be the method prediction. Then, |{𝑎 | 𝑇 (𝑎) = 0 ∧ 𝑃(𝑎) = 1}| , |{𝑎 | 𝑇 (𝑎) = 0}| |{𝑎 | 𝑇 (𝑎) = 1 ∧ 𝑃(𝑎) = 0}| FNR = . |{𝑎 | 𝑇 (𝑎) = 1}| FPR =
(11)
4 A worked example of the multimodal chain-of-thought prompting process is provided in Appendix F. 5 Representative defective applications, together with their tested versions, OS versions, devices, and download sources, are listed in Appendix A.
Fig. 3. Triggered defect counts under conventional and split/fold settings.
TABLE II N UMBER OF DETECTED INSTANCES FOR EACH DEFECT TYPE , TOGETHER WITH THE CORRESPONDING SPLIT / FOLD COUNTS AND SHARES .
TABLE I C OARSE - GRAINED DETECTION COUNTS .
Method
Conventional
Split/Fold
#screenshots #apps #screenshots #apps OwlEye YOLO Ours
1261 1334 1503
22 24 32
Fig. 4. Application-level FPR and FNR under conventional and split/fold settings. Lower is better.
3567 4186 5069
27 30 40
V. R ESULTS A. RQ1: Defect exposure under split-screen/foldable scenarios We first examine whether split-screen and foldable operations expose more interface adaptation defects. For each application, we explore both the conventional and split/fold settings, collect screenshots, manually identify defective screenshots, and count defects by type. Fig. 3 summarizes the results. Split-screen/foldable settings substantially increase defect exposure, especially for layout-related defects. Compared with the conventional setting, text overlap increases from 644 to 1526 (+137%), text truncation from 1242 to 3528 (+184%), and widget occlusion from 414 to 1224 (+196%). By contrast, missing image and null display increase only slightly. These results indicate that stronger spatial constraints and layout reflow amplify defects that are difficult to trigger in full-screen mode. B. RQ2: Coarse-grained comparison at the screenshot and app levels We next compare coarse-grained performance at the application level, without distinguishing defect types. Table I reports the coarse-grained detection counts. Our method consistently detects more defective screenshots and identifies more defect apps than both baselines. Under the conventional setting, it detects 1503 defective screenshots and identifies 32 defect apps, compared with 1261/22 for OwlEye and 1334/24 for YOLO. Under split/fold settings, the advantage becomes larger: our method detects 5069 defective screenshots and identifies 40 defect apps, compared with 3567/27 and 4186/30 for OwlEye and YOLO, respectively.
Defect type
Ours
OwlEye
YOLO
Split/Fold count (Ours)
Text overlap Text truncation Widget occlusion Missing image Null display
1228 2540 1063 192 46
887 2186 612 161 39
995 2398 710 173 41
859 (69.96%) 1803 (70.98%) 765 (71.97%) 31 (16.15%) 8 (17.39%)
Fig. 4 further compares application-level reliability using FPR and FNR. Our method achieves the lowest error rates in both settings. Under the conventional setting, its FPR/FNR are 6.25%/7.14%; under split/fold settings, they are 10.00%/11.11%, still clearly better than the baselines. This shows that our method not only detects more defective apps, but also provides better reliability. C. RQ3: Fine-grained analysis by defect types The third experiment evaluates defect-type-level performance. Table II compares detected instances across five defect types and quantifies the contribution of split/fold settings. Our method detects the largest number of instances for the five major traditional defect types, including text overlap, text truncation, widget occlusion, missing image, and null display. Moreover, text overlap, text truncation, and widget occlusion are mainly exposed under split/fold settings, each with a split share around 70%. Table III reports Precision, Recall, and F1 for the major defect types under split/fold settings. For widget occlusion, our method achieves the best Precision (87.6%), Recall (86.8%), and F1 (87.2%), outperforming OwlEye and YOLO by 28.3 and 21.5 points in F1, respectively. For text overlap, it achieves the highest Recall (80.5%) and best F1 (79.8%), while YOLO attains slightly higher Precision (81.6%). For text truncation, our method achieves the highest Recall (72.0%), whereas OwlEye yields the highest Precision (82.1%) and YOLO the best F1 (73.4%), indicating a precision–recall tradeoff. For the visually intuitive defects missing image and
TABLE III P RECISION , R ECALL , AND F1 SCORES (%) UNDER SPLIT / FOLD SETTINGS .
Defect type
Method Precision Recall
F1
Text overlap
OwlEye YOLO Ours
78.3 81.6 79.2
58.1 65.2 80.5
66.7 72.5 79.8
Text truncation
OwlEye YOLO Ours
82.1 79.5 74.2
62.0 68.0 72.0
70.6 73.4 73.1
OwlEye Widget occlusion YOLO Ours
71.5 75.8 87.6
50.0 58.0 86.8
58.9 65.7 87.2
Missing image
OwlEye YOLO Ours
88.9 92.1 89.7
77.4 83.2 92.3
82.8 87.4 90.9
Null display
OwlEye YOLO Ours
89.7 91.1 90.2
78.0 82.0 92.0
83.4 86.3 91.1
null display, all methods perform strongly, while our method achieves the highest Recall and F1 on both types. VI. C ONCLUSION AND F UTURE W ORK This paper presents an end-to-end framework for interface display defect detection in split-screen, foldable, and other multi-window scenarios. By combining enhanced DroidBotbased scenario triggering, SoM-based visual grounding, and chain-of-thought-guided multimodal reasoning, the framework supports defect detection, localization, and explanation. Experiments on 50 real-world Android applications show that split/fold settings substantially increase the exposure of layoutrelated defects and that our method outperforms representative baselines in both application-level reliability and fine-grained defect detection, especially for semantically challenging defects such as widget occlusion. Future work will extend the framework to more device types and adaptive layout scenarios, improve inference efficiency for large-scale testing, and incorporate richer interaction context to better detect defects arising during continuous user interaction. ACKNOWLEDGMENT This work was partially supported by the National Natural Science Foundation of China (Grant No. 62502356) and the contract research grant from Dongfeng Motor Corporation (Project No. 202501hx1053; Project title: Development Project for a Personalized Driving System Based on End-to-End Autonomous Driving). R EFERENCES [1] Android Developers, “Learn about foldables,” [Online]. Available: https: //developer.android.com/develop/ui/compose/layouts/adaptive/foldables/ learn-about-foldables, 2026, accessed: Mar. 11, 2026. [2] ——, “Support multi-window mode,” [Online]. Available: https://develo per.android.com/develop/ui/views/layout/support-multi-window-mode, 2026, accessed: Mar. 11, 2026.
[3] ——, “Screen compatibility overview,” [Online]. Available: https://de veloper.android.com/guide/practices/screens support, 2026, accessed: Mar. 11, 2026. [4] L. Nie, K. S. Said, and M. Hu, “Sok: An exhaustive taxonomy of display issues for mobile applications,” in Proceedings of the 29th International Conference on Intelligent User Interfaces, 2024, pp. 537–548. [5] Y. Su, C. Chen, J. Wang, Z. Liu, D. Wang, S. Li, and Q. Wang, “The metamorphosis: Automatic detection of scaling issues for mobile apps,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, pp. 1–12. [6] Z. Liu, C. Chen, J. Wang, Y. Huang, J. Hu, and Q. Wang, “Owl eyes: Spotting UI display issues via visual understanding,” in Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering, 2020, pp. 398–409. [7] Y. Li, Z. Yang, Y. Guo, and X. Chen, “DroidBot: A lightweight UIguided test input generator for Android,” in 2017 IEEE/ACM 39th International Conference on Software Engineering Companion (ICSEC). IEEE, 2017, pp. 23–26. [8] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao, “Set-of-Mark prompting unleashes extraordinary visual grounding in GPT-4v,” arXiv preprint arXiv:2310.11441, 2023. [9] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022. [10] J. Scholtz, “Adaptation of traditional usability testing methods for remote testing,” in Proceedings of the 34th Annual Hawaii International Conference on System Sciences. IEEE, 2001, pp. 8 pp.–. [11] H. Liang, H. Song, Y. Fu, X. Cai, and Z. Zhang, “A remote usability testing platform for mobile phones,” in 2011 IEEE International Conference on Computer Science and Automation Engineering, vol. 2. IEEE, 2011, pp. 312–316. [12] O. Machado Neto and M. D. G. Pimentel, “Heuristics for the assessment of interfaces of mobile devices,” in Proceedings of the 19th Brazilian Symposium on Multimedia and the Web, 2013, pp. 93–96. [13] S. R. Humayoun, P. H. Chotala, M. S. Bashir, and A. Ebert, “Heuristics for evaluating multi-touch gestures in mobile applications,” in Proceedings of the 31st International BCS Human Computer Interaction Conference (HCI 2017). BCS Learning & Development, 2017. [14] M. S. Bashir and A. Farooq, “EUHSA: Extending usability heuristics for smartphone application,” IEEE Access, vol. 7, pp. 100 838–100 859, 2019. [15] J. Jeong, N. Kim, and H. P. In, “GUI information-based interaction logging and visualization for asynchronous usability testing,” Expert Systems with Applications, vol. 151, p. 113289, 2020. [16] F. YazdaniBanafsheDaragh and S. Malek, “Deep GUI: Black-box GUI input generation with deep learning,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2021, pp. 905–916. [17] Y. Li, Z. Yang, Y. Guo, and X. Chen, “Humanoid: A deep learningbased approach to automated black-box Android app testing,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 2019, pp. 1070–1073. [18] C. Peng, Z. Zhang, Z. Lv, and P. Yang, “MUBot: Learning to test largescale commercial Android apps like a human,” in 2022 IEEE International Conference on Software Maintenance and Evolution (ICSME). IEEE, 2022, pp. 543–552. [19] M. Pan, A. Huang, G. Wang, T. Zhang, and X. Li, “Reinforcement learning based curiosity-driven testing of Android applications,” in Proceedings of the 29th ACM SIGSOFT international symposium on software testing and analysis, 2020, pp. 153–164. [20] A. Romdhana, A. Merlo, M. Ceccato, and P. Tonella, “Deep reinforcement learning for black-box testing of Android apps,” ACM Transactions on Software Engineering and Methodology (TOSEM), vol. 31, no. 4, pp. 1–29, 2022. [21] Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Chatting with GPT-3 for zero-shot human-like mobile automated GUI testing,” arXiv preprint arXiv:2305.09434, 2023. [22] Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, Z. Tian, Y. Huang, J. Hu, and Q. Wang, “Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model,” in Proceedings of the IEEE/ACM 46th International conference on software engineering, 2024, pp. 1–12.
[23] H. Wen, H. Wang, J. Liu, and Y. Li, “Droidbot-gpt: Gpt-powered ui automation for android,” arXiv preprint arXiv:2304.07061, 2023. [24] G. Jocher and J. Qiu, “Ultralytics YOLO11,” https://github.com/ultraly tics/ultralytics, 2024, version 11.0.0.
A PPENDIX This appendix provides additional implementation details, worked prompting examples, and representative defect cases used in our study. A. Examples of Collected Defective Applications Table IV lists example defective applications collected in our study, together with their tested versions, OS versions, devices, and download sources. TABLE IV E XAMPLES OF COLLECTED DEFECTIVE APPLICATIONS . Application
Tested Version
OS
Test Device
Source
MTG Familiar Minesweeper Signal-Android Super Productivity Raccoon NewPipe-legacy Dagger Cards Score Keeper werewolf Game Clock sockstun Orbot Sky Map toDoListb4a PocketMaps metamask-mobile My Position Super Productivity Metrodroid
3.6.6.dbg.3 1.1 (2) 7.40.2 12.0.1 1.14.0 0.18.6 1.2.0 1.0.3 1.0.1 1.0 2.7 17.4.1 1.9.6 v2.0.0.3 3.7 7.43.0 1.3.5 12.0.1 3.0.0
8 8 12 11 12 8 8 8 8 8 10 15 8 11 8 10 8 11 8
Google Nexus 6 Google Nexus 6 Xiaomi F22 Pro Google Nexus 6 Samsung S10 Google Nexus 6 Google Nexus 6 Google Nexus 6 Google Nexus 6 Google Nexus 6 Pixel 2 Google Pixel 8 Google Nexus 6 vivo v15 Google Nexus 6 iPhone 15 Google Nexus 6 Samsung S10 Google Nexus 6
Google Play Google Play Google Play F-Droid F-Droid GitHub Google Play Google Play Google Play Google Play GitHub GitHub Google Play GitHub Google Play GitHub Google Play F-Droid Google Play
B. Defect Taxonomy and Fine-Tuning Data Distribution To support consistent labeling, model fine-tuning, and evaluation, we define a hierarchical defect taxonomy for GUI adaptation issues in split-screen and foldable scenarios. Based on the collected defect cases, we group the target defects into two major categories: missing/wrong information defects and overlap/occlusion defects. Figure 5 provides representative examples of the major defect categories in our taxonomy. The first category, missing/wrong information defects, refers to cases in which expected interface content is missing, incorrectly rendered, or displayed incompletely. This category includes missing image, null display, and text truncation. Among them, text truncation is further divided into top truncation, bottom truncation, and side truncation, according to which part of the text content is clipped by the layout. The second category, overlap/occlusion defects, refers to cases in which interface elements conflict spatially and one element partially or fully covers another. This category includes widget occlusion, text overlap, and navigation-bar/button overlap. To better characterize different spatial conflicts, widget occlusion is further divided into widget-over-text, text-overwidget, and widget-over-widget cases. This taxonomy serves as the basis for constructing the finetuning dataset and organizing the evaluation results. It also helps distinguish genuine display defects from visually similar but acceptable design patterns, which is particularly important
Fig. 5. Representative examples of the major GUI adaptation defect categories in our taxonomy.
in complex adaptation scenarios where normal overlays and defective occlusions may appear similar at the pixel level. Based on the taxonomy described above, we further refine the major defect categories into finer-grained subtypes to support defect recognition at different levels of granularity. In addition to the two primary GUI defect categories, the finetuning dataset also includes crash and compatibility-related cases, namely app crash, split mismatch, and foldable mismatch. Moreover, the dataset contains normal-display samples, which help the model distinguish genuine GUI defects from visually similar but acceptable interface layouts. Figure 6 shows representative examples of normal and mismatched GUI layouts under foldable and split-screen scenarios.
Fig. 6. Examples of normal and mismatched GUI layouts under foldable and split-screen scenarios.
Table V summarizes the distribution of the fine-tuning data used in our study. C. Enhanced DroidBot Implementation Details We extend DroidBot into an enhanced multi-window testing framework for collecting GUI adaptation data under split-screen and foldable scenarios. As shown in Fig. 7, the framework adopts a modular architecture consisting of a multiwindow controller, a GUI process monitoring module, and a parallel execution engine. The framework supports four window-transition operations, namely Fold Test, Split Test, Drag Up, and Drag Down.
TABLE V D ISTRIBUTION OF THE FINE - TUNING DATA . Major Category
Defect Group
Subtype
Count
– –
Missing image Null display Top truncation Bottom truncation Side truncation
22 7 7 29 6
Missing / wrong information Text truncation
Overlap / occlusion
Widget-over-text 10 Widget occlusion Widget-over-widget 6 Text-over-widget 5 – Text overlap 18 – Navigation-bar/button overlap 2
Crash / Compatibility
– – –
App crash Split mismatch Foldable mismatch
18 2 2
Normal samples
–
Normal display
30
These operations are used to trigger representative windowmode changes during automated exploration. After each window operation, the framework records three types of raw data for subsequent analysis: application information, raw screenshots, and unprocessed widget information. Together, these outputs form standardized screenshot– UI-hierarchy–context triplets, which provide the basis for later SoM construction and multimodal defect reasoning. Enhanced DroidBot Multi-window Policy
Parallel Scripts App State
GUI Process Log
GUI Multi-window Intent Input
Multi-window Controller
Parallel Execution
Fold Test
Split Test
App Info
Raw Screenshot
Drag Up
Drag Down
Raw UI Component Info
Fig. 7. Workflow of the enhanced DroidBot framework for triggering splitscreen and foldable interface states and collecting raw GUI data.
resource identifiers indicate background, divider, shadow, or similar visual-only functions. For container-type widgets, we retain them only when they are clickable or contain useful textual content. This design avoids passing large numbers of non-informative layout containers to the multimodal model. After filtering, the retained components are connected through a spatial adjacency graph. Two components are considered adjacent only when they satisfy overlap constraints and their edge distance is below a direction-specific threshold. Following the original implementation, we use different threshold settings for the horizontal and vertical directions to better fit the layout characteristics of mobile interfaces. Neighboring components are then sorted by distance, and a depth-first traversal is used to generate an ordered component sequence that approximately follows the natural reading order of the interface. This process produces a more compact and semantically meaningful representation for downstream SoM construction and multimodal reasoning. In our implementation, the horizontal adjacency threshold is configured with (20, 120, 0.04), while the vertical threshold is configured with (20, 150, 0.06), reflecting the different spatial characteristics of mobile GUI layouts. The DFS traversal starts from the top-left component after sorting controls by (𝑦 1 , 𝑥1 ), so that the resulting sequence better follows the natural reading order of the interface and preserves local functional groups. Algorithm 1 summarizes the complete extraction and filtering process. E. SoM Construction After collecting screenshot–UI-hierarchy–context triplets with the enhanced DroidBot framework, we construct a Setof-Mark (SoM) representation that aligns visual content with structured widget information. The goal of this step is to establish an explicit correspondence between interface regions and widget-level semantics, so that later multimodal reasoning can refer to specific interface elements more accurately. Based on the retained widgets obtained through the component extraction and filtering procedure described in the previous subsection, we overlay widget bounding boxes and widget identifiers on the screenshot to form the final SoM representation. Different widget types are rendered with different marker styles, while widget identifiers provide explicit anchors for subsequent multimodal reasoning and localization.
D. Component Extraction and Filtering To reduce noise in the raw UI hierarchy, we apply a component extraction and filtering procedure before constructing the final SoM representation. The procedure follows four core principles, namely size filtering, functional categorization, interactivity judgment, and content analysis. Its goal is to preserve components that are meaningful for GUI defect detection while minimizing the interference of redundant or irrelevant elements. In our implementation, extremely small components are removed first. We then exclude decorative elements whose
F. Multimodal Chain-of-Thought Prompting To improve defect diagnosis in complex GUI adaptation scenarios, we design a multimodal chain-of-thought prompting framework that guides the model to jointly reason over the marked screenshot and the structured widget information. Instead of directly predicting a defect label from the image alone, the model is instructed to analyze the interface in a stepby-step manner, so that the intermediate observations remain consistent with the visual evidence and the final decision becomes more interpretable.
Algorithm 1 Component Extraction and Filtering 1: Input: UI hierarchy JSON file 𝑗 𝑠𝑜𝑛 𝑓 𝑖𝑙𝑒 2: Output: Filtered component list 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠, adjacency
graph 𝑎𝑑 𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ 3: Parse 𝑗 𝑠𝑜𝑛 𝑓 𝑖𝑙𝑒 and initialize 𝑎𝑙𝑙 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 ← ∅ 4: for each element 𝑒 in the UI hierarchy do 5: Extract type, text, bounds, and attributes from 𝑒 6: Construct a control object 𝑐 7: Add 𝑐 to 𝑎𝑙𝑙 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 8: end for 9: Initialize 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 ← ∅ 10: for each control 𝑐 in 𝑎𝑙𝑙 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 do 11: if 𝑐.𝑤𝑖𝑑𝑡ℎ < 5 or 𝑐.ℎ𝑒𝑖𝑔ℎ𝑡 < 5 then 12: continue {remove tiny elements} 13: end if 14: if 𝑐.𝑟𝑒𝑠𝑜𝑢𝑟𝑐𝑒 𝑖𝑑 contains any of {scrim, background, divider,
shadow} then 15: continue {remove decorative elements} 16: end if 17: if 𝑐.𝑡𝑦 𝑝𝑒 is a container widget then 18: if not (𝑐.𝑐𝑙𝑖𝑐𝑘𝑎𝑏𝑙𝑒 or 𝑐 has valid text) then 19: continue {remove non-informative containers} 20: end if 21: end if 22: Add 𝑐 to 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 23: end for 24: Initialize 𝑎𝑑 𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ ← ∅ 25: for each pair of controls (𝑐 𝑖 , 𝑐 𝑗 ) in 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 do 26: Check horizontal overlap and vertical overlap 27: Compute horizontal edge distance 𝑑 ℎ and vertical edge dis-
tance 𝑑 𝑣 if (ℎ𝑜𝑟𝑖𝑧𝑜𝑛𝑡𝑎𝑙 𝑜𝑣𝑒𝑟𝑙𝑎 𝑝 ∧ 𝑑 ℎ ≤ 𝜏ℎ ) or (𝑣𝑒𝑟𝑡𝑖𝑐𝑎𝑙 𝑜𝑣𝑒𝑟𝑙𝑎 𝑝 ∧ 𝑑 𝑣 ≤ 𝜏𝑣 ) then 29: Compute Euclidean distance between the centers of 𝑐 𝑖 and 𝑐𝑗 30: Add an edge between 𝑐 𝑖 and 𝑐 𝑗 in 𝑎𝑑𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ 31: end if 32: end for 33: for each node 𝑣 in 𝑎𝑑𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ do 34: Sort neighbors of 𝑣 by distance in ascending order 35: end for 36: Sort 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠 by (𝑦 1 , 𝑥 1 ) from top to bottom and left to right 37: Select the top-left control as the DFS start node 38: Traverse 𝑎𝑑 𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ with DFS to obtain an ordered control sequence 39: Append isolated controls that remain unvisited 40: return 𝑣𝑎𝑙𝑖𝑑 𝑐𝑜𝑛𝑡𝑟𝑜𝑙𝑠, 𝑎𝑑𝑗 𝑎𝑐𝑒𝑛𝑐𝑦 𝑔𝑟𝑎 𝑝ℎ
(a) Original interface
(b) SoM-annotated interface
Fig. 8. Example of SoM construction. The left image shows the original interface, and the right image shows the corresponding SoM-annotated interface.
28:
The prompting framework takes as input a screenshot annotated with SoM markers together with structured widget information, including widget types, coordinates, textual content, and widget identifiers. This design allows the model to simultaneously access visual appearance and interface structure, which is particularly important for diagnosing layoutrelated defects that depend on both semantic meaning and spatial relations. The reasoning process is organized into multiple stages. The model first performs an overall inspection of the interface and identifies suspicious regions or abnormal layout patterns. It then conducts defect-specific analysis for the major defect categories considered in this work, including text overlap, widget occlusion, null display, text truncation, missing image, and navigation-bar/button overlap. In the final stage, the model
Fig. 9. A worked example of the multimodal chain-of-thought prompting framework. The model takes a SoM-annotated screenshot and structured widget information as input, performs step-by-step defect reasoning, and outputs the defect type, affected widgets, problem area, and explanation.
combines visual observations with widget coordinates, widget types, and SoM identifiers to verify the suspected issue and produce the final defect decision. This structured prompting strategy improves the consistency and interpretability of model predictions. Rather than producing only a defect label, the model is encouraged to output the defect type, the relevant widget or region, and a concise explanation of the diagnosis. Fig. 9 presents a worked example of the multimodal chain-of-thought prompting process, in which a text-overlap defect is identified through both visual and coordinate evidence.