Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering

Fusing UI Structure & Semantics for Feature-Oriented App Screen Retrieval & Clustering Arun Krishna Vajjala∗ , Yanfu Yan† , Ajay Krishna Vajjala∗ , Shrunal Pothagoni∗ , Denys Poshyvanyk‡ , Kevin Moran§ ∗ George Mason University, Fairfax, VA, USA, {akrishn, akrish, spothago}@gmu.edu † Zhejiang University, Hangzhou, China, yanfu@zju.edu.cn ‡ William & Mary, Williamsburg, VA, USA, dposhyvanyk@wm.edu

arXiv:2608.08880v1 [cs.SE] 9 Aug 2026

§ University of Central Florida, Orlando, FL, USA, kpmoran@ucf.edu

Abstract—User Interface (UI) programming is challenging due to the complex abstraction gap between code and graphical software representations. To bridge this gap, UI programming tools often rely on screen retrieval and clustering, which require accurate similarity measures based on overlapping features. However, computing feature-oriented similarity is difficult because screens with similar functionality often exhibit design variations. To address this, we propose F RAME (ReinForced UseR InterfAce Screen EMbedding with Graphical Structural ComprEhension), a multi-modal, neuro-symbolic embedding technique. F RAME constructs symbolic, graph-based representations of UI components to encode salient relationships and capture feature patterns across different screens. It leverages large vision-language models for visual and lexical encoding, alongside a novel UI-specific computational geometry algorithm that enables weighted embedding propagation. Across three benchmarks, F RAME outperforms strong baselines by up to 13% MRR in search and 7.6 percentage points in clustering accuracy. A comprehensive ablation study further confirms the benefit of each component, demonstrating F RAME’s potential for enhancing automated UI design and testing tools. Index Terms—Mobile apps, screen understanding, screen embedding

I. I NTRODUCTION The engineering process behind building, testing, and maintaining high-quality user interfaces (UIs) has been long documented to be uniquely challenging [1]. UI programming encompasses several obstacles related to difficulties reasoning about event-based programming, low-testability, limited language support, and programming tool complexity [1], [2], [3]. The key underlying challenge that leads to these obstacles is the need for programmers to navigate a complex abstraction gap, reasoning across three modalities of information: (i) natural language (documentation, requirements, etc.), (ii) code, and (iii) the pixel-based, graphical representation of the UI itself. This reasoning process can be particularly challenging as UI programming tends to follow paradigms which have been illustrated to be difficult to learn and understand, even for experienced programmers [4], [5], [6], [3]. To help overcome these challenges, the research community has invested in creating tools for automating UI design and programming tasks. Underlying many of these tools are the operations of screen retrieval and clustering, which, given † Corresponding author.

Three adjacent Text components illustrate a navigation paradigm, where a single tab is selected A route icon and two text fields indicate a functionality that allows the user to plan a trip A series of icons allow the user to configure different travel options for their trip.

Fig. 1. Example UI Component Relationships & Corresponding Features

a query screen retrieve semantically similar screens from a larger corpus or cluster semantically similar screens together, respectively. For example, recent programming tools related to UI testing rely on screen similarity to perform test transfer [7], enable context specific UI testing scenarios [8], uncover noncrashing bugs [9], and make UI testing more efficient [10]. UI programming tools related to bug report management use screen similarity to identify duplicate UI-based bug reports [11] and perform fault localization [12]. Finally, screen similarity is also important in UI design and implementation to assist in UI prototyping [13], [14] and evolution [15]. Given the increasing impact that models for screen retrieval and clustering are having on modern UI programming tools, their accuracy, generalizability, and robustness are critical. In both retrieval and clustering tasks, the core mechanic is rooted in computing screen similarity according to the semantics, or meaning, of the screens. In the context of UI programming tools screen semantics are generally defined by the features afforded or supported by the UI. As such in this paper, we define the phrase semantic screen similarity to refer to the extent to which a pair of UI screens share similar supported features. Capturing semantic screen similarity can be exceedingly challenging given the wide variations in UI designs that support similar features [16]. Existing methods of measuring screen similarity may cap-

ture some visual, lexical, or even structural context, but they generally fall short in representing the relationships between UI components—relationships that are essential to understanding screen features and functionality. For instance, grouped components like navigation menus provide a screen’s functional layout and meaning, which many current models do not adequately capture. Nuanced relationships between the elements on a given screen can provide important context on supported features in screen retrieval and clustering tasks. Figure 1 illustrates the importance of these relationships, wherein the components grouped in red signify navigation, the components grouped in blue signify a trip planning widget, and the components grouped in green signify trip options. In other words, there exist key limitations in current UI modeling techniques that hinder their effectiveness: (i) Lack of Representation of Screen Features: While general-purpose VLMs treat UI screens as monolithic pixel-grids [17], [18], [19], SE tasks require reasoning about functional modules, groups of components that work together to provide a feature. (ii) Expensive, proprietary, and domain-specific training procedures: most current UI models require expensive training on UI-specific datasets that are difficult to source and label, and quickly become outdated [18], [19]; and (iii) Lack of Shared UI Component Context: Current models are limited in their ability to model joint context across related UI components. To address the limitations of existing techniques for computational UI understanding of screen similarity, we propose F RAME: A ReinForced UseR InterfAce Screen EMbedding With Graphical Structural ComprEhension. Our key insight is that semantically coherent representations of UIs should follow a neuro-symbolic approach, leveraging the representational power of large vision-language models as a generalpurpose information backbone, while explicitly modeling UI features by capturing widget relationships using a symbolic, graph-based structure. F RAME embeddings function as an augmentation of the embeddings from large vision-language models (VLMs) (i.e., CLiP, BERT) and model UI relationships using computational geometry and a knowledge propagation algorithm, while automatically weighting the importance of UI component relationships – and hence features. Because F RAME is an augmentation or extension of existing VLM embeddings, it can benefit from the continued improvement in the representation power of these models, and does not require any further training of VLMs. Instead it functions by imposing strong symbolic priors using human knowledge of how features are constructed. In summary, F RAME overcomes limitations in prior work by transforming “flat” pixel-level data into a structured graph that reflects the developer’s conceptual model of UI functionality by augmenting VLM embeddings without any additional training expense. To illustrate the effectiveness of F RAME’s embedding technique, we apply it to the tasks of screen retrieval and clustering, using three benchmark datasets derived from past software engineering tools supporting UI testing [8], [7], and design [20]. When compared to both strong generalized and UI-specific model baselines, F RAME achieves up to 13%

higher MRR for screen retrieval and 7.6 percentage points higher accuracy in clustering, illustrating how our technique can improve the UI programming tools that our datasets originally supported. The main contributions of this paper are as follows: 1) A UI-aware, spatial knowledge propagation algorithm for encoding the relationship of UI components into a multimodal embedding; 2) A weighting strategy for UI-relationships based on the Vietoris-Rips complex from computational geometry; 3) F RAME – an implementation of the above strategies in conjunction with VLMs for calculating the feature-based similarity of UI screens; 4) The results of a comprehensive empirical evaluation of F RAME in the tasks of both screen retrieval and screen clustering across three SE-specific datasets; 5) An online appendix [21], [22] that contains F RAME’s code, our evaluation data, and our experimental infrastructure to foster replicability. II. C ONSTRUCTING AND M EASURING F EATURE -BASED S IMILARITY WITH F RAME E MBEDDINGS F RAME encodes semantic feature-based properties into UI embeddings which can benefit UI programming tools that rely upon screen similarity calculations. To achieve this, F RAME requires only the screenshot of a mobile app screen to generate the embedding, making it more easily extensible to a variety of contexts and UI programming tools. In this paper, we implement and evaluate F RAME for mobile app screens, given their popularity. However, the general workflow is extensible to a diverse set of UI screens. The key idea underlying F RAME is that semantic screen understanding requires a neuro-symbolic approach. That is, while generalized vectors of images and text derived from VLMs may be able to capture certain properties of UI screens, in order to measure a faithful feature-based screen similarity, we need to encode aspects of UI structure, through widget relationships, given its role in defining features. F RAME accomplishes this by encoding UI widgets as a graph, and using an embedding propagation technique in conjunction with a spatial weighting technique adapted from computational geometry to capture widget relationships. It is important to note that F RAME is a significant augmentation for VLM embeddings, requires no additional training whatsoever, and hence can improve over time as these base models improve. F RAME operates in five main stages to create a semantically accurate representation of the screen, illustrated in Figure 2. 1 First, in the Image Normalization Phase, F RAME performs preprocessing on a given input screenshot image, adjusting its colors and contrast to emphasize the structural boundaries of UI components. 2 Next, in the UI Component Graph Creation phase, F RAME uses a neural object detection technique to localize the boundaries of UI components from the screenshot to create UI component nodes, and then both visual and lexical embeddings for each node are computed, and spatial

Input Image

1 Image Normalization

2 UI Component Graph Creation

CLIP Image Embedding BERT Text Embedding

Neural Edge Detection Greyscale Conversion and Contrast Adjustment

CLIP

[0.9,0.5, 0.2,0.25 ….]

5 Post Processing PCA [0.9,0.5,0.2,0.25….]

Len: 1,792

116

Dimension Reduction [0.9,0.5,0.2,0.25….] Average of Modified Embeddings

Detected Component Bounds

4 Geometric Embedding Weighting

Multimodal UI Component Embeddings

FRAME UI Graph

3 Embedding Propagation

Weighted Embeddings [0.9,0.5,0.2,0.25….]

Output FRAME Embeddings

Normalized UI

Graph Node

Triangle Area As Embedding Weights

Rips Complex Weighting

Radius Based Connection

Propagated Embeddings

Propagated Embeddings

Embedding Propogation based on Spectral Fliter

Original Embeddings

Fig. 2. Overview of the F RAME Screen Embedding Workflow

connections between nodes are constructed. 3 In the Embedding Propagation phase F RAME shares information across connected UI graph components using an approximation of the spectral filter of the constructed UI graph. 4 In the Geometric Embedding Weighting phase, groups of components embeddings are weighted by calculating the VietorisRips complex [23], a technique adapted from computational geometry, using spatial UI centroids. This allows us to capture spatial relationships between components that signify screen features. 5 Finally, in the Post-Processing phase, F RAME uses Principal Component Analysis (PCA) to reduce the dimensionality of the full screen embeddings while preserving the most salient portions of the embedding. In the remainder of this section we describe each phase of F RAME. A. Image Normalization In order for F RAME’s embedding to learn UI relationships without being biased by nuanced stylistic details it utilizes two image preprocessing techniques on a given input screenshot. Screens that offer near functional equivalence can differ widely across stylistic properties, including color palettes, customized icon designs, and tailored fonts. To mitigate the effects of these variations and learn transferable semantic patterns, F RAME converts screens to greyscale and enhances screenshot contrast. This has the additional benefit of making image-based UI component localization in Phase- 2 of F RAME’s workflow more effective. 1) Greyscale Conversion: F RAME’s goal is to provide similarity rooted in supported screen features, rather than stylistic UI patterns. We illustrate how open-domain vision-language models tend to perceive UI screens in Fig. 3, which depicts three images from one of our evaluation datasets that were the most similar when using a CLIP embedding [24]. It is clear that the three screenshots have very similar color palettes, yet they do not share the same structural components and represent three varying sets of functionality (or lack thereof). The leftmost screen is a Splash screen, the middle is a Search screen, and the rightmost screen is a password screen. Visual intuition suggests that these three screens provide different

Fig. 3. Three Images with Similar CLIP Embeddings

functions, however, because color plays a dominant role in VLM embeddings, such as those from CLIP, they are rated as similar. To reduce the seemingly large impact that color plays in VLMs we convert screens to greyscale, which preserve gradients that capture emphasis. F RAME uses the Pillow library’s [25] greyscale conversion to accomplish this. 2) Contrast Adjustment: In addition to converting the input screenshot to greyscale, F RAME alters the images by increasing the contrast by 2× in the image. Contrast allows F RAME to darken the dark pixels in the screen and brighten the brighter pixels in the screen. The resulting image has two main characteristics: (i) enhanced edge visibility; (ii) reduced noise influence – both of which aid in visual UI component detection. We use the Pillow library’s [25] “ImageEnhance” functionality to double the contrast. An example is shown in Figure 2- 1 . B. UI Component Graph Creation To encode the visual structure of a UI screen into a symbolic representation, F RAME builds a graph by plotting nodes according to the absolute spatial coordinates of detected components on the screen. First, we adapt computer vision and deep learning techniques to identify UI components. Second, visual and lexical embeddings are created for each

individual UI component, and edges are established via spatial relationships between components. 1) UI Component Detection: F RAME adapts prior work that uses a combination of classical and neural computer vision techniques to detect UI components on a given screen [26]. We adapt these techniques to extract the bounding boxes of each UI component on a target screen which F RAME uses to crop out an image of each component from the larger screenshot. F RAME only considers components larger than 48 × 48px since that is the standard for minimum accessible interactive icons on an Android screen as per Google’s Accessibility Guidelines [27] – this effectively allows us to filter out smaller elements that might dilute the learned UI component relationships. The centroid of each bounding box is used to signify each component’s absolute location on the screen. 2) UI Graph Creation: Given that both the visual design (e.g., a search icon) and lexical information (e.g., text displayed by a component) contribute to UI meaning, F RAME encodes both types of information for each extracted UI component. To do this, F RAME uses the cropped icon provided from the UI Component Detection procedure and creates two embeddings from it, an image embedding using a large visionlanguage model, and a text embedding using a large language model (LLM). In our implementation of F RAME we elected to use CLIP [24] embeddings to encode visual componentlevel information and BERT [28] to encode lexical information extracted via Optical Character Recognition (OCR) from individual component nodes in the UI graph. Given the number of embeddings that must be calculated for a given screen, a balance of inference efficiency, and the representative power of the model is our key concern. As such, we chose to use the more efficient BERT and CLIP models for text and image embeddings respectively since the token sequences present on UI components are typically short, CLIP has been shown adept at component modeling in the past [12], and the increased representational power of larger GPT-like models does not outweigh the efficiency of “smaller” models when embedding text/visuals from large number of UI components. Image Embedding: As shown in Fig. 2, each node has a visual representation of the UI component that is cropped out of the screen. We pass each image to the Huggingface’s opensource implementation of the CLIP model [29], saving the resulting embedding vector. Text Embedding: We use PyTesseract [30] to extract text from each cropped UI component. We then pre-process the extracted text to remove punctuation and non-text/numeric characters. F RAME uses this “cleaned” text to create the BERT embedding for each node. With the inclusion of the image and text embeddings within each UI component, the nodes in the graph contain a multimodal data representation that is localized to each component. This gives each node a unique set of features that sets it apart from other nodes within the graph and its neighbors. Graph Edges: Once nodes for the graph have been created, F RAME constructs a graph by connecting nodes that are a maximum of 300 pixels in distance from each other according

to the Manhattan distance measure. This number of 300 pixels was arrived at through experimentation on a held-out development set of screens from the RICO [16] dataset, where a systematic grid search of numbers were trialed and the resulting graphs visually inspected by two authors. Through this preliminary study, the two authors concluded that, for the largest majority of screen types, 300px was the most suitable for constructing semantically meaningful graph edges for mobile apps screens with a resolution of 1080 × 1920. For other screen sizes, using a distance roughly equivalent to 1/3 the screen width is likely suitable given our experimental observations. This approach allows for the formation of graph neighborhoods that represent meaningful screen features. C. UI Graph Embedding Propagation Unlike ordinary images, UI screens exhibit distinct component structures which encode information about the types of features afforded by the interface, with more pronounced relationships existing between spatially adjacent components. Therefore, we augment the image/text embedding of each component by its neighbor component embeddings based on the constructed UI graph edges from the prior phase. Specifically, our embedding propagation strategy is derived from the first-order approximation of localized spectral filters on graphs [31], [32], which can be represented as follows: 1

1

S ′ = (IN + wD− 2 AD− 2 )S.

(1)

S ∈ RN ×M represents the matrix of all the image/text embeddings of nodes from G, and S ′ represents the updated matrix by incorporating the information from the neighbor nodes. M denotes the dimension of each image/text embedding (e.g., 768) and N denotes the number of nodes in the graph. A is the adjacency matrix of G without self-connections and D is the degree matrix of A so that the adjacency matrix is normalized by D with respect to both the row and the column. w is a constant to balance the information from the original node/component with structural information from the neighbor nodes/components. By leveraging the embedding propagation strategy, we create shared embeddings wherein features from adjacent nodes are mixed, which serves as means to capture the relationships between UI components that signify features. A visual representation of this process is shown in Fig. 2- 3 . D. Geometric Embedding Weighting For a given UI screen, not all spatial relationships among components are created equal. For example, encoding the three menu bar headers in the travel app presented at the outset of this paper (Figure 1) would be more important than encoding a relationship between one text title and the time picker widget. However, weighting the graph using spatial information is a complex task, which has been studied in topology in the field of mathematics. Simple weighting schemes using direct neighbors may not result in the creation of optimal topological weighting schemes that capture UI components indicating screen features. As such, to properly capture information regarding the topological relationship of various UI components and weight component relationships

Fig. 4. Visualization of Low-Dimensional Simplices

appropriately, F RAME develops a new technique that applies concepts from computational geometry to UI data. The end result of this process is that components that would more typically signify UI features are grouped together, and those that do not remain separate. To do this, F RAME represents the propagated UI embeddings as points in Euclidean space by triangulating the points to create a structure known as a simplicial complex (Fig. 4). These plotted embeddings are used to create an object known as the Vietoris-Rips complex (often called the Rips complex) using GUDHI [23]. Given a parameter ϵ > 0, the Rips complex of embeddings X is made into a simplicial complex that captures the spatial and topological relationships between the points in space using the following definition: Definition 1. Given a set of embeddings X = {x1 , x2 , . . . , xk } ⊂ Rn and an ϵ > 0, a k-simplex σ = [xi1 , xi2 , . . . xik ] is in the Vietoris-Rips Complex Rips ϵ (X ) if and only if: Bϵ (xij ) ∩ Bϵ (xij′ ) ̸= ∅ where Bϵ (xij ) is an open ball of radius ϵ centered at xij . ϵ > 0 is the radius of the open balls used to determine edge connections within the embedding space. For small values of ϵ, the balls are too small to intersect and result in a complex with only the original embeddings. On the opposite end, if ϵ is too large, every feature in the input image is related to every other feature. We selected ϵ by performing a parameter study from 0.1 to 1 in increments of 0.05. Our ideal value of ϵ captures the average number of features (i.e., UI components) on a given screen, which we found typically ranges from 20100. The average number of triangles is determined using the Area-based Triangulated Embedding (ATE) method proposed by Krishna Vajjala et al. [33]. Through a small scale empirical study with two authors on a held out portion of our data, we found that this method of deriving triangle numbers struck a reasonable balance between sparsity and completeness that captured important feature-oriented UI component groups, minimizing less important connections between components. Through this process, we determined ϵ = 0.5 was most suitable, and produced an average of 100 2-simplices in the Rips complex. An illustration of the overlapping balls and the resulting simplices is shown in Fig. 5. By employing the Rips complex on the propagated UI embeddings F RAME topologically captures the relationship between an object in an image with all other objects in the image as shown in Fig. 5. Once the simplex is created, we consider the embeddings that make 2-simplices (2D triangles) and use the area of the resulting simplex as a weight for the

Fig. 5. Process of Creating the Rips Complex Using a Set of Embeddings

involved embeddings. The area of a given 2-simplex can be calculated using Heron’s formula [34]: p Area = S(S − A)(S − B)(S − C) (2) where S is the semi-perimeter of the 2-simplex and A, B, and C are the lengths of its three edges. The area is then used to weigh the involved embeddings. Unlike simple pairwise distances (1-simplices) which often capture noisy, unrelated nearby elements, the Vietoris-Rips complex identifies cohesive neighborhoods that accurately represent distinct UI features. We use the 2-simplex (triangle) area for weighting because components consuming more screen real-estate are typically perceived as more “functionally dominant” and semantically important within the interface’s layout. F RAME then employs the Area-based Triangulated Embedding method (ATE), proposed by Krishna Vajjala et al. [33], for the weighted average of the embeddings. This creates a weighted relation between embeddings that are spatially close to each other, resulting in a weighted preservation of information between those three embeddings. These embeddings are then concatenated together to create an embedding for an entire screen, as described in the next subsection. This process is shown in Fig. 2- 4 .

E. Embedding Post-Processing The process of creating a distance-similar graph, propagating the embeddings within it, and consolidating them using the area-based triangulation creates a semantically enhanced set of embeddings that encode the relationships of UI components which represent screen features. To create its final screen embedding, F RAME combines the graph-based semantically enhanced embeddings with a CLIP embedding of the original greyscale and contrast enhanced input image, in order to capture general global image features, resulting in a 1,792dimension embedding. F RAME then further processes these embeddings using Principal Component Analysis (PCA) [35] to reduce potential noise in the resulting larger embedding. F RAME uses Scikit-Learn’s implementation of PCA applied to the training splits of our three experimental datasets, and the resulting F RAME embeddings were reduced to a dimensionality of 116. To ensure that arbitrary future embeddings result in size 116 without re-computing the PCA, F RAME uses PCA’s transformation matrix and performs a dot product with the derived 1,792 dimensional vector for future images.

III. D ESIGN OF E MPIRICAL E VALUATION In this section, we describe the procedure we used to evaluate F RAME. To achieve our study goals, we formulated the following four research questions (RQs): • RQ1 How does F RAME perform against baselines in screen retrieval tasks? • RQ2 How does F RAME perform against baselines in screen clustering tasks? • RQ3 How do F RAME ’s embedding propagation, weighting, and image processing contribute to its effectiveness? • RQ4 Why does F RAME outperform baselines in screen retrieval? A. Evaluation Datasets We focus F RAME’s evaluation on screen retrieval and clustering as these tasks have been shown to be important in software engineering tools/tasks [14], [36], [12], [37], [11]. We evaluate F RAME with three different datasets derived from prior work that use semantic screen similarity to aid in UI testing (e.g., the Avgust [7] and Aurora [8] datasets) and UI design (e.g., the Enrico [20] dataset). 1) Aurora Dataset: The Aurora dataset, created by Khan et al. [8], was derived to support a UI testing tool that classified UI screens into various design patterns so that it can intelligently test the identified screens. This dataset was derived by sampling 1% of the images from the larger RICO [16] dataset, and clustering them to bootstrap a labeling process. The authors of the paper then manually grouped 1,370 UI screens into 21 semantic categories, using a rigorous, independent labeling procedure with high agreement [8]. For our retrieval task, we perform a stratified sampling of 10% across categories to serve as screen queries, whereas the other 90% of the dataset represents the corpus. 2) Avgust Dataset: The Avgust dataset was introduced by Zhao et al. [7], also in support of automated cross-app mobile testing. It has 25 labels corresponding to the purpose of a given screen (e.g., shopping cart) across 2,475 total screens. This dataset was manually labeled by four authors who mutually agreed on each screen’s category. This is a larger dataset with approximately 99 screens per category on average. We again use the same 90/10 split for corpus/queries. 3) Enrico Dataset: The Enrico dataset is designed for topic classification tasks for UI screens to assist in UI Design. It consists of 1,460 unique UI screens labeled by their design categories (e.g., Profile screen) by two authors. The screens span 20 labels, creating an uneven distribution of class support with the largest class having 265 screens and the smallest class having only nine screens. We use an 85/15 corpus/query split for this dataset given its smaller size. The Enrico dataset is perhaps the most challenging to classify or cluster due to its small support across screen categories. These datasets were selected for their diversity and relevance to UI programming tools, such as automated UI test generation and UI design. Though originally developed for distinct end-to-end software engineering tasks, these datasets capture diverse functionalities and nuanced categorizations

of UI screens. The carefully curated labels, established by multiple authors in their respective studies, offer a reliable benchmark to assess F RAME’s effectiveness in identifying specific screens for clustering and retrieval tasks. Notably, none of the images in Avgust, Aurora, and Enrico overlap, although some label categories recur due to the limited, finite existence of various app screen categories. This selection represents an extensive and diverse set of Android screens, supporting F RAME’s generalizability. B. Baseline Embedding Techniques We compare F RAME to four key baselines: CLIP, BLIP, Screen2Vec, and BERT. We use these baselines to measure F RAME’s efficacy compared to popular tools. We were not able to use more specialized baseline UI models such as ScreenAI [38], VUT [18], UIBert [19], or Graph4GUI [39], as these techniques are not publicly available. However, it should be noted that F RAME is largely complementary to these techniques. Additionally, this study aimed to provide a reproducible, open-source evaluation, which influenced the selection of the included state-of-the-art embedding techniques. 1) CLIP: CLIP [24] embeddings represent images and corresponding image keywords in a shared embedding space, with a contrastive learning mechanism that aims to minimize the distance between corresponding image-description pairs, enabling cross-modal understanding. 2) BLIP: Salesforce’s BLIP embeddings [40] extend the concept behind the multimodal shared embedding space of CLIP by using slightly different encoders and a feature that generates and aligns image captions for pre-training. 3) Screen2Vec: Li et al.’s Screen2Vec [17] is designed as an embedding specifically for screens that captures and embeds information using runtime UI metadata and text. 4) BERT: Despite being primarily designed for text processing, BERT [28] can be re-purposed to encode the text displayed on UI screens. We selected CLIP, BLIP, Screen2Vec, and BERT as our baseline models because each offers a different approach to embedding creation relevant to UI understanding: CLIP and BLIP provide multi-modal vision-language embeddings, Screen2Vec focuses on UI-specific text and hierarchy, and BERT captures textual information. We excluded larger LLMs such as GPT-4/5 from our retrieval/clustering tasks for three reasons: (i) it is difficult to represent screen-retrieval tasks via prompting techniques due to limitations with the context windows of LLMs, and most commercial frontier models do not offer APIs for generating image embeddings; (ii) image or text-based LLM-based embeddings tend to perform similarly or worse to vision-language models such as CLIP in general UI understanding tasks [41], with LLMs performing better at processing UI Code [42]; (iii) computing large numbers of embeddings for practical applications of semantic screen search using LLMs is likely to be costly. C. RQ1 - Screen Retrieval Methodology To evaluate F RAME for the task of screen retrieval, we formulate a query with each UI screen in a held out query set

TABLE I F RAME P RE -P ROCESSING A BLATION S TUDY C ONFIGURATIONS Acronym Greyscale Contrast NG-C ✗ ✓ NG-NC ✗ ✗ G-NC ✓ ✗

for each dataset, and calculate the cosine similarity, using a given embedding technique, against each screen in the corpus set, forming a ranked list of retrieved screens. We then adopt the widely used MRR and HR@k (Hit Rate) metrics [43], [12] to evaluate the effectiveness of F RAME compared to our baseline techniques. HR@k measures the percentage of queries for which an approach retrieves at least one correct screen within the top-k screens. Here a correct screen is one that shares the same label as the query. MRR calculates the average of the reciprocals of ranks at which the relevant items are retrieved. Higher numbers indicate better performance across metrics. D. RQ2 - Screen Clustering Methodology To answer this RQ we generate F RAME and CLIP embeddings across the three datasets and evaluate their clustering performance (using K-means clustering, with k set to the number of labels within each dataset) across nine common clustering metrics: Sum of Squared Errors (SSE), Clustering Accuracy, Fowlkes-Mallows Index (FMI), Homogeneity, Completeness, V-Measure, Davies-Bouldin Index (DBI), Adjusted Rand Index (ARI), and Normalized Mutual Information (NMI). This experiment allows us to examine how well F RAME is able to segment groups of screens using its enhanced embeddings compared to CLIP, which was the best performing baseline in the screen retrieval task. E. RQ3 - Ablation Study Methodology The goal of this experiment is to determine whether F RAME’s various features and pre-processing techniques contribute to its effectiveness in screen retrieval. We divide the ablation study into two parts: evaluating pre-processing configurations and analyzing F RAME’s specific embedding components. We utilize only the Avgust and Aurora datasets for this evaluation, excluding Enrico due to its inherent class imbalance and small sample size. The first part of the study examines F RAME’s greyscale and contrast image pre-processing in conjunction with all model components. As shown in Table I, we evaluate three configurations: (i) NG-C, representing contrast without greyscale; (ii) NG-NC, which includes neither augmentation; and (iii) G-NC, representing greyscale without contrast. The second phase evaluates embedding variants using a three-tuple notation, as detailed in Table II. For example, in the tuple C-PB-PC, the first grouping indicates the use of the full-screen CLIP embedding (C or NC); the second grouping denotes the use of propagated BERT-based component text embeddings (PB or NPB); and the third grouping determines whether propagated and weighted CLIP embeddings are utilized (PC or NPC) with N indicating not present.

TABLE II F RAME A BLATION S TUDY C ONFIGURATIONS Acronym Whole Screen CLIP Node BERT Node CLIP NC-PB-PC ✗ ✓ ✓ C-PB-NPC ✓ ✓ ✗ C-NPB-PC ✓ ✗ ✓ C-NPB-NPC ✓ ✗ ✗ NC-PB-NPC ✗ ✓ ✗ NC-NPB-PC ✗ ✗ ✓ TABLE III R ETRIEVAL RESULTS OVER 3 DATASETS , WHERE THE BEST RESULTS ARE IN BOLD , AND * INDICATES A p- VALUE LESS THAN 0.05 FROM A TWO - TAILED PAIRED T- TEST BETWEEN F RAME AND THE BEST BASELINE . Dataset

Aurora

Avgust

Enrico

Metrics

Screen2Vec

BERT

BLIP

CLIP

FRAME

p-value

HR@1 HR@5 HR@10 MRR HR@1 HR@5 HR@10 MRR HR@1 HR@5 HR@10 MRR

0.3750 0.3027 0.2604 0.4853 0.3922 0.3185 0.2739 0.5116 0.2895 0.2389 0.2063 0.4271

0.3194 0.2486 0.2187 0.4363 0.8270 0.7788 0.7141 0.8908 0.2212 0.2019 0.1822 0.3849

0.4097 0.3944 0.3639 0.5479 0.8509 0.7937 0.7522 0.8861 0.2837 0.2712 0.2577 0.4534

0.4722 0.4013 0.3590 0.5963 0.8549 0.7788 0.7364 0.8913 0.3365 0.3048 0.2740 0.5051

0.5625* 0.4652* 0.4166* 0.6729* 0.8823* 0.8329* 0.7827* 0.9184* 0.3894* 0.3096 0.2903 0.5391

8.56 × 10−15 6.02 × 10−22 3.38 × 10−26 5.09 × 10−14 2.90 × 10−10 7.41 × 10−23 0.00150 9.42 × 10−10 0.0176 0.0954 0.0687 0.0586

F. RQ4 - Qualitative Study on Screen Retrieval To evaluate the performance of F RAME in comparison to CLIP embeddings, we conduct a qualitative analysis using the Avgust dataset. We randomly select a test image and retrieve the three most similar images based on the embeddings from each model. By visually inspecting these top matches, we assess the ability of each embedding to capture and represent the visual and structural characteristics of UI screens accurately. IV. E MPIRICAL R ESULTS A. RQ1 : F RAME Screen Retrieval Effectiveness F RAME performs significantly better in screen retrieval tasks compared to the state-of-the-art baseline methods, as shown in Table III. Specifically, F RAME outperforms Screen2Vec, BERT, BLIP, and CLIP in every metric across all three datasets. F RAME significantly outperforms baselines across three diverse datasets, exhibiting an average improvement of 16.0% on the sparse Aurora dataset and 4.9% on the Avgust dataset. Despite the increased difficulty of the Enrico dataset characterized by class imbalance, F RAME maintains superior performance with minor increases in hit-rate and MRR. Specifically, compared to the strongest baseline, CLIP, F RAME achieved a 12.8% MRR improvement on Aurora and a 3.0% improvement on Avgust, with most results across all three benchmarks reaching statistical significance. Collectively, these results over highly varied datasets demonstrate F RAME’s effectiveness in accurately ranking similar UI screens toward the top of retrieval lists. In order to better understand the advantages in performance that F RAME offers, we manually examined both the metrics and the lists of retrieved screens for the HR@10 metric between F RAME and CLIP, the best performing baseline, across individual screen categories across the datasets. We found that F RAME consistently outperformed CLIP across all

CLIP Embeddings Clustering

TABLE IV C OMPARISON OF F RAME AND CLIP U SING C LUSTERING M ETRICS ACROSS T HREE DATASETS

Enrico Data SSE Clustering Accuracy FMI 9832.0852 0.0074 0.1372 68604.9113 0.0543 0.2314 Homogeneity Completeness V-Measure CLIP 0.0083 0.0081 0.0082 FRAME 0.1364 0.1362 0.1363 DBI ARI NMI CLIP 6.8759 -0.0025 0.0082 FRAME 2.7673 0.1012 0.1363 Avgust Data Embedding SSE Clustering Accuracy FMI CLIP 17044.7382 0.0072 0.1394 FRAME 54236.0280 0.1669 0.4170 Homogeneity Completeness V-Measure CLIP 0.0112 0.0107 0.0109 FRAME 0.4742 0.4668 0.4705 DBI ARI NMI CLIP 7.2990 0.0032 0.0109 FRAME 2.1801 0.3183 0.4705 Aurora Data Embedding SSE Clustering Accuracy FMI CLIP 86165.4671 0.0017 0.1403 FRAME 47169.3284 0.0217 0.2799 Homogeneity Completeness V-Measure CLIP 0.0192 0.0208 0.0199 FRAME 0.2530 0.2702 0.2613 DBI ARI NMI CLIP 11.5100 -0.0015 0.0199 FRAME 3.5316 0.1637 0.2613 Embedding CLIP FRAME

screen types except for four of Avgust’s twenty-five different screen types: SignIn, Menu, SignUp, and Account screens. We speculate that F RAME’s lower accuracy on these screens stems from their inherently high variability, reliance on textual semantics, and flexible layouts, which favor CLIP’s multi-modal understanding. In particular, SignIn and SignUp screens often share similar structures and input fields, making it challenging for F RAME to distinguish them based on functionality alone. B. RQ2 : F RAME Screen Clustering Effectiveness Our results suggest that F RAME embeddings outperform CLIP embeddings across almost all metrics, except for SSE, in the Enrico and Avgust datasets, as seen in Table IV. Specifically, F RAME achieves a Clustering Accuracy of 0.0543 on the Enrico dataset and 0.1669 on the Avgust dataset, compared to CLIP’s 0.0074 and 0.0072, respectively. This shows that F RAME performs better at correctly grouping similar screens. F RAME outperforms CLIP across both internal and external clustering metrics, achieving significantly higher FMI (0.4170 vs. 0.1394), V-Measure (0.4705 vs. 0.0109), and ARI (0.3183 vs. 0.0032) on the Avgust dataset. These results, along with high homogeneity and completeness scores, present F RAME’s ability to produce clusters aligned with true functional structures. While CLIP achieves lower SSE—indicating tighter clusters—this tightness sacrifices accuracy through over-generalization. In contrast, F RAME’s lower DBI values (e.g., 2.1801 vs. 7.2990 for Avgust) confirm better cluster separation and a refined ability to distinguish unique UI features.

Clusters home popup sign_in item_detail items menu get_started sign_up

1.5

1.0

0.5

0.0

0.5

1.0

1.5

2.0

1.5

1.0

0.5

0.0

0.5

1.0

1.5

FRAME Embedding Clustering – Avgust Dataset Fig. 6. CLIP Embedding Clustering

Clusters home popup sign_in item_detail items menu get_started sign_up

10

5

0

5

10

10

5

0

5

10

Fig. 7. F RAME Embedding Clustering – Avgust Dataset

Visualizations in Figs. 6 and 7 show that CLIP creates tighter, more homogeneous clusters that sacrifice accuracy due to a lack of structural granularity. In contrast, F RAME effectively identifies and groups functionally similar screens despite minor outliers. While 2D projections limit feature detail, the quantitative data in Table IV supports these visual insights, confirming F RAME’s superior ability to differentiate UI features. C. RQ3 : F RAME Component Ablation Study 1) Pre-Processing Ablation Study: We first present the results for the pre-processing configurations in Table VI. The NG-C (contrast only) variant is the least effective configuration, ranking last on all eight dataset–metric combinations. It is also outperformed by NG-NC (no augmentation) on every metric, indicating that raising contrast on a full-color screenshot actively hinders component detection rather than aiding it. The G-NC (greyscale only) variant is the strongest single-augmentation configuration, and the best alternative to the full model on Aurora, where it leads NG-NC on three of four metrics. On Avgust the two are comparable: G-NC ties on HR@1 and leads on MRR, but trails slightly on HR@5 and HR@10. This suggests that removing color bias is most beneficial on visually diverse corpora such as Aurora. Overall, greyscale conversion acts as a prerequisite for contrast

TABLE V C OMPONENT ABLATION OVER 2 DATASETS BETWEEN F RAME AND ITS VARIANTS , WHERE THE BEST RESULTS ARE IN BOLD .

Dataset

Aurora

Avgust

Metrics

NC-NPB-PC

NC-PB-NPC

C-NPB-NPC

NC-PB-PC

C-PB-NPC

C-NPB-PC

FRAME

HR@1 HR@5 HR@10 MRR HR@1 HR@5 HR@10 MRR

0.2777 0.1944 0.1631 0.3880 0.7960 0.6901 0.5929 0.8380

0.3750 0.2972 0.2791 0.5026 0.8352 0.7850 0.7360 0.8724

0.4722 0.4013 0.3590 0.5963 0.8549 0.7788 0.7364 0.8913

0.4166 0.2750 0.2458 0.5136 0.8666 0.8023 0.7498 0.9012

0.5138 0.4486 0.4180 0.6448 0.8588 0.8282 0.7721 0.9014

0.4513 0.4180 0.3618 0.6077 0.8705 0.8219 0.7674 0.9114

0.5625 0.4652 0.4166 0.6729 0.8823 0.8329 0.7827 0.9184

TABLE VI P RE - PROCESSING ABLATION OVER 2 DATASETS BETWEEN F RAME AND ITS GREYSCALE / CONTRAST VARIANTS , WHERE THE BEST RESULTS ARE IN BOLD . Dataset

Aurora

Avgust

Metrics

NG-C

NG-NC

G-NC

FRAME

HR@1 HR@5 HR@10 MRR HR@1 HR@5 HR@10 MRR

0.4930 0.4069 0.3638 0.6171 0.8505 0.8119 0.7658 0.9094

0.5347 0.4250 0.3805 0.6484 0.8784 0.8298 0.7827 0.9118

0.5208 0.4527 0.4138 0.6492 0.8784 0.8266 0.7799 0.9138

0.5625 0.4652 0.4166 0.6729 0.8823 0.8329 0.7827 0.9184

enhancement to be useful – contrast alone is detrimental, whereas combining the two yields the best or tied-best result on all eight measurements. 2) Embedding Ablation Study: Table V details the performance of F RAME’s various component configurations. The full F RAME variant, utilizing all embeddings, outperforms all others except for HR@10 on the Aurora dataset. Variants lacking the global screen-level CLIP embedding (NC-NPB-PC, NC-PB-NPC, and NC-PB-PC) show poor performance on the Aurora dataset, as they rely solely on localized component propagation without broader screen context. While incorporating screen-level context (C-NPB-PC and C-PB-NPC) improves results, these configurations still underperform relative to the full model. These results confirm that embedding propagation is essential for capturing cohesive UI screen semantics.

Fig. 8. Comparison of the Top-3 Most Similar Screens for Each Embedding Type

query, and tend to capture general patterns, instead of features indicated by UI component relationships. V. D ISCUSSION

D. RQ4 : Qualitative Analysis

A. Analysis of Retrieval Failures

In this section we illustrate the qualitative benefits that F RAME provides in regards to screen structure. Fig. 8 offers insight into how various models identify similar screens. The screen on the left is the test image and the three images on the right are the three most similar screens to the test image using the F RAME and CLIP embeddings. Similarity is computed using cosine similarity. The evaluation of modal screens shown in Fig. 8 using the embeddings further reinforces the strengths of F RAME in identifying similar images based on component relationships. F RAME’s selected images for modal screens show strong functional similarity, including consistent modal placement and content within the modal. This highlights F RAME’s capability to recognize and prioritize the overall feature set of modal screens, making it suitable for tasks requiring transferable understanding of UI screens across UI design variations. In comparison, CLIP’s retrieved screens are far less related to the

While F RAME consistently outperformed CLIP, we observed lower accuracy on four specific screen types: SignIn, Menu, SignUp, and Account. We attribute this to the inherent structural similarity between SignIn and SignUp screens, which often share near-identical input field layouts. In these cases, the model relies heavily on textual semantics, where CLIP’s multimodal pre-training offers a slight advantage over F RAME’s structural priors. B. The “Tightness vs. Accuracy” Trade-off A key finding in our clustering evaluation is the discrepancy between SSE and other performance metrics. While CLIP generally achieves lower SSE (indicating “tighter” clusters), this tightness is often misleading. Visualizations in Figs. 6 and 7 reveal that CLIP-based embeddings force diverse screens into a small, overlapping area of the embedding space. This overgeneralization sacrifices the granularity needed to distinguish

functional features. In other words, while clusters are tighter, they occupy the same semantic embedding space, making it difficult to discern discrete types of screens. In some cases, this could also lead to a deceptively high accuracy, where a majority of screens fit within the area of overlap of these small clusters. However, this is not a good representation of the discriminatory power of the baseline model, as indicated by our other measured metrics. In contrast, F RAME’s lower DBI values (2.1801 vs. CLIP’s 7.2990) indicate better-separated clusters that align more closely with true structural features. C. Neuro-Symbolic vs. Language/Vision Model-Based Representations The comparison with BERT/CLIP suggests that pure LMbased or VLM-based textual descriptions lack the structural precision captured by F RAME’s approach. While LMs can be adept at high-level categorization, they often omit the spatial relationships and component neighborhoods that define UI features. F RAME ’s use of graph propagation and geometric weighting allows it to maintain structural fidelity that generalized descriptions cannot match. VI. R ELATED W ORK & N OVELTY S TATEMENT In this section, we discuss existing multimodal UI learning techniques, and discuss their limitations in context, differentiating and explicitly illustrating F RAME’s novelty. A. UI Learning Models Baechler et al. [38] proposed ScreenAI, a Vision-Language Model for UI comprehension. ScreenAI integrates vision and language processing to understand and interact with UIs and infographics. ScreenAI makes use of a text-based representation of screens and a novel UI component labeling task to excel in screen question answering. However, this model does not directly encode UI relationships and is not immediately applicable to the retrieval and clustering tasks F RAME targets. Li et al. [18] proposed VUT, a UI transformer for multimodal, multi-task user interface modeling. This approach considers the hierarchical structure, image, and language properties of the screen to create their screen representations. Bai et al. [19] proposed UIBert, a transformer-based joint image-text model that learns generic feature representations for a UI and its components. This embedding is similar to VUT in its goal to create an embedding using image, text, and structural screen properties. Screen2Vec [17], UIBert [19], and VUT [18] do not directly capture the spatial and functional relationships between UI components, often reducing structural information to flat representations. This limits their effectiveness in tasks such as duplicate detection, clustering, and app categorization, where understanding component interdependence is crucial. The closest related work to our own was introduced by Jiang et al. [39] in Graph4GUI, a graph neural network-based representation of UIs. This graph comprises element nodes, representing properties like appearance and size, and constraint nodes, representing layout constraints such as alignment and grouping. However, this model is tailored for UI generation,

and represents the structural properties of UI components, as opposed to attempting to capture feature-oriented component relationships. It is important to note that the only publicly available model discussed is Screen2Vec [17], and further each model tends to target different UI tasks, making them difficult to apply to screen↔screen similarity calculations. B. UI Search and Similarity Guigle, proposed by Bernal-Cárdenas et al. [36], is a UI search engine. It uses metadata information to facilitate a filterbased search, rather than an embedding-based search. Behrang et al. introduced GuiFetch [14], which computes similarity scores at the app level with UI sketches using keywords and GUI hierarchy properties. GeminiScope, proposed by Mao et al. [44], is a GUI similarity metric that uses the leaf nodes in the hierarchy tree to determine the position of the elements on the screen. They compute the absolute values of UI-related features, such as position and size, and use these metrics as their numerical representation for the screen. However, GeminiScope does not consider relations between the embeddings. C. UI Comprehension To modify or test a UI, it is necessary to have a semantic understanding of the UI. Many tools use the Android screen hierarchy [19], [18], [17], [45], however, the hierarchy is not always available. Wu et al. introduce an approach called Screen Parsing [46] to reverse engineer the screen hierarchy using computer vision techniques. Spotlight, introduced by Li et al. [47], uses computer vision to create a vision-language model architecture that can assist in screen comprehension tasks. F RAME differs from both of these techniques in both its goal and its UI representation. Novelty Statement: In summary, F RAME differs from the techniques discussed in the prior three subsections in a number of key ways. First, it is the first feature-oriented approach for calculating UI similarity, combining both generalized component representations using VLMs, and a symbolic representation of UI structure in a graph form for encoding features. Second, F RAME is the first graph-based technique to leverage embedding propagation and weighting techniques that capture explicit UI component relationships representing features. While some other techniques (e.g., Graph4GUI) encode hierarchy information into a graph, this information is largely structural and not captured to represent features. Third, F RAME is an enhancement for existing large vision-language models that does not require additional training, while taking advantage of the continuing rapid advancements of such models. Fourth, our current implementation of F RAME does not require additional model training, making use of strong UI priors to effectively capture UI information. Together, these properties constitute F RAME’s novelty. VII. C ONCLUSIONS In this paper, we presented F RAME, a neuro-symbolic approach for creating UI screen embeddings that excel in

computing semantic screen similarity. We measured the effectiveness and generalizability of F RAME on various open source UI datasets in comparison to strong baselines. Our results indicate that F RAME is effective and outperforms key baselines, offering a novel approach for creating embeddings for UI similarity. F RAME is constructed by design as an extensible model that can take advantage of ever-improving large language and vision model representations. DATA AVAILABILITY The code and data associated with this paper along with the experimental infrastructure for evaluating F RAME are provided in an archived replication package [21] and an accompanying source code repository [22] to facilitate replication and future work on the topics of screen retrieval and clustering. ACKNOWLEDGMENTS This research is supported in part by NSF grants CCF2441355 and CCF-2311469. Any opinions, findings, and conclusions expressed herein are the authors’ and do not necessarily reflect those of the sponsors. R EFERENCES [1] B. Myers, “Challenges of HCI design and implementation,” interactions, vol. 1, no. 1, pp. 73–83, Jan. 1994. [Online]. Available: http: //doi.acm.org.proxy.wm.edu/10.1145/174800.174808 [2] B. A. Myers and M. B. Rosson, “Survey on user interface programming,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ser. CHI ’92. New York, NY, USA: Association for Computing Machinery, 1992, p. 195–202. [Online]. Available: https://doi.org/10.1145/142750.142789 [3] D. I. Samudio and T. D. LaToza, “Barriers in front-end web development,” in 2022 IEEE Symposium on Visual Languages and HumanCentric Computing (VL/HCC), 2022, pp. 1–11. [4] M. B. Rosson, S. Maass, and W. A. Kellogg, “Designing for designers: an analysis of design practice in the real world,” in Proceedings of the SIGCHI/GI Conference on Human Factors in Computing Systems and Graphics Interface, ser. CHI ’87. New York, NY, USA: Association for Computing Machinery, 1986, p. 137–142. [Online]. Available: https://doi.org/10.1145/29933.30873 [5] A. B. Tucker, Computer Science Handbook, Second Edition. Boca Raton, FL, USA: CRC Press, 2004. [6] K. Moran, B. Li, C. Bernal-Cárdenas, D. Jelf, and D. Poshyvanyk, “Automated reporting of GUI design violations for mobile apps,” in Proceedings of the 40th International Conference on Software Engineering, ser. ICSE ’18. New York, NY, USA: ACM, 2018, p. 165–175. [Online]. Available: https://doi.org/10.1145/3180155.3180246 [7] Y. Zhao, S. Talebipour, K. Baral, H. Park, L. Yee, S. A. Khan, Y. Brun, N. Medvidović, and K. Moran, “Avgust: automating usage-based test generation from videos of app executions,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 421–433. [Online]. Available: https://doi.org/10.1145/3540250.3549134 [8] S. A. Khan, W. Wang, Y. Ren, B. Zhu, J. Shi, A. McGowan, W. Lam, and K. Moran, “Aurora: Navigating UI tarpits via automated neural screen understanding,” in 2024 IEEE International Conference on Software Testing, Verification and Validation, ser. ICST’16, Apr. 2024. [9] J. Wang, Y. Jiang, T. Su, S. Li, C. Xu, J. Lu, and Z. Su, “Detecting non-crashing functional bugs in Android apps via deep-state differential analysis,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2022. New York, NY, USA: Association for Computing Machinery, 2022, p. 434–446. [Online]. Available: https://doi.org/10.1145/3540250.3549170

[10] S. Feng, M. Xie, and C. Chen, “Efficiency matters: Speeding up automated testing with GUI rendering inference,” in Proceedings of the 45th International Conference on Software Engineering, ser. ICSE ’23. IEEE Press, 2023, p. 906–918. [Online]. Available: https://doi.org/10.1109/ICSE48619.2023.00084 [11] Y. Yan, N. Cooper, O. Chaparro, K. Moran, and D. Poshyvanyk, “Semantic GUI scene learning and video alignment for detecting duplicate video-based bug reports,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3639163 [12] A. Saha, Y. Song, J. Mahmud, Y. Zhou, K. Moran, and O. Chaparro, “Toward the automated localization of buggy mobile app UIs from bug descriptions,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA’24), 2024. [13] K. Moran, C. Bernal-Cárdenas, M. Curcio, R. Bonett, and D. Poshyvanyk, “Machine learning-based prototyping of graphical user interfaces for mobile apps,” IEEE Transactions on Software Engineering, vol. 46, no. 2, pp. 196–221, 2020. [14] F. Behrang, S. P. Reiss, and A. Orso, “Guifetch: supporting app design and development through GUI search,” in Proceedings of the 5th International Conference on Mobile Software Engineering and Systems, ser. MOBILESoft ’18. New York, NY, USA: Association for Computing Machinery, 2018, p. 236–246. [Online]. Available: https://doi.org/10.1145/3197231.3197244 [15] S. Salma, S. M. H. Mansur, Y. Zhang, and K. Moran, “Guievo: Automated evolution of mobile app UIs,” in Proceedings of the 21st International Conference on Mining Software Repositories, ser. MSR ’24. New York, NY, USA: Association for Computing Machinery, 2024, p. 335–347. [Online]. Available: https://doi.org/10.1145/3643991. 3644936 [16] B. Deka, Z. Huang, C. Franzen, J. Hibschman, D. Afergan, Y. Li, J. Nichols, and R. Kumar, “Rico: A mobile app dataset for building data-driven design applications,” in Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’17. New York, NY, USA: ACM, 2017, p. 845–854. [Online]. Available: https://doi.org/10.1145/3126594.3126651 [17] T. J.-J. Li, L. Popowski, T. Mitchell, and B. A. Myers, “Screen2Vec: Semantic embedding of GUI screens and GUI components,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, ser. CHI ’21. ACM, May 2021. [Online]. Available: http://dx.doi.org/10.1145/3411764.3445049 [18] Y. Li, G. Li, X. Zhou, M. Dehghani, and A. Gritsenko, “VUT: Versatile UI transformer for multi-modal multi-task user interface modeling,” 2021. [19] C. Bai, X. Zang, Y. Xu, S. Sunkara, A. Rastogi, J. Chen, and B. A. y Arcas, “UIBert: Learning generic multimodal representations for UI understanding,” 2021. [20] L. A. Leiva, A. Hota, and A. Oulasvirta, “Enrico: A high-quality dataset for topic modeling of mobile UI designs,” in Proc. MobileHCI Adjunct, 2020. [21] “FRAME replication package,” Zenodo, https://doi.org/10.5281/zenodo. 21386573, 2026. [22] “FRAME source code repository,” https://github.com/SageSELab/ FRAME, 2026. [23] C. Maria, P. Dlotko, V. Rouvreau, and M. Glisse, “Rips complex,” in GUDHI User and Reference Manual, 3rd ed. GUDHI Editorial Board, 2023. [Online]. Available: https://gudhi.inria.fr/doc/3.9.0/group rips complex.html [24] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” 2021. [25] A. Clark, “Pillow (pil fork) documentation,” 2015. [Online]. Available: https://buildmedia.readthedocs.org/media/pdf/pillow/latest/pillow.pdf [26] M. Xie, “UIED: UI element detection — detecting UI elements from UI screenshots or drawings,” https://github.com/MulongXie/UIED, 2021. [Online]. Available: https://github.com/MulongXie/UIED [27] Google, “Android developers: Accessibility,” https://developer.android. com/guide/topics/ui/accessibility, 2022. [28] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” 2019. [29] “Huggingface CLIP library,” 2022. [Online]. Available: https:// huggingface.co/docs/transformers/en/model doc/clip [30] “Pytesseract,” 2022. [Online]. Available: https://github.com/h/pytesseract

[31] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” in 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. [Online]. Available: https://openreview.net/forum?id=SJU4ayYgl [32] M. Defferrard, X. Bresson, and P. Vandergheynst, “Convolutional neural networks on graphs with fast localized spectral filtering,” in Proceedings of the 30th International Conference on Neural Information Processing Systems, ser. NIPS’16. Red Hook, NY, USA: Curran Associates Inc., 2016, p. 3844–3852. [33] A. Krishna Vajjala, D. Meher, S. Pothagoni, Z. Zhu, and D. Rosenblum, “Vietoris-rips complex: A new direction for cross-domain cold-start recommendation,” in Proceedings of the SIAM International Conference on Data Mining (SDM), 2024. [34] E. Weisstein, “Heron’s formula,” MathWorld, 2003. [35] K. Pearson, “On lines and planes of closest fit to systems of points in space,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, vol. 2, no. 11, pp. 559–572, 1901. [36] C. Bernal-Cárdenas, K. Moran, M. Tufano, Z. Liu, L. Nan, Z. Shi, and D. Poshyvanyk, “Guigle: A GUI search engine for Android apps,” in 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2019, pp. 71–74. [37] N. Cooper, C. Bernal-Cárdenas, O. Chaparro, K. Moran, and D. Poshyvanyk, “It takes two to tango: Combining visual and textual information for detecting duplicate video-based bug reports,” in Proceedings of the 43rd International Conference on Software Engineering, ser. ICSE ’21. IEEE Press, 2021, p. 957–969. [Online]. Available: https://doi.org/10.1109/ICSE43902.2021.00091 [38] G. Baechler, S. Sunkara, M. Wang, F. Zubach, H. Mansoor, V. Etter, V. Cărbune, J. Lin, J. Chen, and A. Sharma, “Screenai: A visionlanguage model for ui and infographics understanding,” arXiv preprint arXiv:2402.04615, 2024. [39] Y. Jiang, C. Zhou, V. Garg, and A. Oulasvirta, “Graph4GUI: Graph neural networks for representing graphical user interfaces,” in Proceedings of the CHI Conference on Human Factors in Computing

Systems, ser. CHI ’24. ACM, May 2024, p. 1–18. [Online]. Available: http://dx.doi.org/10.1145/3613904.3642822 [40] J. Li, D. Li, C. Xiong, and S. Hoi, “BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in ICML, 2022. [41] J. Wu, Y.-H. Peng, A. Li, A. Swearngin, J. P. Bigham, and J. Nichols, “UIClip: A data-driven model for assessing user interface design,” arXiv preprint arXiv:2404.12500, 2024. [42] P. Duan, J. Warner, Y. Li, and B. Hartmann, “Generating automatic feedback on UI mockups with large language models,” arXiv [cs.HC], 2024. [43] J. Mahmud, N. De Silva, S. A. Khan, S. H. Mostafavi, S. M. H. Mansur, O. Chaparro, A. A. Marcus, and K. Moran, “On using GUI interaction data to improve text retrieval-based bug localization,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ser. ICSE ’24. New York, NY, USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/10.1145/3597503.3608139 [44] J. Mao, J. Bian, H. Ma, Y. Jia, Z. Liang, and X. Jiang, “Robust detection of Android UI similarity,” in 2018 IEEE International Conference on Communications (ICC), 2018, pp. 1–6. [45] E. Schoop, X. Zhou, G. Li, Z. Chen, B. Hartmann, and Y. Li, “Predicting and explaining mobile UI tappability with vision modeling and saliency analysis,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI ’22. New York, NY, USA: Association for Computing Machinery, 2022. [Online]. Available: https://doi.org/10.1145/3491102.3517497 [46] J. Wu, X. Zhang, J. Nichols, and J. P. Bigham, “Screen parsing: Towards reverse engineering of UI models from screenshots,” in The 34th Annual ACM Symposium on User Interface Software and Technology, ser. UIST ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 470–483. [Online]. Available: https://doi.org/10.1145/3472749.3474763 [47] G. Li and Y. Li, “Spotlight: Mobile UI understanding using visionlanguage models with a focus,” 2023.

Related documents

• Device independent graphical display description#133626
• Designable Visual Markers for Mobile Human-Computer Interaction#1100122
• Profiling Through a Digital Mobile Device#474045
• A Benchmarking Framework for Context-aware XR Interfaces#1104665
• Seeing Without Understanding: Large Language Model Evaluation of Mobile User Interface Quality, Failure Taxonomy, and Architectural Explanation#972490