Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

Hallucination-Free GUI Grounding via Regression-Free Layout-Aware Matching

Yuke Li1 Xuehan Hou1 School of Electronic and Computer Engineering, Peking University, Shenzhen, China 2301212734@stu.pku.edu.cn, 2401212849@stu.pku.edu.cn 1

arXiv:2608.09654v1 [cs.AI] 10 Aug 2026

Abstract GUI agents are shifting from metadata-dependent large language models to purely visual multimodal large language models (MLLMs) that operate directly on screenshots. The core task, GUI grounding, requires translating abstract user instructions into precise element coordinates. This task faces a persistent dual obstacle: conventional grounding models lack the semantic richness to interpret abstract instructions, while end-to-end MLLMs suffer from coordinate hallucinations caused by deficient fine-grained perception. We propose a regression-free framework where a frozen MLLM performs instruction parsing and a dedicated grounding model handles precise localization without learning any coordinate regression. A frozen MLLM first elaborates the abstract instruction into a structured visual description rich in layout cues. These descriptions are then fed to a novel Layout-Aware GUI Grounding Model, which performs regression-free localization by matching against layout-prior candidates, inherently suppressing hallucinations and avoiding expensive fine-tuning. The grounding model is trained with only Text/Icon binary labels, requiring no coordinate regression parameters. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end systems; on Mind2Web, it raises success rate and element selection rate by more than 15%. These results demonstrate that decoupling instruction understanding from layout-aware localization effectively resolves the core challenges of GUI interaction.

1

Introduction

Teaching GUI agents to execute instructions such as “close all windows” demands substantially more than recognizing buttons on a screen. It requires the agent to read interface text, comprehend spatial relationships among elements, and map ambiguous natural language to precise pixel coordinates—all without access to the underlying application metadata that developers ordinarily take for granted. Consider a typical multi-window desktop scenario: File Explorer, a browser, and a document editor each possess their own title bar and a small close button in the upper-right corner. A human instinctively associates the title “File Explorer” with the corresponding close icon, whereas an agent that confuses titles or misjudges the spatial extent of “upper-right” will close the wrong window, or fail entirely. Endowing GUI agents with this seemingly ordinary human capability—robust, text-aware, layout-sensitive localization—is the central problem this paper sets out to solve. Existing approaches each encounter fundamental bottlenecks. Methods that parse interface metadata (e.g., HTML DOM trees or Android view hierarchies) are often inapplicable in sandboxed or secure environments where such structural data is unavailable [16, 43, 44, 37, 35]. To circumvent metadata dependence, end-to-end multimodal large language models (MLLMs) such as CogAgent and SeeClick directly predict coordinates from screenshots [12, 41, 4]. However, their visual encoders, designed Preprint.

Figure 1: Framework of our proposed method. This approach integrates a Multimodal Large Language Model (MLLM) to interpret user instructions and produce detailed visual caption. These descriptions are subsequently utilized by a specially designed layout-aware grounding model to pinpoint the locations of user interface (UI) elements with enhanced precision.

for natural images, lack inductive biases for GUI layout regularities, leading to persistent coordinate hallucinations — geometrically plausible yet incorrect predictions — and failure on spatially qualified instructions [5, 27, 42, 39, 8]. Engineering wrappers do not resolve this: OpenCLaw, a highly engineered GUI agent, still relies on the underlying MLLM’s zero-shot visual capacity. Traditional grounding models such as Grounding DINO [25, 31] further suffer from weak text understanding and no spatial layout perception, making them helpless for text-spatial queries. Fine-tuning on coordinate supervision can partially suppress hallucinations but demands expensive manual annotations and generalizes poorly. These three paths converge on a single hard truth: the GUI grounding field lacks a model that natively understands interface layout structure and supports open-vocabulary matching without costly coordinate regression or external markup. Building that missing model is our goal. Designing such a model means satisfying three constraints simultaneously: it must predict precise pixel coordinates without ever learning a coordinate regression mapping; it must internalize the basic layout primitives of a GUI—text blocks and icons—without requiring instance-level semantic labels; and it must handle spatially modified queries (“upper-right,” “left side”) without per-query spatial annotations, while remaining robust when spatial information is irrelevant. These constraints give rise to three tightly related difficulties. First, frozen cross-modal matching risks losing spatial precision, yet any reintroduction of regression opens the door to hallucinations; the pipeline must remain regression-free while still producing fine-grained, coordinate-level accuracy. Second, standard detectors do not distinguish text from icons and treat GUIs as indistinct visual patches, but full category-level bounding-box annotation is prohibitively expensive; the model must acquire this discriminative layout sense from only a handful of images labeled with two classes—Text and Icon. Third, spatial queries require positional bias, yet brute-force annotation of every possible spatial relation does not scale, and a heavy positional encoding head easily overfits while harming performance on spatially neutral queries; spatial awareness must be injected through an extremely lightweight mechanism that learns to shift attention when spatial cues are present and to remain nearly silent otherwise. We address all three difficulties together with a unified Layout-Aware GUI Grounding Model, built upon a decoupled two-stage architecture. A fully frozen MLLM first converts the user instruction and screenshot into a structured textual description rich in layout cues—for example, “the close button in the upper-right corner of the ‘File Explorer’ window”—without producing any coordinates. The core of our model, a dedicated grounding module, then takes this description and localizes the target through two tightly integrated steps: it generates a compact set of layout-prior candidates by domain-adapting a detector on a small set of GUI images labeled only with Text and Icon, and it selects the best-matching candidate via frozen CLIP cross-modal similarity, entirely without learning any coordinate regression parameters. A lightweight geometric fusion module—a single learnable linear projection that injects spatial encoding into the visual feature—handles spatially sensitive queries with negligible overhead, while remaining nearly silent for plain queries. In our running example, the “File Explorer” text candidate naturally scores highly against the generated description, and the geometric fusion nudges the highest score toward the actual close icon, delivering a clean, regression-free hit. 2

Every component of this design directly targets a specific weakness of existing approaches. The frozen matching pathway eliminates the regression-induced hallucinations that plague MLLMs. Layout-prior candidate generation compensates for the layout blindness of generic detectors and end-to-end models. Lightweight geometric fusion handles spatially modified queries without demanding costly spatial annotations. Importantly, the entire grounding module is trainable with only a small amount of generic layout labels and a handful of spatial training pairs. On ScreenSpot-Pro, our method achieves over 20% improvement in grounding accuracy over end-to-end state-of-the-art systems, and on Mind2Web it raises the success rate and element selection rate by more than 15%. Controlled ablation studies further confirm that layout-prior candidate generation and lightweight geometric fusion each contribute substantially to these gains. This work makes the following contributions in overview: • Propose a dedicated Layout-Aware GUI Grounding Model. We introduce a visual grounding model specifically designed for interface layout structures and open-vocabulary matching. By replacing coordinate regression with frozen cross-modal matching, this model fundamentally prevents coordinate hallucinations while delivering precise pixel-level localization. • Design a layout-prior injection and lightweight geometric fusion mechanism. This mechanism, using only a small set of Text/Icon-level annotations, endows the grounding model with both text semantic understanding and spatial layout perception. It eliminates the dependency on expensive fine-grained coordinate supervision and enables the model to handle spatially sensitive queries with negligible parameter overhead. • Build a decoupled intent-driven grounding framework. We embed the proposed grounding model into a two-stage framework where a frozen MLLM handles instruction parsing and our model performs precise localization. This decoupled architecture fully harnesses the semantic understanding capability of MLLMs and the fine-grained perception advantage of the specialized grounding model, achieving leading performance on multiple benchmarks while avoiding the high costs of full fine-tuning.

2

Related Work

2.1

Autonomous GUI Navigation

Early research explored GUI automation using rule-based engines or traditional machine learning [33, 23, 10, 22, 17]. With the progress of large language models, LLM-driven agents have become mainstream, enhancing web interaction through in-context learning [43], self-refinement [16], or specialized training for website interaction [11]. However, text-only LLMs cannot directly process visual interfaces and depend on GUI metadata or platform-specific customization [43, 16, 22, 7, 36]. Recent vision-driven agents employing multimodal large language models show promise [12, 5, 32, 26], yet their performance remains constrained by the fine-grained perception limitations of MLLMs. Efforts to improve perception through specialized datasets face high construction costs and computational demands. This work proposes a decoupled framework that separates task planning from visual grounding, flexibly integrating an MLLM for semantic understanding with a specialized layout-aware model for precise localization, significantly improving accuracy while avoiding full fine-tuning. 2.2

Multimodal Large Language Models

The evolution of multimodal large language models (MLLMs) [34, 1–3, 19, 35, 18, 13] has unlocked new capabilities in cross-modal understanding. Early frameworks such as CLIP [28] and ALIGN [15] established vision-language joint representation learning, while BLIP [18] introduced cross-attention mechanisms for visual question answering and image captioning. GPT-4V [1] subsequently enabled visual in-context learning and inspired lightweight adaptation frameworks like LLaVA [24] and MiniGPT-4 [45]. Despite these advances, existing MLLMs face two critical challenges in GUI automation: insufficient fine-grained perception makes distinguishing visually similar elements difficult, and weak adaptability to dynamic layouts hinders the capture of spatial-semantic relationships of functional controls. Recent approaches address these limitations through domain adaptation— PIX2ACT++ [32] employs hierarchical screenshot encoding, while CogAgent [12] enhances element 3

localization via high-resolution visual tokenizers—yet both still depend on large-scale annotated data and fail to fully exploit inherent GUI design principles. This work proposes a task-deconstructed dual-stage localization framework that decouples GUI grounding into semantic parsing and visual grounding. By combining existing MLLMs’ semantic understanding with a specially trained layout-aware localization model, we achieve precise element grounding without full fine-tuning, fully leveraging interface layout patterns to enhance cross-platform accuracy and robustness while reducing computational costs. 2.3

Visual grounding

In related work on visual grounding models[21, 25, 31], approaches like GLIP[21] and Grounding DINO[25] have significantly advanced open-vocabulary detection. GLIP reformulates object detection as a phrase grounding task, achieving language-aware visual representation learning, while Grounding DINO further enhances open-set object detection capability through its Transformer-based architecture. However, these methods still exhibit notable limitations in GUI automation scenarios: they are primarily designed for natural images, lacking sufficient understanding of the unique visual patterns and spatial layout characteristics specific to interface elements. We have developed an independently trained visual grounding model that substantially improves element localization accuracy and robustness by deeply exploring layout structure information within GUI interfaces. While extracting visual features, our model explicitly incorporates existing layout patterns in GUI interfaces, thereby overcoming the limitations of traditional methods in layout awareness for GUI contexts.

3

Method

Preliminary. GUI visual grounding translates natural language instructions into precise element coordinates on an interface screenshot S ∈ RH×W ×3 , outputting a bounding box b = (xmin , ymin , xmax , ymax ). Existing methods suffer from a fundamental trade-off: conventional grounding models lack semantic understanding, while end-to-end multimodal large language models (MLLMs) introduce coordinate hallucinations due to deficient fine-grained perception. We propose a decoupled intent-driven grounding framework that splits the task into instruction semantic elaboration and precise visual grounding. A training-free MLLM first produces structured visual descriptions from abstract instructions; then a Layout-Aware lightweight matching model performs accurate pixel-level grounding without full-scale fine-tuning. This separation allows the MLLM to focus on semantic reasoning while the specialized grounding model concentrates on fine-grained visual perception, effectively suppressing hallucinations. Details of each phase are given in Sections 3.1 and 3.2, with training procedures in Section 3.3. 3.1

Instruction Elaboration

The user’s original instruction U is typically expressed as an abstract functional description (e.g., “click the login button”, “navigate to the personal information page”), lacking specific visual cues that can be directly exploited for visual grounding. Inspired by the powerful cross-modal reasoning and commonsense knowledge of MLLMs, we introduce a frozen-parameter MLLM (e.g., Qwen-VL, GPT-4V) as the intent parser. In this stage, the MLLM receives the current screenshot I and the user instruction U , and is guided via in-context prompt engineering to produce a structured visual description Q. This description precisely characterizes the visual attributes and spatial features of the target element. For example, “click the login button” is concretized as “a blue rectangular button labeled ‘Login’ located in the center of the page”. Since the MLLM does not undergo any gradient updates, this stage completely avoids the cost of large-scale model fine-tuning and the risk of catastrophic forgetting, while providing high-quality, semantically rich queries for downstream grounding. 3.2

Precise Localization

After obtaining the visual query Q, the subsequent pipeline is responsible for mapping it to a precise location in the image. In contrast to directly adopting an end-to-end learnable coordinate regression, we design a two-stage Layout-Aware grounding module that explicitly introduces a text/icon binary 4

Figure 2: The overall architecture of the proposed layout-aware grounding model for GUI grounding, comprising three main stages: (1) text feature extraction, where visual queries are encoded into text embeddings to capture both textual and icon semantics of target elements; (2) layout-aware image feature extraction, where GUI image embeddings fused with position embeddings are encoded, followed by query selection to provide keys and values to a decoder that takes content queries and initial anchors as inputs to generate candidate regions, which are then fed into a layout encoder to produce layout-aware image embeddings; and (3) visual-textual feature alignment, where the text embeddings and layout-aware image embeddings are matched in a cross-modal manner to predict the bounding box of the target element. prior from the interface and geometric feature fusion, effectively suppressing coordinate hallucinations while avoiding full-scale fine-tuning. 3.2.1

Layout-Prior Candidate Region Generation

The first stage is handled by a GUI-domain-adapted DINO detector[40]. We perform lightweight fine-tuning using only a small set of screenshots annotated with the two most basic layout types— “Text” and “Icon”. This design enables the detector to learn only to distinguish between text blocks and icon blocks during domain transfer, without involving any specific semantic categories; hence we refer to it as Layout-Aware candidate generation. The DINO Transformer decoder outputs an embedding vi ∈ Rd containing primary visual semantics for each predicted box. Meanwhile, we explicitly extract its normalized geometric attributes: h i w (1) gi = xc , yc , w, h, , w · h , h where (xc , yc ) are the relative coordinates of the region center, and (w, h) are the relative width and height, all normalized to [0, 1]. This geometric vector reserves an interface for subsequent spatial enhancement. 3.2.2

Hallucination-Resistant Cross-Modal Feature Matching

After obtaining candidate regions, we abandon the traditional learnable regression head and instead adopt a frozen-parameter feature matching strategy to produce the final coordinates. Specifically, we introduce the pre-trained vision-language model CLIP (ViT-B/32)[29] as the feature extractor. For each candidate region ri , we crop the image and extract the visual feature fi ∈ Rdc using the CLIP image encoder; meanwhile, the visual query Q obtained from the first stage is encoded as a text feature t ∈ Rdc . After ℓ2 normalization, we compute the cosine similarity: si =

fi · t . ∥fi ∥∥t∥

(2)

The candidate region with the highest similarity i∗ = arg maxi si is selected as the grounding result, and its bounding box is directly output as the predicted coordinates. This matching paradigm does not rely on learning any category-specific coordinate regression parameters, fundamentally eliminating the coordinate bias and hallucinations that may be introduced by full fine-tuning. 5

Simultaneously, to ensure the model’s responsiveness to spatially sensitive queries (e.g., “the return button in the upper left corner”), we incorporate an extremely lightweight geometric feature fusion module. For a candidate region ri , its geometric vector gi ∈ R6 is first mapped via a learnable single-layer linear projection to a spatial offset of the same dimension as the CLIP visual feature: hi = Wg gi + bg ,

(3)

where Wg ∈ Rdc ×6 and bg ∈ Rdc are the only few learnable parameters. The offset is then additively fused into the original visual feature to obtain the layout-enhanced visual representation: f̃i = fi + hi .

(4)

During matching, the enhanced f̃i replaces the original fi in the similarity computation. This design has very few learnable parameters, making training efficient. The additive fusion form allows the model to naturally learn an offset close to zero for spatially irrelevant queries, preserving CLIP’s generalization ability; for spatially relevant queries, the geometric features provide positional compensation to the visual representation, steering the similarity scores toward regions that match the spatial expectation. The training of this module requires only a small number of samples with spatial annotations, while all other model parameters remain frozen. 3.3

Training Details

Training Data Construction. We construct the training dataset following the pipeline illustrated in Figure 3. First, we sample approximately 200k screenshots from widely used GUI datasets. Using automated parsing tools[27], we extract all textual and iconic regions from these images. Each extracted region is then fed into Qwen3-VL[2] to generate a corresponding visual description, resulting in a collection of region-caption pairs. Training Configuration. For the domain adaptation of the DINO detector (Stage 1), we use 200k GUI screenshots labeled only with “Text” and “Icon”, split 9:1 for training and validation. The model employs a Swin-Tiny backbone initialized from COCO 12-epoch weights and is trained on 8 RTX 3090 GPUs with batch size 1 per GPU (effective 8) for 50 epochs. We use AdamW with base learning rate 1 × 10−4 , backbone learning rate 1 × 10−5 , weight decay 1 × 10−4 , and cosine annealing. Data augmentation consists solely of random scaling. All classification heads beyond the two layout classes are randomly initialized. In the matching and layout enhancement stage (Stage 2), we freeze CLIP ViT-B/32 and train only the geometric projection matrix Wg and bias bg , initialized from a standard normal distribution. We construct positive pairs from manually annotated spatial queries and target regions, with negatives sampled in-batch, and optimize using Adam with learning rate 1 × 10−3 and batch size 256 for 20 epochs on the same GPU setup. Loss Function. The domain adaptation stage for the DINO detector employs a multi-task detection loss following the standard DINO formulation: Ldet = λreg Lreg + λgiou Lgiou + λcls Lcls ,

(5)

where Lreg is the L1 regression loss for bounding box refinement, Lgiou is the generalized IoU loss for spatial alignment, and Lcls is the focal loss for the binary classification between “Text” and “Icon” categories. Following standard practice, we set λreg = 5.0, λgiou = 2.0, and λcls = 2.0. The matching and layout enhancement stage is trained with a contrastive loss (InfoNCE). Formally, for a mini-batch of B spatial query-region pairs, the loss is defined as: B

Lalign = −

1 X exp(sim(f̃i , ti )/τ ) log PB , B i=1 j=1 exp(sim(f̃j , ti )/τ )

(6)

where sim(·, ·) denotes cosine similarity, τ is a temperature hyperparameter set to 0.07, and f̃i and ti are the geometry-enhanced visual feature and the text feature of the i-th description, respectively. The detection loss Ldet is only applied during the domain adaptation stage, while the contrastive loss Lalign is only applied during the matching and layout enhancement stage. 6

Figure 3: Training Dataset Construction Pipeline. We first sampled 200k images from common GUI datasets, then utilized parsing tools to extract icon and text regions from these images. Subsequently, we employed Qwen3-VL to generate image descriptions for these regions, ultimately producing region-caption pairs for model training.

4

Experiments

In this section, we conducted experiments on both GUI grounding and GUI agent downstream tasks to validate the effectiveness of the proposed framework and the typography-aware grounding model. For the instruction elaboration phase, we employed Qwen2.5-VL-72B[3].

4.1

GUI Visual Grounding

We comprehensively evaluate our method on the ScreenSpot[5] benchmark, a specialized dataset designed for visual grounding in professional application interfaces. This dataset contains 1,272 single-step interaction tasks collected from diverse professional software environments including productivity tools, creative applications, and enterprise systems across mobile (iOS/Android), desktop (macOS/Windows), and web platforms. For more rigorous evaluation, we extend our assessment to the ScreenSpot-Pro benchmark[20], a specialized dataset comprising 1,581 instruction-ground truth pairs collected from 23 professional applications across Windows, macOS, and Linux platforms. This dataset features meticulously annotated tasks with complex interface elements from specialized software environments including AutoCAD and Office suites. Both benchmarks encompass a wide spectrum of professional GUI components including textual elements in business contexts and domain-specific symbolic icons, providing comprehensive coverage of real-world professional usage scenarios. Compared Methods & Evaluation. Our comparative analysis encompasses several categories of models: advanced multimodal large language models including GPT-4o[14], Qwen2.5VL-72B[3]; specialized GUI understanding models such as SeeClick[5], Fuyu[4], CogAgent[12], UGround[9], and OS-Atlas-7B[38]. Following established evaluation protocols for web agents, we employ click accuracy as our primary metric, defined as the percentage of test samples where the model’s predicted coordinates fall within the ground truth element bounding box. This evaluation methodology maintains consistency with previous work like SeeClick, ensuring fair and comparable performance assessment across different approaches. Results. As shown in Table 1, our method demonstrates superior performance compared to other models, achieving an average accuracy of 87.1% across Mobile, Desktop, and Web platforms. Table 2 further demonstrates its effectiveness with an average score of 41.3% across six specialized domains. These results validate the efficacy of our proposed framework, where the task-decoupling design effectively leverages the complementary strengths of both MLLMs and dedicated grounding models to achieve robust localization performance. 7

Mobile

Desktop

Web

Model

Text

Icon/Widget

Text

Icon/Widget

Text

Icon/Widget

Average

Fuyu CogAgent GPT-4o Qwen2.5VL-72B SeeClick Uground Ours

8.4 8.4 25.6 29.0 78.0 82.8 95.2

6.6 6.6 24.5 26.2 52.0 60.3 81.4

6.2 6.2 14.4 37.8 72.2 82.5 94.4

2.9 2.9 19.3 26.9 30.0 63.6 78.9

6.5 6.5 9.6 7.4 55.7 80.4 92.7

3.4 3.4 6.3 6.1 32.5 70.4 80.1

5.7 5.7 16.6 22.2 53.4 73.3 87.1

Table 1: Results of different LVLMs on ScreenSpot. MODEL

Dev

Creative

CAD

Scientific

Office

OS

AVG

GPT-4o SeeClick (7B) MiniCPM-V (7B) CogAgent (18B) ShowUI (2B) UGround-7B OS-Atlas-7B Ours

1.3 0.5 2.8 7.3 10.1 17.7 21.3 44.9

0.3 0.6 1.3 5.3 4.2 14.9 16.4 34.5

1.1 1.7 3.6 6.2 4.4 10.9 9.9 27.3

1.0 2.5 5.4 13.4 10.9 19.0 25.3 37.4

0.9 0.8 2.8 9.2 12.9 26.0 26.2 55.7

0.0 1.6 2.3 3.0 6.6 10.9 17.4 48.2

0.6 1.1 3.0 7.7 7.7 16.5 18.9 41.3

Table 2: Results of different LVLMs on ScreenSpot-Pro. 4.2

GUI Agents

We evaluate the effectiveness of our method in both smartphone and web interaction environments using the Android in the Wild (AITW)[30] and Mind2Web[6] benchmarks. The AITW dataset contains 30,000 natural language instructions and 715,000 multi-step operation trajectories, covering real-world mobile interaction scenarios including system settings adjustment, application switching, and form filling. The Mind2Web dataset focuses on web environments, comprising over 2,000 tasks from 137 real websites across 31 different domains, providing a diverse testing environment for building general web agents. Compared Methods & Evaluation. We compare our approach with various state-of-the-art methods, including task-finetuned models such as Qwen-VL-7B[2], and SeeClick[5], as well as GPT-4V[1], and Qwen2.5VL-72B[3] evaluated in a training-free setting. Our method maintains consistent output format with SeeClick, indicating target elements through coordinate outputs to ensure precise localization and interaction with GUI elements. For evaluation metrics, we adopt the screen-wise action matching score as the primary metric in the AITW[30] dataset, while computing Element Accuracy, Operation F1 score, and Step Success Rate for the Mind2Web dataset[6]. For vision-based methods, a prediction is considered correct when the predicted coordinates fall within the bounding box of the target element. Results. As shown in Table 3, our framework achieves an average improvement of 7.9% over the baseline on the AITW dataset, with a 24.2% gain in task success rate compared to using Qwen2.5VL72B alone. On the more challenging Mind2Web dataset (Table 4), our method achieves substantial gains in Element Accuracy and Step Success Rate across all task configurations. The Operation F1 score shows a narrower margin, which we attribute to our use of training-free MLLMs that trade task-specific optimization on action classification for stronger generalization. Model SeeClick QwenVL-7Bfinetuned GPT-4V Qwen2.5VL-72B Ours

General

Install

GoogleApps

Single

WebShopping

Overall

54.0 49.5 31.4 35.1 61.5

66.4 59.9 32.1 50.1 71.4

54.9 46.9 29.1 51.6 60.6

63.5 64.7 39.7 41.2 72.9

57.6 50.7 22.5 39.2 63.5

59.3 54.3 31.0 43.4 67.6

Table 3: Average scores of different methods on AITW.

8

Model

Cross-Task

Cross-Website

Cross-Domain

Ele.Acc Op.F1 Step SR Ele.Acc Op.F1 Step SR Ele.Acc Op.F1 Step SR QwenVL-7Bfinetuned SeeClick GPT-4V CogAgent Qwen2.5VL-72B Ours

15.9 28.3 5.1 22.4 27.7 44.9

86.7 87.0 64.0 53.0 74.5 75.8

13.3 25.5 4.6 17.6 24.3 41.2

13.2 21.4 5.2 18.4 25.6 43.8

83.5 80.6 61.0 42.2 72.7 75.4

9.2 16.4 4.2 13.4 23.1 39.3

14.1 23.2 5.2 20.6 24.8 40.1

84.3 84.8 62.7 42.0 72.9 75.2

12.0 20.8 4.6 15.5 23.6 37.3

Table 4: Comparsion of methods on Mind2Web.

Figure 4: (a) Instruction elaboration: abstract user commands are transformed into structured visual captions. (b)–(c) Icon detection and text recognition after pre-training. (d) Final localization via fine-tuning with visual captions. 4.3

Qualitative Results and Case Studies

Figure 4 illustrates the processing pipeline of our framework through two representative cases. Subfigure (a) shows abstract user commands transformed into structured visual captions by the frozen MLLM, capturing key visual attributes and spatial cues. Subfigures (b) and (c) demonstrate robust icon and text detection via layout-prior candidate generation, effectively distinguishing text blocks from icon regions. Subfigure (d) presents the final precise localization results produced by regression-free cross-modal matching. Together, these visualizations validate the effectiveness of each design component and the seamless coordination across the entire grounding pipeline.

5

Conclusion

Our approach resolves the core dilemma in GUI grounding by decoupling instruction elaboration from precise localization. A frozen MLLM translates abstract instructions into structured visual descriptions; a dedicated Layout-Aware GUI Grounding Model then localizes targets via layout-prior candidate generation and regression-free cross-modal matching, inherently suppressing coordinate hallucinations without expensive fine-tuning. On ScreenSpot-Pro and Mind2Web, our method achieves over 20% and 15% improvement respectively over end-to-end state-of-the-art systems, with especially large gains on spatially sensitive queries—demonstrating a practical path toward hallucination-free GUI grounding. This work contributes: (1) a Layout-Aware GUI Grounding Model that replaces coordinate regression with frozen cross-modal matching, preventing hallucinations while delivering precise localization; (2) a layout-prior injection and lightweight geometric fusion mechanism that acquires text understanding and spatial layout perception from minimal Text/Icon-level annotations; and (3) a decoupled intentdriven framework that separates instruction parsing from localization, harnessing MLLM semantics and specialized fine-grained perception without full fine-tuning.

9

References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. [4] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar. Introducing our multimodal models, 2023. [5] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935, 2024. [6] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114, 2023. [7] Hiroki Furuta, Kuang-Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction-finetuned foundation models. In ICLR, 2024. [8] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024. [9] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024. [10] Izzeddin Gur, Ulrich Rueckert, Aleksandra Faust, and Dilek Hakkani-Tur. Learning to navigate the web. In International Conference on Learning Representations. [11] Izzeddin Gur, Hiroki Furuta, Austin V Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. In ICLR, 2024. [12] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281–14290, 2024. [13] Xuehan Hou, Meng Fan, Pengchong Qiao, Zesen Cheng, Yian Zhao, Lei Zhu, Kaiwen Cheng, Chang Liu, and Jie Chen. Anchor Frame Bridging for Coherent First-Last Frame Video Generation. In International Conference on Learning Representations (ICLR), 2026. [14] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. [15] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904–4916. PMLR, 2021. [16] Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36:39648–39677, 2023. [17] Gang Li and Yang Li. Spotlight: Mobile ui understanding using vision-language models with a focus. In The Eleventh International Conference on Learning Representations. [18] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning, pages 12888–12900. PMLR, 2022. [19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023.

10

[20] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786, 2025. [21] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10965–10975, 2022. [22] Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03776, 2020. [23] Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802, 2018. [24] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. [25] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pages 38–55. Springer, 2024. [26] Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, et al. Autoglm: Autonomous foundation agents for guis. arXiv preprint arXiv:2411.00820, 2024. [27] Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. Omniparser for pure vision based gui agent. arXiv preprint arXiv:2408.00203, 2024. [28] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. [29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. [30] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Androidinthewild: A large-scale dataset for android device control. Advances in Neural Information Processing Systems, 36: 59708–59728, 2023. [31] Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, et al. Grounding dino 1.5: Advance the" edge" of open-set object detection. arXiv preprint arXiv:2405.10300, 2024. [32] Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina N Toutanova. From pixels to ui actions: Learning to follow instructions via graphical user interfaces. Advances in Neural Information Processing Systems, 36: 34354–34370, 2023. [33] Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR, 2017. [34] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. [35] Bryan Wang, Gang Li, and Yang Li. Enabling conversational interaction with mobile ui using large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–17, 2023. [36] Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 543–557, 2024.

11

[37] Qinzhuo Wu, Weikai Xu, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, and Shuo Shang. Mobilevlm: A vision-language model for better intra-and inter-ui understanding. In EMNLP (Findings), 2024. [38] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218, 2024. [39] Yuhao Yang, Yue Wang, Dongxu Li, Ziyang Luo, Bei Chen, Chao Huang, and Junnan Li. Aria-ui: Visual grounding for gui instructions. arXiv preprint arXiv:2412.16256, 2024. [40] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In The Eleventh International Conference on Learning Representations. [41] Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. arXiv preprint arXiv:2309.11436, 2023. [42] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024. [43] Longtao Zheng, Rundong Wang, Xinrun Wang, and Bo An. Synapse: Trajectory-as-exemplar prompting with memory for computer control. In NeurIPS 2023 Foundation Models for Decision Making Workshop. [44] Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023. [45] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.

12

Related documents

• FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting#155428
• AI Prototyper: A Figma Plugin for Decomposition-Based GUI Prototyping with LLMs#373455
• Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding#1109337
• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations#617051
• UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations#616551