arXiv CS open access

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Kyeongseon Kim et al. · arxiv_cs

arXiv CS · Papers

artificial-intelligence knowledge-representation reasoning

GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves t

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints Kyeongseon Kim*, 1 , Jiyeon Son*, 1 , Tae-Hyun Oh1 1 KAIST Correspondence: ella94.ai@gmail.com

arXiv:2610.32536v1 [cs.AI] 23 Oct 2026

Abstract GUI grounding benchmarks ask whether an agent can localize a target that is already visible in a static screenshot. Real interfaces often violate this assumption: a target may be off-screen, appear only after a state change or hover, remain occluded, or become actionable after a delay. This leaves the ability to recover target visibility unevaluated. To measure this reveal step, we introduce Interactive GUI Grounding (IGG)1 , a benchmark in which an agent must first make the target visible through interaction and then click it. IGG contains 601 tasks across 16 controllable GUI environments and seven visibility-constraint sub-types, with a compact action space of scrolling, moving, dragging, waiting, and clicking. In our human study, participants achieve 94.6% accuracy, while GPT5.5 reaches 79.4% and the strongest evaluated open-source VLM reaches only 31.6%. To identify the source of this gap, we conduct two controlled interventions. Telling Qwen3-VL which interaction to perform raises its accuracy only from 30.8% to 33.9%, whereas performing the interaction for the model raises it to 81.9%. The bottleneck is executing the reveal interaction, not recognizing the target once it is exposed. We further show that this capability is learnable, demonstrating that visibility recovery can be strengthened consistently across model families.

Introduction

GUI agents increasingly rely on vision-language models (VLMs) to map natural-language instructions to screen locations (Wu et al., 2025; Mittal et al., 2026; Koh et al., 2024; Kumbhar et al., 2026; Gao et al., 2024; Zheng et al., 2024). Existing grounding benchmarks evaluate this mapping within a single screenshot and typically assume * Equal contribution. The benchmark and code are available at https:// igg-bench.github.io/. 1

that the target is already visible (Cheng et al., 2024; Li et al., 2025). However, routine interfaces often violate this assumption (Figure 2): a target may lie outside the viewport (Burigat et al., 2012), sit in a collapsed panel, be covered by a popup, be distinguishable from its neighbors only on hover, or activate only after a delay. In each case, no static grounding model, however accurate, can localize the target in the initial screenshot. The agent must first select and execute an interaction that reveals it, and only then recognize and click it. Static grounding benchmarks leave this reveal step unmeasured, and navigation benchmarks fold it into end-to-end task success, where it cannot be separated from planning. In light of this gap, this paper asks whether current GUI agents can make a target visible before grounding it and, when they cannot, at which stage they fail. To fill this gap, we introduce Interactive GUI Grounding (IGG) to measure this missing step. An IGG episode decomposes success into four stages: selecting a reveal action, executing it at the correct location or with the correct parameters, recognizing the target once it appears, and localizing the final click. We call the first two stages visibility recovery. The remaining stages, recognizing the target once it appears and localizing the click, coincide with static grounding (Section 3.1). Thus, like static grounding, IGG scores a single final click. Unlike static grounding, it additionally requires the agent to recover the target’s visibility beforehand. Because interaction serves only to expose the target, IGG isolates visibility recovery without introducing the broader planning demands of full GUI navigation. IGG contains 601 tasks across 16 controllable GUI environments modeled on familiar applications. Tasks fall into seven sub-types under three constraint types, Viewport, Multi-State, and Temporal/Manipulation, and are solved with five actions: S CROLL, M OVE, D RAG, WAIT, and C LICK. De-

Static GUI Grounding + Scroll Constraint Type 1: Viewport Constraint Revealed by viewport movement + MOVE Constraint Type 2: Multi-State Constraint Revealed by state-changing interactions 1. Observability

2. Disambiguation

3. State Control

4. Temporal Control

Visual Disambiguation

Hidden State Activation

Removable Occlusion

+ DRAG, WAIT Constraint Type 3: Temporal/Manipulation Constraint Revealed over time or by manipulation action

SCROLL

MOVE

CLI CK

WAIT

DRAG

Delayed State Activation

Dynamic Target

Movable Occlusion

Figure 1: Overview of Interactive GUI Grounding (IGG). Left: representative visibility constraints require observability , disambiguation , state control , or temporal control before the target can be clicked. Center: these capabilities and a compact action space define Viewport Constraint, Multi-State Constraint, and Temporal/Manipulation Constraint. Right: human solvability exceeds model grounding accuracy on all three constraint types, and the gap is largest on viewport constraints for Qwen3-VL and on temporal and manipulation constraints for GPT-5.5.

terministic initial states and stable target identities make evaluation replayable at click time, while the environments preserve the interaction patterns that make visibility recovery difficult: hover reveals, popups, delayed activation, carousels, and movable occlusions. Our experiments identify interaction execution as the main bottleneck in IGG. Telling a model which interaction to perform provides little improvement, whereas performing the reveal action for it recovers most of its accuracy. SFT on successful interaction trajectories followed by online GRPO further improves visibility recovery across Qwen3-VL, InternVL3.5, and GUI-Owl-1.5. Our contributions are: • We formulate interactive GUI grounding as visibility recovery followed by target recognition and localization, separating it from both static grounding and end-to-end navigation. • We introduce IGG, a benchmark of 601 tasks, 16 controllable GUI environments, and seven visibility-constraint sub-types, together with a click-time evaluator and human-verified groundtruth interaction trajectories. • We provide controlled interventions, heuristic analyses, cross-viewport and paired mock-real evaluations, and training studies that identify where current agents fail and show consistent improvements across three VLM architectures.

Related Work

GUI grounding. GUI grounding maps a naturallanguage instruction to the corresponding interface element (Cheng et al., 2024; Tang et al., 2025). ScreenSpot (Cheng et al., 2024), ScreenSpotPro (Li et al., 2025), and OSWorld-G (Xie et al., 2025) evaluate localization within a screenshot where the target is visible. Recent methods improve this setting through rich screenshot supervision (Gao et al., 2024), diffusion-based prediction (Kumbhar et al., 2026), UI decomposition (Xie et al., 2025), and multi-turn refinement (Mittal et al., 2026). Universal grounding extends localization to agent settings (Gou et al., 2025), but still predicts a target from the currently observed state. IGG instead evaluates the preceding visibility-recovery step: the agent must change the interface before the target can be recognized and localized. GUI navigation Mind2Web (Deng et al., 2023), WebArena (Zhou et al., 2024), OSWorld (Xie et al., 2024) evaluate multi-step task completion on web and desktop interfaces, and agent-side improvements such as interaction-history compression (Zhu et al., 2025) are measured against their end-to-end success. Because only end-to-end success is scored, planning, exploration, and grounding failures collapse into a single signal. IGG instead scores one narrower capability, visibility recovery followed by a single evaluated click (Section 3.1). Just as static grounding was separated from navigation and became a standard evaluation axis (Cheng et al.,

Table 1: Minimal action space A for visibility recovery.

3.2

Action

We categorize tasks by the interaction required to recover target visibility (Figures 1 and 2).

Role

SCROLL(dir, ∆px ) MOVE(xpx , ypx )

Scroll to reveal off-screen regions. Move the pointer to trigger hover states or expose disambiguating cues. DRAG(startX, Drag an element between the startY, endX, specified start and end coordinates. endY) WAIT(∆tms ) Pause for a specified duration to handle delayed or asynchronous UI updates. CLICK(xpx , ypx ) Click the specified location; grounding is evaluated at this step.

2024), IGG isolates a step that navigation benchmarks contain only implicitly, and it complements rather than replaces them.

The IGG Benchmark

Our goal is to measure visibility recovery in isolation and reproducibly. To this end, we formalize the task (Section 3.1), categorize the interactions it requires (Section 3.2), describe the environments and how tasks were built (Section 3.3), specify the evaluation protocol (Section 4.1), and validate the benchmark with human studies (Section 4.5). 3.1

Problem formulation

An IGG task is specified by an initial interface state in which the target is not visible or not actionable, a goal-only instruction q that identifies the target but not the procedure for exposing it, a target element with a stable identifier, and an evaluator that scores a C LICK as successful if and only if the clicked element’s identifier matches the target’s. Each task also carries one human-verified ground-truth trace, which is withheld from the agent and used only for efficiency metrics. At step t, the agent receives the screenshot st , the instruction q, and its action history ht−1 = (a1 , . . . , at−1 ), selects an action from A = {S CROLL, M OVE, D RAG, WAIT, C LICK}, and receives the next screenshot st+1 (Table 1). IGG uses screenshot-only observations and restricts interaction to these five reveal-and-click actions. The episode ends with success on the first click that hits the target and with failure when the action budget is exhausted. A click elsewhere applies any defined state transition, and the episode continues.

Visibility constraint taxonomy

Constraint Type 1: Viewport Constraint. The target is outside the current viewport but reachable within the same UI state. • CT1-1. Scroll-based Exploration: The target is off-screen and must be revealed by scrolling within the current state. Constraint Type 2: Multi-State Constraint. The target, or the visual cue needed to identify it, is hidden by the current UI state and must be revealed through a state-changing interaction. • CT2-1. Visual Disambiguation: The target is visually similar to other elements and can only be distinguished through hover-based cues such as tooltips or labels, requiring a M OVE before the final C LICK. • CT2-2. Hidden State Activation: The target is contained in a collapsed or inactive UI region, such as a dropdown, tab, or side panel, that must be opened before clicking. • CT2-3. Removable Occlusion: A blocking element, such as a popup or modal, hides the target and must be dismissed before the target can be clicked. Constraint Type 3: Temporal & Manipulation Constraint. Reaching the target requires temporal control or spatial manipulation. • CT3-1. Delayed State Activation: The target is initially inactive and becomes interactable only after a temporal delay or asynchronous event, requiring a WAIT before clicking. • CT3-2. Dynamic Target Interaction: The target resides in an automatically transitioning element, such as a carousel or ticker, and must be reached by waiting for the correct state or by using navigation controls. • CT3-3. Movable Occlusion: A UI element occludes the target due to its position or size, requiring a D RAG to reposition or resize the occluding element. 3.3

Environments and task construction

We want environments that are deterministic enough to score a click programmatically, yet rich enough in interaction to make visibility recovery difficult. IGG comprises 601 tasks across 16 controllable environments modeled on widely used

Constraint Type 1

Constraint Type 2

Constraint Type 3

Scroll to reveal

Hover to reveal

Wait to activate

Environment: Document Editor Instruction: “Click the Monthly Partnership Check-in document”

Constraint: Off-screen

Environment: Community Chat

Environment: Video Platform

Instruction: “Find the Creator Dock server” Constraint: Hover-based disambiguation

Instruction: “Click the Skip Ad button” Constraint: Delayed activation

Figure 2: Examples of the three IGG constraint types: viewport, multi-state, and temporal or manipulation constraints. Additional examples appear in Appendix E.2.

applications such as Gmail, Google Calendar, VS Code, Figma, Spotify, and YouTube (full list in Appendix A).2 Each environment is a deterministic web application: every task starts from a fixed initial state, state transitions are reproducible, and every element carries a stable identifier, so the evaluator decides success at click time without heuristic matching. At the same time, the environments implement the interaction patterns that visibility recovery requires, including hover reveals, collapsible panels, popups, delayed activation, carousels, and draggable occluders. Tasks are built in three steps. A multimodal agent first drafts a goal-only instruction and a reference trace for each target. Annotators then rewrite instructions that leak the procedure (“Click ‘Reply’ on Minji’s message” rather than “Hover over the message and click ‘Reply’ ”, as in Figure A10) and add a natural disambiguating cue, such as an author, title, or surrounding content, when several elements share a label (Appendix C.3). Finally, annotators replay every trace to confirm that the target is unique and reachable. Figure 3 shows the resulting distribution over sub-types and environments.

2 Real applications are named as canonical references only, and no affiliation with or endorsement by their owners is implied.

Evaluation and Experiments

4.1

Evaluation setup

IGG combines task specifications, 16 controllable GUI environments, and a Playwright-driven Chromium evaluator at a 1440×900 viewport. On each CLICK, the evaluator compares the clicked element ID with the target ID; a match ends the episode, while a mismatch returns the updated screenshot. Models receive only screenshots, the goal-only instruction, and action history. The DOM, accessibility tree, target ID, and groundtruth interaction trajectory remain hidden. Each baseline is evaluated once per task after an environment reset; main runs use an eight-action budget and greedy decoding unless noted. Table D provides the prompts, decoding settings, and action formats. Metrics.

We report the following metrics:

• Grounding Accuracy (GA) ↑ : the fraction of tasks in which the agent clicks the target element within the action budget. • Action Cost (AC) ↓ : mean number of actions executed until success or budget exhaustion. • Reference Efficiency Ratio (RER) ↑ : mean of |gi |/|ai | over successful tasks, where gi is the annotated ground-truth trajectory and ai is the agent trajectory; values above 1 indicate trajec-

(a) Visibility Constraint Types

Table 2: Results by constraint type and sub-type.

Movable 60

Delayed 70

RER ↑

TT (ms) ↓

0.70 0.73 0.62 0.65

14718 15246 21985 56355

1.16 1.13 1.16 1.08

8773 10862 14682 49103

Constraint Type 2-1: Visual Disambiguation Qwen2.5-VL-7B 18.0 6.96 1.52 22.5 6.51 1.73 Qwen3-VL-8B GUI-Owl-1.5-8B 27.0 6.22 1.71 86.5 2.91 1.40 GPT-5.5

8048 11287 13350 44982

Constraint Type 2-2: Hidden State Activation Qwen2.5-VL-7B 9.6 7.44 0.93 26.0 6.54 0.97 Qwen3-VL-8B GUI-Owl-1.5-8B 23.1 6.67 0.87 GPT-5.5 87.5 3.73 0.91

9181 11251 17408 56739

Constraint Type 2-3: Removable Occlusion 18.9 7.13 0.81 Qwen2.5-VL-7B Qwen3-VL-8B 56.6 5.08 0.89 GUI-Owl-1.5-8B 54.7 4.96 0.95 GPT-5.5 98.1 3.02 0.89

9193 9384 11569 41038

Model Scroll 167

Dynamic 58

2. Multi-State Constraint (40.9%) Removable 53

Qwen2.5-VL-7B Qwen3-VL-8B GUI-Owl-1.5-8B GPT-5.5

10.8 15.0 15.0 86.2

7.37 7.26 7.27 4.29

Constraint Type 2: Multi-State Constraint Visual 89

Qwen2.5-VL-7B Qwen3-VL-8B GUI-Owl-1.5-8B GPT-5.5

Hidden 104

(b) Tasks per platform

Productivity

Spreadsheet

Email Client

Document Editor 35

Workspace Pages

Planner Dev & Design

Design Canvas 35

Developer Hub 32

Code Editor

AC ↓

Constraint Type 1: Viewport Constraint

1. Viewport Constraint (27.8%)

3. Temporal & Manipulation (31.3%)

GA (%) ↑

Social

Community Chat 34

Social Feed Media

14.6 31.3 31.3 89.4

7.20 6.21 6.14 3.28

Constraint Type 3: Temporal & Manipulation Constraint 50

Music Player

Qwen2.5-VL-7B Qwen3-VL-8B GUI-Owl-1.5-8B GPT-5.5

Streaming Service 34

Video Platform Lifestyle

News Portal

E-commerce

Maps

30 0

1-1 Scroll-based

20 30 Number of Tasks

2-1 Visual Disambiguation 2-2 Hidden State Activation 2-3 Removable Occlusion

3-1 Delayed State Activation 3-2 Dynamic Target Interaction 3-3 Movable Occlusion

Figure 3: Distribution of IGG tasks. (a) Task distribution across the three constraint types and their sub-types. (b) Number of tasks per platform, broken down by subtype.

42.6 44.1 46.8 60.1

6.02 6.08 5.78 4.82

0.52 0.63 0.73 0.97

24467 10907 11864 134642

Constraint Type 3-1: Delayed State Activation Qwen2.5-VL-7B 85.7 4.57 0.50 95.7 3.74 0.63 Qwen3-VL-8B GUI-Owl-1.5-8B 94.3 3.53 0.67 GPT-5.5 95.7 2.50 0.97

19820 6626 7412 20603

Constraint Type 3-2: Dynamic Target Interaction Qwen2.5-VL-7B 32.8 5.81 0.58 25.9 7.02 0.60 Qwen3-VL-8B GUI-Owl-1.5-8B 36.2 6.33 0.83 GPT-5.5 37.9 6.47 0.84

45702 12631 13558 134949

Constraint Type 3-3: Movable Occlusion Qwen2.5-VL-7B 1.7 7.90 1.00 Qwen3-VL-8B 1.7 7.90 1.00 1.7 7.88 2.00 GUI-Owl-1.5-8B GPT-5.5 40.0 5.93 1.10

9362 14235 15421 267392

Overall

tories shorter than the annotated ground-truth trajectory. • Total Time (TT) ↓ : mean elapsed wall-clock time per task. GA measures success; AC and RER measure interaction efficiency. TT is reported descriptively: local and API-backed models use different serving stacks, so their runtimes are not hardwarenormalized. 4.2

Main results

We compare Qwen2.5-VL-7B (Bai et al., 2025b), Qwen3-VL-8B (Bai et al., 2025a), and GUI-Owl1.5-8B (Xu et al., 2026) with the commercial reference model GPT-5.5.

Qwen2.5-VL-7B Qwen3-VL-8B GUI-Owl-1.5-8B GPT-5.5

22.3 30.8 31.6 79.4

6.88 6.46 6.34 4.04

0.72 0.85 0.89 0.92

15334 12094 15830 77876

Table 2 reports grounding accuracy (GA) and efficiency by constraint type and sub-type; Table 3 aggregates GA by application. Figure 4 compares humans and model baselines, and Appendix A10 provides extended per-application metrics. Figure 5 shows the corresponding accuracyefficiency trade-off. On the validation comparison, GPT-5.5 is substantially more accurate, but averages 66.5 seconds per task compared with 14.2 seconds for the trained Qwen3-VL and requires paid API inference. Evaluating GPT-5.5 over the

Human (94.6%) GPT-5.5 (79.4%)

Qwen3-VL (30.8%)

(a) Humans, GPT-5.5, and open-source VLMs.

GPT-5.5 (79.4%) GPT-5.4-mini (72.4%) Qwen3-VL (30.8%)

GUI-Owl 1.5 (31.6%) Qwen2.5-VL (22.3%)

(b) Additional model baselines.

Figure 4: IGG exposes a large interaction gap. Human results summarize the solvability study, whereas model results span the complete benchmark. Table 3: Grounding Accuracy (GA) (%) by application. App

Ours (59.2%)

Qwen2.5-VL Qwen3-VL GUI-Owl-1.5 GPT-5.5

Social Feed Planner Developer Hub Code Editor Community Chat Document Editor Design Canvas Spreadsheet Email Client News Portal Workspace Pages Streaming Service E-commerce Music Player Maps Video Platform

23.5 15.2 11.4 25.0 22.4 32.5 8.9 16.7 47.6 9.7 25.7 7.7 16.7 20.0 16.7 55.9

29.4 21.2 25.7 21.9 26.5 37.5 8.9 33.3 50.0 29.0 34.3 20.5 43.3 36.0 26.7 50.0

38.2 24.2 25.7 28.1 34.7 42.5 13.3 23.8 54.8 29.0 22.9 17.9 33.3 32.0 26.7 58.8

61.8 81.8 57.1 87.5 61.2 85.0 82.2 83.3 88.1 71.0 85.7 87.2 86.7 86.0 70.0 94.1

Overall (601)

22.3

30.8

31.6

79.4

Table 4: Grounding accuracy (%) when a model solves the complete task, is told the required interaction, or receives the interface after the reveal interaction has been performed. Condition Full task Interaction specified Reveal performed

GPT-5.4 Qwen3-VL Mini (Low) 30.8 33.9 81.9

72.4 – 90.4

complete benchmark cost approximately $74.9. 4.3

GPT-5.5 (80.0%) GPT-5.4-mini (66.7%)

Where does interactive grounding fail?

We intervene at two points in the four-stage decomposition from Section 3.1. In the first condition, the model is told which reveal interaction is required but must determine its location and parameters. In the second, we perform the ground-truth reveal trajectory and ask the model only for the final click.

Figure 5: Accuracy-efficiency trade-off on the validation set. GPT-5.5 offers substantially higher accuracy, while the trained Qwen3-VL is faster and runs without paid API inference.

Naming the interaction changes Qwen3-VL by only 3.1 points, whereas completing it raises accuracy by 51.1 points. Providing the correct revealaction category is therefore insufficient. Most remaining errors arise from choosing its location or parameters and executing the reveal correctly, before target recognition and final-click localization are evaluated. Target-agnostic heuristics further separate easy and hard reveal patterns (Appendix F.3). Fixed scrolling, hovering, and waiting reveal 68.8%, 61.4%, and 87.1% of their matched tasks, whereas popup dismissal and occluder dragging reach 17.6% and 16.9%. The difficult cases require identifying which element controls visibility and how to manipulate it.

Failed Qwen3-VL trajectories show the same pattern. When the ground-truth trajectory requires D RAG, S CROLL, or M OVE, failed runs omit that action in 59/59, 207/247, and 71/96 cases, respectively (Appendix F.2). This motivates first expanding the policy’s action support with SFT, then improving execution with online feedback. 4.4

Table 5: Human task solvability on the 21-task suite (15 participants, 315 trials). Sub-type Trials Success (%) ↑ Time (s) ↓

Learning missing interaction priors

We ask whether training can supply reveal actions that base VLMs rarely use. D RAG provides the clearest case: Qwen2.5-VL, Qwen3-VL, and GUIOwl-1.5 emit no drag actions on the held-out tasks. Setup. After excluding two tasks whose groundtruth trajectories could not be replayed reliably, we use 479 tasks for training and reserve 120 for evaluation. The held-out tasks contribute neither SFT examples nor GRPO rollouts. Supervised training on successful trajectories (Hu et al., 2022). We fine-tune the model on human-verified successful trajectories. Let xt denote the instruction, screenshot, and interaction history at step t, and let at be the next ground-truth action. We optimize a weighted next-action negative log-likelihood (Pomerleau, 1991): LSFT (θ) = −E(xt ,at )∼D [w(at ) log πθ (at | xt )] . For standard behavior cloning, w(at ) = 1. In the frequency-balanced variant (Cui et al., 2019), we increase w(at ) for rare reveal actions such as M OVE, WAIT, and D RAG; details are in Appendix G. The model still receives goal-only instructions. Only the training distribution changes. Refining interactions with online feedback (Shao et al., 2024). We then apply online GRPO under the same eight-action interaction budget as main evaluation. The reward is defined over browserexecuted trajectories rather than text strings: R(τ ) = Rout (τ ) + Rproc (τ ) − λstep |τ | − Rerr (τ ). Rout rewards click-time target success, Rproc gives bounded credit for visibility-changing progress, and Rerr penalizes wrong clicks, empty clicks, invalid actions, and repeated ineffective actions. Exact weights and optimization hyperparameters are provided in Appendix G. The two stages serve different roles. SFT broadens action support, after which GRPO selects among those actions using online feedback. If a

CT1-1 CT2-1 CT2-2 CT2-3 CT3-1 CT3-2 CT3-3

45 45 45 45 45 45 45

93.3 100.0 95.6 88.9 97.8 97.8 88.9

23.3 15.9 20.0 21.0 14.2 22.9 21.1

Overall

315

94.6

19.8

Table 6: Human fidelity ratings averaged over 15 participants on a five-point Likert scale: visual realism (VR), interaction realism (IR), and task executability (TE). Environment

VR ↑

IR ↑

TE ↑

Music Player Video Platform E-commerce Developer Hub Code Editor News Portal Maps

4.93 4.73 4.67 4.67 4.67 3.93 3.93

4.80 4.87 4.80 4.53 4.80 4.60 4.07

5.00 4.80 4.80 4.80 4.87 4.60 4.40

Average

4.50

4.64

4.75

seed policy never samples D RAG, GRPO observes no successful drag rollout to reinforce. Frequencybalanced SFT brings rare reveal actions into the policy’s sampled support, enabling GRPO to optimize their selection and execution. Results. The two-stage procedure raises Qwen3VL from 40.0% to 59.2% GA (Figure 6). On Movable Occlusion, frequency-balanced SFT emits D RAG in 7/8 required-drag tasks and solves 4/8; GRPO raises this to 5/8. Initialization also matters on Dynamic Target Interaction: GRPO reaches 60.0% from the frequency-balanced seed but 30.0% from the standard SFT seed. The same procedure improves InternVL3.5 from 28.3% to 47.5% and GUI-Owl-1.5 from 35.0% to 52.5% (Table A8), showing consistent gains across three architectures. 4.5

Human solvability and environment fidelity

A failure on IGG is informative only if the task is solvable and the environment is faithful to the interface it models. Before comparing models, we therefore verify both with the same 15 participants (protocol details in Appendix H). Task solvability. Participants solve a fixed suite of 21 tasks, three from each sub-type, using the

Qwen3-VL (40.0%) SFT (54.2%)

SFT+GRPO (59.2%)

Share of actions (%)

Baselines

Ours

Qwen2.5-VL

GUI-Owl

(a) Per-sub-type gains from baseline to SFT and SFT+GRPO.

SFT

Qwen3-VL

move

scroll

drag

wait

SFT + GRPO

(b) Action-prior shift across baselines, SFT, and SFT+GRPO.

Figure 6: SFT expands action support, and GRPO selects among those actions. The best trained model improves Qwen3-VL from 40.0% to 59.2% GA on the 120-task validation split. Baseline VLMs solve some validation tasks without emitting every action in the ground-truth trajectory, but emit no drag actions. SFT introduces these actions into the policy’s support, after which GRPO improves task success.

Share of actions (%)

Baselines

Naive GRPO

Prior-balanced GRPO

Qwen2.5 GUI-Owl

Qwen3

N-005 move

N-010

N-015 scroll

N-019 drag

PB-005

PB-010

PB-015

PB-019

Grounding Accuracy (%)

Baselines

Ours

70 60

51.7

50 40

54.2

55.0

59.2

56.7

57.5

update 15

update 19

+19.2 pt

40.0

30 20 10 0

wait

(a) Action share across GRPO variants.

Qwen3-VL

SFT naive SFT balanced update 5

update 10

(b) Validation GA across GRPO checkpoints.

Figure 7: GRPO ablation. GRPO is effective only when initialized from the frequency-balanced SFT seed: it reaches 59.2% GA, while GRPO from the standard SFT seed does not improve over its seed model. The frequencybalanced seed gives the policy access to rare reveal actions, after which online reward improves their selection and execution.

same instructions and interfaces given to the models, without the constraint label, target ID, target coordinates, or reveal procedure. A trial ends on the correct target click or after eight click-or-drag attempts. Scrolling and hovering are recorded but do not count, so the limit is comparable to, though not identical with, the eight-action model budget. Participants succeed in 298 of 315 trials (94.6%) with a mean completion time of 19.8 seconds (Table 5). Together with the replay verification of every reference trace, this indicates that the model failures reported below do not stem from unsolvable tasks or ambiguous instructions. Environment fidelity. The same participants compare seven representative environments with their real-interface counterparts and rate visual realism, interaction realism, and task executability on five-point scales, giving mean scores of 4.50, 4.64, and 4.75 (Table 6). Appendix C.2 comple-

ments these ratings with visual-similarity metrics and a preliminary paired mock–real evaluation on 16 tasks, one per environment, in which model ordering is preserved across the two settings and the Qwen3-VL model trained in Section 4.4 leads in both.

Conclusion

We introduced IGG to make visibility recovery a measurable component of GUI grounding. Our analyses show that knowing which interaction is required is not enough; current models most often fail in deciding where and how to carry it out. Training on successful interaction trajectories followed by online feedback improves this capability consistently across model families. IGG therefore provides a focused setting for evaluating and learning the interactions that must precede a grounding click.

Limitations

IGG uses deterministic mockups for controlled and reproducible evaluation. Our fidelity analyses, including paired mock-real evaluation, suggest that these environments capture relevant visual and interaction patterns, but they do not replace evaluation on live interfaces. Interactive grounding can support accessibility and UI testing, but the same capability may also be misused for automated interaction. We therefore release only controlled mockups that are not connected to external services.

Acknowledgments This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No. RS-2026-25518317 (50%), Development of AI memory mechanism that reflects human cognitive principles; No. RS2024-00457882 (50%), National AI Research Lab Project), InnoCORE program of the Ministry of Science and ICT(N10250156), and Samsung Electronics Co., Ltd (Project Code: IO260114-1516101).

References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others. 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. Preprint, arXiv:2502.13923. Stefano Burigat, Luca Chittaro, and Andrea Vianello. 2012. Dynamic visualization of large numbers of off-screen objects on mobile devices: an experimental comparison of wedge and overview+ detail. In Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services, pages 93–102. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. 2021. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9650–9660.

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332. Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie. 2019. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268–9277. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114. Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. 2023. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, volume 36, pages 50742–50768. Yuan Gao, Kunyu Shi, Pengkai Zhu, Edouard Belval, Oren Nuriel, Srikar Appalaraju, Shabnam Ghadar, Zhuowen Tu, Vijay Mahadevan, and Stefano Soatto. 2024. Enhancing vision-language pre-training with rich supervisions. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13480–13491. IEEE. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning Representations. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 881–905. Shrinidhi Kumbhar, Haofu Liao, Srikar Appalaraju, and Kunwar Yashraj Singh. 2026. Towards gui agents: Vision-language diffusion models for gui grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 27493–27502. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng

Chua. 2025. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786. Himangi Mittal, Gaurav Mittal, Nelson Daniel Troncoso, and Yu Hu. 2026. See, point, refine: Multiturn approach to gui grounding with visual feedback. arXiv preprint arXiv:2604.13019. Dean A Pomerleau. 1991. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88–97. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865. Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, and 1 others. 2025. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143. Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, and 1 others. 2025. Scaling computer-use grounding via user interface decomposition and synthesis. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094. Haiyang Xu, Xi Zhang, Haowei Liu, Junyang Wang, Zhaozai Zhu, Shengjie Zhou, Xuhao Hu, Feiyu Gao, Junjie Cao, Zihua Wang, and 1 others. 2026. Mobileagent-v3. 5: Multi-platform fundamental gui agents. arXiv preprint arXiv:2602.16855. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. Gpt-4v(ision) is a generalist web agent, if grounded. In Forty-first International Conference on Machine Learning.

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Aman Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations. Xiyue Zhu, Peng Tang, Haofu Liao, and Srikar Appalaraju. 2025. Turbocharging web automation: The impact of compressed history states. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3644–3651. Association for Computational Linguistics.

Mockup Applications

Table A2: Grounding accuracy (%) at five viewport sizes.

IGG comprises 16 controllable applications spanning common GUI domains. Each environment combines deterministic state transitions and stable target identities with interaction patterns modeled on a familiar interface. Table A1 lists the references, and Figures A1-A5 show representative states. Table A1: The 16 mockup environments and their reference interfaces. Mockup

Reference

Twitter/X Google Calendar GitHub VS Code Web Discord Google Docs Figma Google Sheets Gmail Google Notion Netflix Walmart Spotify Google Maps YouTube

Design Choices

B.1

Viewport robustness

The main benchmark fixes the viewport at 1440 × 900 so that initial target visibility and constraint type remain constant across models. Responsive layouts can change whether a target is hidden and which interaction reveals it, so the viewport is part of the task specification. To test whether the learned behavior depends on this layout, we regenerate the 601 tasks at five viewport sizes corresponding to desktop, mobile portrait, tablet, split-screen, and zoomed displays, and keep the 127 tasks that remain valid in every layout (i.e., the target is initially hidden and the reference trace replays successfully). The subset contains 103 training and 24 held-out tasks, so the study measures whether behavior learned at 1440× 900 survives a layout change rather than held-out generalization. On this subset, the trained Qwen3VL model outperforms its base model by 11.1-17.3 points at every viewport, including mobile portrait (Table A2).

Viewport

Base Trained Gain

1440 × 900 390 × 844 1024 × 768 720 × 900 1152 × 720

43.3 43.3 42.5 48.0 39.4

59.8 60.6 55.9 59.1 52.8

+16.5 +17.3 +13.4 +11.1 +13.4

Additional Results and Analyses

C.1

Per-environment results

Table A10 breaks the main results down by environment. Open-source models trail GPT-5.5 in every environment. Design Canvas is the hardest environment for open-source models (8.9–13.3% GA) and Video Platform and Email Client the easiest, whereas GPT-5.5 is weakest on Developer Hub, Community Chat, and Social Feed. C.2

Mockup fidelity

Each task starts from a resettable state, executes deterministic transitions, and uses a stable target ID for click-time evaluation. The environments implement the visibility-changing interactions measured by IGG, including hidden panels, hover reveals, popups, delayed elements, carousels, and movable occlusions. We evaluate their fidelity to real-interface counterparts using visual similarity, human ratings, and paired task execution. Visual similarity. We measure visual similarity between each mockup and its corresponding real-application reference listed in Table A1. We compare each matched mock-real pair against non-matched pairs formed by pairing the mockup with the other 15 real-application references. We compute feature-space cosine similarities using DINO (Caron et al., 2021) and SigLIP (Zhai et al., 2023), and perceptual distance using DreamSim (Fu et al., 2023). As shown in Table A3, matched pairs are substantially more similar than non-matched pairs, with higher DINO similarity (0.7724 vs. 0.5442), higher SigLIP similarity (0.8094 vs. 0.6211), and lower DreamSim distance (0.2529 vs. 0.5229). These results indicate that the mockups preserve application-level visual structure beyond generic webpage appearance. Human fidelity. Participants compare seven environments with their real-interface references and rate visual realism, interaction realism, and task ex-

(a) Planner (Google Calendar)

(b) Document Editor (Google Docs)

(d) Workspace Pages (Notion)

(e) Email Client (Gmail)

Figure A1: Representative states from productivity mockup applications.

(a) Code Editor (VS Code Web)

(b) Design Canvas (Figma)

Figure A2: Representative states from development and design mockup applications.

(a) Community Chat (Discord)

(b) Social Feed (Twitter/X)

Figure A3: Representative states from social mockup applications.

(a) Music Player (Spotify)

(b) Streaming Service (Netflix)

Figure A4: Representative states from media mockup applications.

(a) News Portal (Google)

(b) E-commerce (Walmart)

Figure A5: Representative states from lifestyle mockup applications.

Table A3: Visual similarity between mockups and realapplication references. DINO and SigLIP are cosine similarities, where higher is better. DreamSim is a perceptual distance, where lower is better. Pairing

Count DINO ↑ SigLIP ↑ DreamSim ↓

Matched Non-matched

16 240

0.7724 0.5442

0.8094 0.6211

0.2529 0.5229

Table A4: Task-level fidelity on 16 paired mock-real tasks. Succ. denotes the number of successful trials out of 16. Model

Env.

Succ.

GA ↑

AC ↓

RER ↑

Qwen2.5-VL

Mock Real

3/16 3/16

18.8 18.8

7.25 7.69

0.45 0.42

Qwen3-VL

Mock Real

4/16 5/16

25.0 31.3

7.06 7.06

0.54 0.35

GUI-Owl-1.5

Mock Real

6/16 6/16

37.5 37.5

6.13 6.88

0.55 0.57

Qwen3 SFT+GRPO

Mock Real

9/16 9/16

56.3 56.3

5.06 5.00

0.72 0.70

ecutability. The three mean ratings are 4.50, 4.64, and 4.75 out of 5 (Table 6). Task-level mock-real fidelity. We run four models on 16 paired mock and real-interface tasks. Model ordering is preserved: Qwen3-VL SFT+GRPO leads in both settings with 56.3% GA, followed by GUI-Owl-1.5, Qwen3-VL, and Qwen2.5-VL (Table A4). Across the four model families, mock and real GA are strongly consistent (Pearson r = 0.98; Spearman ρ = 1.00). The paired set contains one task per application. Figures A6 and A7 show representative outcomes. C.3

Instruction design

Instructions name the target but not the reveal procedure. Annotators avoid action hints such as “scroll down,” “hover over the icon,” or “wait until the button appears,” and add a natural cue, such as an author, document title, or surrounding content, when several elements share a label. For example, “Scroll down and click Kai” becomes “Find Kai in the direct message list,” and “Drag the sticky note away and click QA Checklist” becomes “Click the QA Checklist note without deleting any notes.” Multimodal LLMs propose candidate instructions and reference traces, and annotators correct the wording, repair traces where needed, and replay every task to verify that the target is unique and reachable.

Effect of procedural hints. As a diagnostic, we also evaluate instructions that state the intermediate procedure. Unlike the reveal action given condition of Section 4.3, which names only the action type, these hints describe the procedure. With such hints, Qwen3-VL rises from 30.8% to 56.9% GA over all 601 tasks, but hints do not solve precise manipulation: hinted Qwen3-VL solves 0 of 8 heldout movable-occlusion tasks, whereas SFT+GRPO solves 5 of 8 without any test-time hint. The main benchmark therefore retains goal-only instructions, and the training gains reflect an executable interaction prior rather than instruction following.

Evaluation Protocol

The open-source baselines are Qwen/Qwen2.5-VL-7B-Instruct, Qwen/Qwen3-VL-8B-Instruct, and mPLUG/GUI-Owl-1.5-8B-Instruct, and the commercial baseline is gpt-5.5. Each model is evaluated once per task with greedy decoding from a reset environment. At each turn, the agent receives the system prompt, task instruction, screenshot, step index, coordinate space, and action history. It returns one JSON action. Figure A8 shows the prompt, and the shared action schema is: {"type": "scroll", "x": 720, "y": 480, "dy": 50} {"type": "move", "x": 512, "y": 340} {"type": "drag", "startX": 10, "startY": 20, " endX": 80, "endY": 100} {"type": "wait", "ms": 1500} {"type": "click", "x": 512, "y": 340}

A click succeeds only when the clicked element ID matches the annotated target ID. Otherwise the environment applies any defined state transition and returns the next screenshot. Navigation outside the task environment is disabled, and the episode ends on success or budget exhaustion. The drag schema (startX, startY, endX, endY) is identical in the prompt, the evaluator, and Table 1.

Dataset Examples

E.1

Task specification format

Each task record contains a goal-only instruction, a target ID, an initial state, and one successful reference trace (Listing 1). The evaluator keeps the target ID hidden from the model and uses the trace only to compute interaction-efficiency metrics. Reference traces may refer to elements by stable selectors, which the replay tool resolves to the center

(a) Qwen2.5-VL

(b) Qwen3-VL

Figure A6: Paired examples in which the model fails in both the controlled and real-interface task.

(a) GUI-Owl-1.5

(b) Qwen3-VL SFT+GRPO

Figure A7: Paired examples in which the model succeeds in both the controlled and real-interface task.

You are an Interactive GUI Grounding agent. Your ONLY goal is to locate the single UI element described in the instruction and click on it. The element may NOT be visible in the current screenshot. In that case, take the minimum interaction needed to reveal it (scroll, hover, expand, wait, drag away an occluder, etc.) and then click the element. This is a grounding task, not a navigation task. Do NOT try to complete any higher-level workflow; just find the target and click it. ... Action space (emit exactly ONE action per turn): - Scroll a region: {"type": "scroll", "x": <int>, "y": <int>, "dy": <int>} dy > 0 scrolls down, dy < 0 scrolls up. Use (x, y) to pick which scroll container gets the event (e.g. a left sidebar vs the main feed). - Move the cursor: {"type": "move", "x": <int>, "y": <int>} Triggers hover effects, tooltips, and hover-reveal menus. - Drag: {"type": "drag", "startX": <int>, "startY": <int>, "endX": <int>, "endY": <int>} Use to nudge an occluding popup aside without fully closing it, or to move sliders/panels. - Wait: {"type": "wait", "ms": <int>} Use when the target is gated by a timer, async load, ad countdown, auto-rotating carousel, etc. - Click: {"type": "click", "x": <int>, "y": <int>} Grounding trigger. If (x, y) lands on the target element, the task ends immediately with success. If it lands on anything else, the task continues and the screen state may change. Strategy: 1. Read the instruction. Identify a unique visual description of the target. 2. Scan the screenshot. If the target is visible, click its center. 3. If not visible, reason about why: is it offscreen, in a collapsed panel, hidden behind a popup, revealed only on hover, or not loaded yet? 4. Pick the single cheapest action that could make it visible or confirm it. 5. Avoid exploratory clicks on random UI elements; each click is evaluated. Output format (STRICT): Reply with exactly ONE line containing a single JSON object for the next action. No prose, no markdown, no code fences, no explanation. ALL coordinates are SEPARATE integer keys, NEVER a list and NEVER a nested object. ...

Figure A8: Excerpt of the system prompt used for IGG model evaluation. Ellipses indicate omitted formatting examples.

of the element’s current bounding box, whereas models must emit screenshot coordinates. Listing 1: Task specification example. The task targets “Kai” in the direct message list (Community Chat application, CT1-1 : Scroll-based Exploration). The groundtruth trajectory consists of a scroll action followed by a click on the revealed target.

Failure Case Analysis

F.1

Representative cases

Across Qwen2.5-VL, Qwen3-VL, and GUI-Owl1.5, shared failures often omit the action that would reveal the target. Table A5 summarizes three patterns, and Figure A16 shows representative traces.

{ "id": "messaging-01-find-kai-direct-messagelist", "sub_type": "1-1", "instruction": "Find ‘Kai’ in the direct message list.", "target_id": "messaging.dm.kai", "initial_state_id": "1", "ground-truth trajectory": [ {"action": "SCROLL", "dy": 1000}, {"action": "CLICK", "selector": "messaging.dm .kai"} ]

F.2

Trace-derived action-prior diagnostic

The representative failure cases in Section F.1 show agents spending the budget without the needed reveal action. For each failed Qwen3-VL trajectory, we compare the model actions with the groundtruth trajectory and record whether each required reveal action appears at least once. F.3

Target-agnostic reveal heuristics

}

E.2

Sub-type examples

Figures A9–A15 show one representative task for each IGG sub-type. Each example pairs the initial state, where the target is not directly actionable, with the final state at the time of the successful target click.

To separate reveal patterns that a fixed rule can solve from those that require target-specific decisions, we evaluate five target-agnostic heuristics, each on the sub-type it is designed for: scrolling the page center in fixed steps, hovering over common icon regions in order, clicking common dismiss locations, waiting in one-second steps, and dragging visible occluders toward the screen edge. A heuristic succeeds when the target becomes visible,

(a) Initial state

(b) Final state at click time

Figure A9: CT1-1 : Scroll-based Exploration (E-commerce). “Click ‘Cart’ for ‘Wireless Mechanical Keyboard’.”

(a) Initial state

(b) Final state at click time

Figure A10: CT2-1 : Visual Disambiguation (Community Chat). “Click ‘Reply’ on Minji’s message.”

(a) Initial state

(b) Final state at click time

Figure A11: CT2-2 : Hidden State Activation (Document Editor). “Click ‘Rename’ on ‘Product Research Weekly Brief’.”

(a) Initial state

(b) Final state at click time

Figure A12: CT2-3 : Removable Occlusion (Video Platform). “Click the ‘Late-night Studio Vlog’ video card.”

(a) Initial state

(b) Final state at click time

Figure A13: CT3-1 : Delayed State Activation (Community Chat). “Click ‘Send’ for the pptx upload.”

(a) Initial state

(b) Final state at click time

Figure A14: CT3-2 : Dynamic Target Interaction (News Portal). “Click ‘Pre-Order Now’ in the ‘NexPhone’ ad.”

(a) Initial state

(b) Final state at click time

Figure A15: CT3-3 : Movable Occlusion (Design Canvas). “Click the ‘QA Checklist’ note without deleting any.”

Table A5: Representative failure cases selected from tasks failed by Qwen2.5-VL-7B, Qwen3-VL-8B, and GUIOwl-1.5-8B. Failure mode

Sub-type

Task

Observed trace pattern

Spatial manipulation CT3-3 : Movable Design Canvas: find the QA Checklist All three models exhausted the budget is not attempted Occlusion note without changing zoom. without using a drag operation. Qwen2.5VL-7B and Qwen3-VL-8B produced eight scrolls; GUI-Owl-1.5-8B produced one wait followed by seven scrolls. Dynamic target state CT3-2 : Dynamic News Portal: find the Glass Harbor The models interacted with the wrong is not tracked Target Interaction begins a 24-hour pilot headline in state of the ticker. Qwen3-VL-8B clicked Breaking News. the ticker controls and ended on a different headline, while the other models scrolled or expanded unrelated page content. Hover-based ambigu- CT2-1 : Visual Community Chat: find the Creator The target server is identifiable only ity is not resolved Disambiguation Dock server. after hovering over ambiguous server icons. Qwen2.5-VL-7B repeatedly scrolled, while Qwen3-VL-8B and GUIOwl-1.5-8B repeatedly clicked icons without first revealing their labels.

Table A6: Trace-derived action-prior diagnostic for Qwen3-VL failed trajectories. Counts are non-exclusive because one task can require multiple setup actions. Required setup action Failed tasks Omitted D RAG S CROLL M OVE

59 247 96

WAIT

Rate Diagnostic interpretation

59 100.0% Drag is absent from failed traces. 207 83.8% Scroll is often replaced by clicks or local moves. 71 74.0% Hover and disambiguation moves are often skipped. 1 33.3% Few failed tasks require explicit waiting.

instruction: Find the QA Checklist note without changing zoom.

instruction: Find 'Glass Harbor begins a 24-hour pilot …

Figure A16: Representative failed traces. The figure visualizes example failed traces from two representative cases, showing how models spend the interaction budget without performing the visibility-changing action needed to reveal the target.

unobstructed, and actionable (Table A7). Fixed rules often resolve scrolling, hovering, and waiting, but rarely resolve removable or movable occlusions. This agrees with the trace analysis in Section F.2, where D RAG is absent from every failed trajectory that requires it.

imum image pixels 1,382,400, and 500 steps with checkpoints every 100 steps. Frequency balancing oversamples C LICK/S CROLL/M OVE/WAIT/D RAG by 1/1/2/2/4, expanding the data to 1,460 examples and the drag examples from 55 to 220.

GRPO starts from an SFT LoRA seed and collects online browser rollouts from the 479 training tasks under the same eight-action budget used in evaluation. We use group size 6, temperature 1.0, top-p 0.95 for rollout sampling, learning rate 2 × 10−6 , PPO-style clipping with ϵ = 0.2, maximum gradient norm 1.0, and up to 20 update groups. We evaluate saved LoRA checkpoints with greedy decoding on the fixed 120-task validation split.

Training Details

Our two-stage procedure uses SFT to introduce rare reveal actions and online GRPO to optimize their execution from browser feedback. G.1

Training Setup

Two tasks whose ground-truth trajectories could not be replayed reliably during corpus construction are excluded before splitting the remaining 599 tasks into 479 training tasks and 120 held-out validation tasks. Every included ground-truth trajectory is human-corrected and verified by replay; validation tasks contribute neither SFT examples nor GRPO rollouts. G.2

SFT

The ground-truth trajectories yield 1,141 actionlevel examples with normalized coordinates. We use LoRA rank 8, alpha 32, dropout 0.05, all-linear language-model target modules, a frozen vision tower and projector, bf16, learning rate 1 × 10−4 , effective batch size 8, maximum length 8192, max-

G.3

Online GRPO

Online interaction reward. The reward is computed from browser-executed traces. Let S denote target-click success; T the action count; W nonempty wrong clicks; E empty clicks; I invalid actions; A repeated ineffective actions; and R repeated wrong-click locations. The bounded bonus B rewards reveal progress (Table A9). Successful traces with prior wrong clicks are capped at 1.2, shortcut successes at 0.8, and runtime or parsing failures are excluded from policy updates.

Table A7: Reveal success of target-agnostic interaction rules on their corresponding constraint sub-types.

G.4

Constraint sub-type

Interaction rule

CT1-1 : Scroll-based Exploration CT2-1 : Visual Disambiguation CT2-3 : Removable Occlusion CT3-1 : Delayed State Activation CT3-3 : Movable Occlusion

Scroll center by fixed steps Hover common icon areas Click common dismiss areas Wait in 1-s steps Drag visible occluder aside

Cross-architecture results

We apply the same recipe, with the same data, budget, and hyperparameters, to InternVL3.5-8B and GUI-Owl-1.5-8B. All three architectures improve by a similar margin: Qwen3-VL from 40.0% to 59.2%, InternVL3.5 from 28.3% to 47.5%, and GUI-Owl-1.5 from 35.0% to 52.5% (Table A8). The models differ in coordinate tokenization and pretraining, so the gain is not specific to one model family. Table A8: Grounding accuracy before and after SFT+GRPO for three model architectures. Model

Base

SFT+GRPO

Qwen3-VL-8B InternVL3.5-8B GUI-Owl-1.5-8B

40.0 28.3 35.0

59.2 47.5 52.5

Human Study Details

H.1

Study setup

This appendix provides implementation details for the two human studies summarized in Section 4.5. Fifteen participants completed two browser-based protocols connected to the same environments as model evaluation. In the solvability study, every participant completed a subtype-balanced suite of 21 tasks—three from each of the seven sub-types and at least one from every environment—yielding 315 trials. In the fidelity study, participants compared seven controlled environments with realinterface references and rated visual realism, task executability, and interaction realism on five-point scales. H.2

Task-solvability interface

Participants receive one instruction and interact directly with the browser. The interface ends a trial when the participant clicks the target or reaches eight click or drag attempts. It records scroll sessions after an 800-ms debounce for interaction

Reveal success (%) 68.8 61.4 17.6 87.1 16.9

counts and records hover events without applying them to the attempt limit (Figure A17). H.3

Fidelity-test interface

For each paired example, participants see the instruction, controlled environment, and realinterface reference, then rate visual realism, task executability, and interaction realism (Figure A18).

Table A9: GRPO reward components used in the diagnostic training experiments. Positive progress bonuses are bounded per rollout and are applied only when the browser state or target evidence changes in the intended direction. Component

Symbol Value / rule

Target-click success S Unsuccessful rollout ¬S Step cost T Wrong non-empty click W Empty click E Invalid action I Repeated ineffective action A Repeated wrong-click location R Hover / move progress B Hidden-state progress B Drag progress B Wait progress B Clean visible-target click B Reward clipping –

+2.2 if successful −1.0 base term −0.03 per action −0.65 per event −1.00 per event −1.00 per event −0.20 per event −0.25 per event +0.45 +0.45 +0.75; 0.7× if no state change +0.35; 0.5× if no delayed change +0.20 [−4.0, 2.8]

Table A10: Per-application GA, AC, RER, and TT for Qwen3-VL and GPT-5.5 on all 601 tasks. Qwen3-VL App

AC RER

Social Feed 29.4 6.47 Planner 21.2 6.88 Developer Hub 25.7 6.94 Code Editor 21.9 6.81 Community Chat 26.5 6.82 Document Editor 37.5 6.05 Design Canvas 8.9 7.44 Spreadsheet 33.3 6.24 Email Client 50.0 5.38 News Portal 29.0 6.55 Workspace Pages 34.3 6.40 Streaming Service 20.5 7.00 E-commerce 43.3 6.40 Music Player 36.0 5.80 Maps 26.7 6.47 Video Platform 50.0 5.91

GPT-5.5 TT GA

AC RER

0.93 17514 61.8 4.59 0.79 13098 81.8 3.88 0.66 9763 57.1 5.57 1.26 9575 87.5 3.88 0.64 9357 61.2 4.82 0.99 8451 85.0 3.48 1.25 13951 82.2 4.33 0.84 8920 83.3 2.93 0.75 10493 88.1 3.38 0.80 19204 71.0 4.77 0.86 8830 85.7 3.71 0.67 17224 87.2 4.05 0.60 17047 86.7 4.40 1.15 12580 86.0 3.60 1.25 11972 70.0 4.53 0.63 8158 94.1 3.15

1.04 69757 0.83 61011 0.69 138919 0.97 59747 1.03 108102 1.04 63538 0.94 93199 1.04 124023 0.97 63120 0.80 52984 0.93 30489 0.71 52801 0.72 49700 1.04 63597 0.95 166930 0.88 38354

(a) Start page

(d) Task complete (success)

(b) Task instruction

(e) Interaction limit reached (failure)

Figure A17: The web-based interface used in the human study. (a) Participants begin by entering an ID and reviewing the task format. (b) Each task is introduced with the instruction text. (c) Participants then interact with the live mockup; the interaction counter is shown in the top-right corner. (d) The task ends with success when the participant clicks the correct target, or (e) with failure when the eight click-or-drag attempt limit is reached.

(a) Start page

(b) Rating interface

Figure A18: The web-based interface used in the human fidelity test. (a) Participants enter the fidelity-study session. (b) For each application, participants compare the mock interface with the real-application reference and rate visual realism, task executability, and interaction realism.

IGG: A Benchmark for Interactive GUI Grounding under Visibility Constraints

Analyze & explore related research in Conceptio

Related documents