FineState-Bench: Benchmarking State-Conditioned Grounding for Fine-grained GUI State Setting Fengxian Ji1,2∗ , Jingpu Yang2∗ , Zirui Song1∗ , Yuanxi Wang2 , Zhexuan Cui2 , Yuke Li2 , Qian Jiang2 , Xiuying Chen1† 1 MBZUAI, United Arab Emirates 2 Northeastern University, China {fengxian.ji, zirui.song, xiuying.chen}@mbzuai.ac.ae {jingpuyang290, yuanxiwang89}@gmail.com [email protected],[email protected],[email protected] Abstract
arXiv:2604.27974v1 [cs.CV] 30 Apr 2026
Despite the rapid progress of large visionlanguage models (LVLMs), fine-grained, stateconditioned GUI interaction remains challenging. Current evaluations offer limited coverage, imprecise target-state definitions, and an overreliance on final-task success, obscuring where and why agents fail. To address this gap, we introduce FineState-Bench, a benchmark that evaluates whether an agent can correctly ground an instruction to the intended UI control and reach the exact target state. FineState-Bench comprises 2,209 instances across desktop, web, and mobile platforms, spanning four interaction families and 23 UI component types, with each instance explicitly specifying an exact target state for fine-grained state setting. We further propose FineState-Metrics, a four-stage diagnostic pipeline with stage-wise success rates: Localization Success Rate (SR@Loc), Interaction Success Rate (SR@Int), Exact State Success Rate at Locate (ES-SR@Loc), and Exact State Success Rate at Interact (ES-SR@Int), and a plug-and-play Visual Diagnostic Assistant (VDA) that generates a Description and a bounding-box Localization Hint to diagnose visual grounding reason via controlled w/ vs. w/o comparisons. On FineState-Bench, exact goal-state success remains low: ES-SR@Int peaks at 32.8% on Web and 22.8% on average across platforms. With VDA localization hints, Gemini-2.5-Flash gains +14.9 ESSR@Int points, suggesting substantial headroom from improved visual grounding, yet overall accuracy is still insufficient for reliable fine-grained state-conditioned interaction Github.
1
Introduction
Recent advances in LVLMs have enabled a new class of GUI agents that can execute natural∗ †
These authors contributed equally. Corresponding author.
language instructions on real-world software interfaces (Nguyen, 2024; Wen et al., 2024). By integrating visual understanding with languageconditioned decision making, such agents have shown promising capability in operating complex applications across desktop, web, and mobile environments. Representative systems such as CogAgent (Hong et al., 2024) and AppAgent further demonstrate the potential of this paradigm for practical human computer interaction. To support systematic progress, prior benchmarks including AITW (Gur et al., 2024; Ma et al., 2026) and ScreenSpot (You et al., 2024; Qin et al., 2025) have provided standardized testbeds for evaluation. Despite rapid progress in LVLM-based GUI agents (Zhang et al., 2025b; Ma et al., 2026), achieving fine-grained state-conditioned interaction remains challenging in practice. In many real applications, a single instruction requires setting a UI control to an exact target state using only the agent’s first predicted interaction point, such as adjusting a slider to a precise value, selecting an exact date/time, or choosing a specific color. However, agents that perform strongly under coarse success criteria or standard grounding benchmarks such as ScreenSpot/SeeClick (Cheng et al., 2024), Android in the Wild (Rawles et al., 2023), and VisualWebArena (Koh et al., 2024) can still fail to reliably reach precise target states. A key limitation lies in current evaluation practices. First, at the benchmark and task-definition level, existing evaluations for GUI and web/mobile agents predominantly focus on end-to-end task completion or click-level grounding. This design under-represents state-conditioned interaction scenarios, and the target specifications are often insufficiently precise to enable unambiguous verification of intermediate or final target states (Rawles et al., 2023; Deng et al., 2023; Zhou et al., 2023; Koh et al., 2024; Lu et al., 2024; Zeng et al., 2025; Cao et al., 2026). Second, at the evaluation-
1
Static Data Creation Process 2
Filtering from Dataset 3
A-1. Slider A-2. Knob A-3. Steeper A-4. Seek Bar
Manual Screenshot
Human Annotation
4 LLM Generation Pipeline
5
Manual Verification
A-5. Chart Point B-1. Switch B-2. Check Box B-3. Radio Group
VDA Improve
(a) Input Screenshot x Instruction I
(b)
VDA Model w/ Description w/o Localization Hint Output Describe (structured description d)
GUI Element B-4. Tabs C-2. Color Picker B-5. Segmented C-3. Date Picker B-6. Accordion C-4. Time Picker C-1. Rating C-5. List Box
Locate (TargetBBox B=[x1.y1,x2.y2])
D-1. Drag Reorder D-2. Zoom Pan … D-7. Table Column
Execute Action
(c) Visual Diagnosis
(d)
Correct Border: [0.30, 0.15,0.34,0.22]
Correct Border: [0.53, 0.31,0.55,0.33]
Correct Border: [0.30, 0.15,0.34,0.22]
Correct Border: [0.50, 0.45,0.54,0.52]
Category: Range
Category: Option
Category: Specific Data
Category: Table Column
Coordinate: Click[0.29, 0.18]
Coordinate: Click[0.19, 0.15]
Coordinate: Click[0.19, 0.15]
Coordinate: Click[0.49, 0.45]
Figure 1: FineState-Bench overview and VDA. (a) Static data creation pipeline (filtering, supplementation, annotation, instruction/state drafting, verification). (b) VDA-assisted evaluation that appends target-region localization hints for controlled comparisons. (c) Fine-grained interaction taxonomy with four families and 23 UI component types. (d) Example instances with precise target boxes and normalized interaction points.
protocol and metric level, many studies primarily report aggregate outcomes such as final task success or overall accuracy. Such coarse metrics collapse the entire perception–grounding–interaction pipeline into a single score, obscuring where failures occur and preventing fine-grained diagnostic analysis and failure attribution (Koh et al., 2024; Xue et al., 2025; Liang and Zhang, 2025a; Liang and Zhou, 2025; Yang et al., 2025a). As a consequence, prior work may arrive at inconsistent conclusions regarding failure sources. To address these limitations, we introduce FineState-Bench, a cross-platform benchmark of 2,209 instances for single-step, fine-grained stateconditioned GUI exact state setting with exact goalstate verification. Rather than modeling full interactive trajectories, FineState-Bench focuses on a controlled static single-step setting to isolate fine-grained state-conditioned grounding and exact state-setting ability. Agents are evaluated under a single-step, point-based protocol, as illustrated in Figure 1 and Figure 4. Unlike proxybased evaluations, each instance provides exact goal-state labels and dual-region annotations (a
control-extent locate box and an interactable-core box) under current/target configurations, enabling unambiguous verification of exact goal-state attainment. Built on these annotations, we propose FineState-Metrics, a stage-wise diagnostic pipeline (SR@Loc, SR@Int, ES-SR@Loc, ESSR@Int) that decomposes performance from component grounding to interactable-core grounding and point precision, and finally exact goal-state attainment. To further quantify the grounding errors, we introduce a plug-and-play Visual Diagnostic Assistant (VDA) that optionally appends a Description and/or a Localization Hint; experiments show that while current agents exhibit a low success floor, VDA yields substantial gains, indicating insufficient accuracy for broad fine-grained stateconditioned interactions. Our main contributions are: (1) We define and study fine-grained, state-conditioned GUI state setting with explicit goal-state labels and exact verification in desktop/web/mobile platforms. (2) We build and release FineState-Bench featuring dualregion annotations under current/target configurations for component localization and interactable-
core precision, enabling fine-grained, reproducible evaluation. (3) We introduce FineState-Metrics and VDA to decompose failures and perform controlled input-augmentation analysis, and we benchmark 8 representative agents to quantify how much performance is limited by interactable-core localization and point precision.
2
Related Work
GUI Agents. GUI agents have advanced rapidly with large vision-language and multimodal models (Zhang et al., 2024a; Nguyen et al., 2024). Early studies often relied on general-purpose models for GUI operation (Yang et al., 2023; Adept AI, 2024; Zheng et al., 2024a), while later work increasingly builds GUI-specialized agents and UI grounding models (Hong et al., 2024; Wang et al., 2024b; Lin et al., 2024; Wu et al., 2024; Li et al., 2025; Gou et al., 2025; Chen et al., 2024a; Qin et al., 2023). These systems span mobile and desktop platforms (Wang et al., 2024a; Nong et al., 2024; Jiang et al., 2025; Liu et al., 2025; Fu et al., 2024; Zhang et al., 2025a; Yang et al., 2025b, 2026), yet reliably achieving state-conditioned exact state setting remains challenging. In practice, small execution errors can accumulate into user-visible failures, especially for precise controls such as sliders, pickers, and professional UI widgets, motivating evaluations that stress fine-grained state manipulation rather than only task completion. GUI Agent Evaluation. Existing evaluations include interactive end-to-end benchmarks (Zhou et al., 2023; Koh et al., 2024; Rawles et al., 2024; Xie et al., 2024; Zhang et al., 2024d; Chen et al., 2025; Zhang et al., 2024b) and offline grounding benchmarks on screenshots or professional software (Deng et al., 2023; Zhao et al., 2024; Qian et al., 2024; Dardouri et al., 2024). While these benchmarks improve realism or grounding assessment, they often under-cover state-conditioned interaction scenarios and lack precise target-state specifications for unambiguous verification. Moreover, many protocols emphasize aggregate success rates, limiting deeper analysis (Zheng et al., 2024b; Zhang et al., 2024c). Recent benchmarks probe robustness or distribution shifts, but they typically do not isolate failures from mis-perception, mislocalization, or incorrect state outcomes when exact goal-state attainment is required.
Diagnosis and Failure Attribution. Recent work highlights that coarse metrics can obscure bottlenecks and yield divergent conclusions about failure sources (Shlomov et al., 2024). Related efforts improve grounding or data coverage (Chen et al., 2024b) or analyze reliability issues (Liu et al., 2024b; Liang and Zhang, 2025b), but controlled diagnosis for disentangling visual grounding from non-visual interaction and state-control factors remains limited.
3 FineState-Bench 3.1 Problem Definition FineState-Bench targets fine-grained, stateconditioned GUI state setting with exact goal states. Given a screenshot x and two instructions (I 0 , I 1 ), an agent predicts two points (p0 , p1 ): p0 indicates the target control’s location in the current UI, and p1 indicates the operation location required to reach the intended fine-grained goal state. Specifically, p0 corresponds to the current instruction I 0 for locating the target control, whereas p1 corresponds to the target instruction I 1 for specifying the operation location toward the intended goal state. Each instance is defined 0 , B 0 , B 1 , B 1 ), as τ = (x, I 0 , I 1 , c∗ , sgoal , Bloc int loc int ∗ where c is the target control and sgoal is the desired goal state. All boxes are axis-aligned rectangles in normalized screen coordinates, each parameterized by (xmin , ymin , xmax , ymax ) ∈ [0, 1]4 ; for trajectory-based actions, we define pt (t ∈ {0, 1}) as the final release (action-commit) point, since it determines the resulting state in typical GUI systems. 0 denotes the locate box covering the current Bloc 0 denotes the box of visible extent of c∗ , while Bint the operation-relevant element/region in the current configuration for reaching sgoal . To account for controls whose position or size may change during the intended operation, we additionally anno1 denotes the lotate target-configuration boxes: Bloc ∗ 1 cate box of c in the target configuration, and Bint denotes the corresponding box of the operationrelevant element/region in the target configuration for reaching sgoal . We write p ∈ B when a predicted point p falls inside box B; we compare 0 , B 1 ) and p1 against (B 0 , B 1 ). p0 against (Bloc loc int int This enables evaluation under both the current (·0 ) and target (·1 ) configurations, capturing potential layout changes during the intended operation. Each control c has a precise, quantifiable state
Specific Data Toggle Option
3.3 FineState-Metrics
Numeric Range
153
208
View Manipilation 128
180
W eb
147
186
Static
Numeric Range
Specific Data
Desktop
221
176
Numeric Range
Specific Data 182
Toggle Option 193
(a) Word Cloud
Toggle Option
e bil Mo
View Manipilation
View Manipilation
213
222
(b) Task Type Distribution
Figure 2: Visualization of instruction keywords and task type composition in FineState-Bench.
s(c), such as a slider value, toggle status, selected option. This two-point design decouples locating the target control in the current UI (p0 ) from specifying the goal-directed operation location (p1 ), reducing interference from multi-step interactions and enabling fine-grained diagnosis of localization versus operation. 3.2
Dataset Construction and Annotation
Benchmark Composition. FineState-Bench contains 2,209 high-quality static GUI statesetting instances collected from three platforms: Desktop (810), Web (701), and Mobile (698), with data sourcing and quality control detailed in Appendix A. Each instance includes: (i) a screenshot x, (ii) two instructions (I 0 , I 1 ) for current localization and target operation specification, (iii) an exact goal-state label sgoal with fine-grained state annotations for the target control, and (iv) four geometric annotations for the target control in the current and target configurations 0 , B 0 , B 1 , B 1 ). (Bloc We further verify the int loc int reliability of the fine-grained state and geometric annotations through a double-annotation study; details are provided in Appendix D. Interaction Taxonomy. To ensure systematic coverage, we group instances into four interaction families: (1) Numerical and Range Adjustment; (2) State Toggling and Option Selection; (3) Specific Data-Type Selection; and (4) Content Organization and View Manipulation. Instruction Statistics. Figure 2(a) summarizes instruction keywords, highlighting common interaction verbs such as select, click, drag, and adjust, alongside state-related entities including volume, brightness, time, and color. Figure 2(b) reports the task-type distribution across the benchmark.
While overall task success provides a coarse measure of agent capability, it offers limited insight into where fine-grained state control fails along the perception-to-interaction pipeline. FineStateBench is designed not only to score overall success, but also to attribute failures to specific capability factors in fine-grained state control. As defined in §3.1, we evaluate each instance using the two predicted points (p0 , p1 ) and the resulting (recorded/annotated) target state s1 (c∗ ). We introduce four diagnostic Success Rates aligned with the four pipeline stages of Perception, Localization, Interaction, and State Correctness. Perception Success Rate measures instruction understanding and goal identification by checking whether the agent correctly identifies the target control and intended goal state from the inputs (I 0 , I 1 ), independent of any point prediction. Localization Success Rate (SR@Loc) measures coarse component grounding (control by checking whether the predicted control-location point p0 falls inside the locate box of the target control in 0 , while the current configuration, i.e., p0 ∈ Bloc Interaction Success Rate (SR@Int) measures finegrained interactable-core grounding and point precision by checking whether the predicted operation point p1 falls inside the operation-relevant (statechanging) region in the current configuration, i.e., 0 . Exact State Success Rate (ES-SR) p1 ∈ Bint measures exact goal-state attainment by checking whether the resulting (recorded/annotated) state of the target control reaches the exact goal state, s1 (c∗ ) = sgoal . Accordingly, Exact State Success Rate at Locate (ES-SR@Loc) and Exact State Success Rate at Interact (ES-SR@Int) measure exact goal-state success conditioned on p0 ∈ 1 and p1 ∈ B 1 , respectively, i.e., under the Bloc int target-configuration locate/interact boxes (see Appendix F for interpretation and failure-attribution rules):
] 1 ∑ [ (i) (i) I p0 ∈ Bloc 0 , N N
SR@Loc =
i=1
N ] 1 ∑ [ (i) (i) SR@Int = I p1 ∈ Bint 0 , N i=1
N ( ) ] 1 ∑ [ (i) (i) (i) (i) I p0 ∈ Bloc 1 ∧ s1 c∗(i) = sgoal , ES-SR@Loc = N i=1 N ( ) ] 1 ∑ [ (i) (i) (i) (i) I p1 ∈ Bint 1 ∧ s1 c∗(i) = sgoal . ES-SR@Int = N
(1)
i=1
Here N is the number of instances, (i) indexes an instance, I(·) is the indicator function, and ∧ de-
Platform
Benchmark
State-Control Evaluation Properties
Number
Desktop
Mobile
Website
Target State Labels
Bloc /Bint
Stage-wise Diag
ScreenSpot (Cheng et al., 2024) ScreenSpot-v2 (Wu et al., 2024) ScreenSpot-Pro (Zhao et al., 2024) WebClick (Andreux et al., 2025) VisualWebBench (Liu et al., 2024a) UI-Vision (Nayak et al., 2025) OSWorld-G (Xie et al., 2025)
Yes Yes Yes – – Yes Yes
Yes Yes – – – – –
Yes Yes – Yes Yes – –
No No No No No No No
No No No No No No No
No No No No No No Yes
1272 1272 1581 1639 1536 1464 564
FineState-Bench
Yes
Yes
Yes
Yes
Yes
Yes
2209
Table 1: Comparison of FineState-Bench with representative static GUI benchmarks in terms of evaluation metrics and diagnostic capabilities. We summarize whether each benchmark supports Target State Labels verification, dualregion geometric supervision (Bloc /Bint ), and stage-wise diagnostic metrics that disentangle localization accuracy, point-level precision, and exact state attainment. Fails
Input Screenshot x Instruction I Success
(a) Original Pipeline GUI Agent
Perception
Grounding
VDA Model
Screenshot x Instruction I
Execute Grouding
(b) VDA-Augmented Pipeline
COMPONENT (w/o) INFORMATION. Description Bounding Box: [0.310, 0.286, 0.669, 0.714]
Input
Incorrect Grouding
(structured description d)
Component Center: (0.489, 0.500)
(B=[x1.y1,x2.y2])
GUI Agent + VDA Hint
Input
Coordinates are
normalized to 0-1 (w/o) Localization Hint range.
Correct Grouding
Execute Grouding
VDA Hint
Figure 3: Baseline and VDA-augmented pipelines for the target instruction I 1 . VDA (GPT-4o) first produces a structured Description of the target UI element and then a Localization Hint B̂ 1 , which is appended to the agent input when predicting p1 . The performance gap in ES-SR@Int quantifies the visual grounding bottleneck for goal-directed operation.
nostic Assistant (VDA) as a diagnostic tool for controlled comparisons. We use w/ and w/o to denote with and without VDA. VDA evaluates the same agent with and without an explicit Localization Hint under the target instruction I 1 , isolating whether failures arise from inaccurate grounding of the goal-directed operation region or from visual perception factors. As shown in Figure 3, VDA produces a structured Description and a Localization Hint (a target-region bounding box) for I 1 ; either the Description, the Localization Hint, or both are appended to the agent input. Concretely, given (x, I 1 ), VDA predicts B̂ 1 ∈ [0, 1]4 as the Localization Hint: B̂ 1 = L(x, I 1 ), where L is instantiated with GPT-4o. Using ESSR@Int under I 1 , we quantify the visual grounding bottleneck via the w/ vs. w/o VDA gap: ∆vis = ES-SR@Int(w/ VDA) − ES-SR@Int(w/o VDA). (2)
∆vis estimates the performance recoverable from improved visual grounding under this Localization Hint interface.
notes conjunction. For example, in Fig.4 (D5), SR@Loc is counted as success if p0 lands within the TreeView’s cur0 ), whereas SR@Int requires rent visible extent (Bloc 1 0 ) p to hit the current operation-relevant region (Bint that can trigger the intended state change. If the 1 but the resulting state still differs agent hits Bint from sgoal , ES-SR@Int remains unsuccessful, indicating a state-setting error beyond location/operation prediction.
VDA Design. VDA follows a two-step describethen-localize procedure under I 1 to produce a highfidelity Localization Hint. First, given (x, I 1 ), VDA generates a Description of the target UI element, including its functional role or state, discriminative visual cues, and spatial relations to anchors. This Description is used for disambiguation and is optionally provided to the agent. Then, conditioned on the screenshot, the instruction I 1 , and the generated Description, VDA predicts B̂ 1 in normalized coordinates as the Localization Hint.
3.4
Plug-and-Play Integration of VDA. For each instance, when predicting p1 under I 1 , the evaluated agent receives its standard inputs optionally augmented with the VDA-predicted Localization
Visual Diagnostic Assistant
Diagnostic Use of VDA. To conduct an in-depth study of the factors influencing errors in finegrained operations, we introduce the Visual Diag-
Hint B̂ 1 . Importantly, the intervention can be the Description, the Localization Hint, or both. This design enables controlled ablations that isolate the effect of the Localization Hint from textual disambiguation.
for exact goal-state attainment, and leveraging the stage-wise success rates (SR@Loc, SR@Int, ESSR@Loc) for diagnosis and bottleneck attribution.
4
Table 2 shows consistently low success for exact state setting, with ES-SR@Int as the primary metric (see Appendix F for metric interpretation and rule-based failure attribution). Even the strongest model, UGround-7B, reaches only 32.8% ES-SR@Int on Web and 22.8% on average. Performance can also collapse on specific platforms despite reasonable localization (e.g., Gemini-2.5Flash: 17.6% on Mobile vs. 0.7% on Desktop), indicating that robust fine-grained state setting remains difficult even on static screenshots. FineState-Metrics attributes most errors to the transition from coarse component grounding to interactable-core grounding. Across models, SR@Loc is substantially higher than SR@Int, suggesting that agents often localize the correct con0 ) but miss the state-changing core trol (p0 ! ∈!Bloc 0 ). Moreover, SR@Int is when executing (p1 ! ∈!B / int typically close to ES-SR@Int, implying that once the predicted point hits the interactable core, the exact goal state is usually achieved. This stage-wise degradation is visualized in Fig. 5, highlighting interactable-core grounding as the dominant bottleneck for broad fine-grained, state-conditioned interaction.
Benchmark Characteristics and Analysis
Table 1 shows that representative offline/static GUI benchmarks mainly test whether an agent can identify and click the intended element, or respond to higher-level prompts, without requiring verifiable post-interaction state changes. As a result, models may score well even if the interaction is not precise enough to reach an exact target state. Moreover, they often lack a clear distinction between a control’s visible extent and the operation-relevant region that changes its state, making failures hard to interpret whether the model mis-grounded the control, clicked an ineffective region, or failed to set the correct state. FineState-Bench addresses this gap by evaluating whether an agent can set a target control to an exact goal state with a single-step, single-point interaction. Each instance provides a goal state and dual bounding-box supervision, as shown in Figure 4: the locate box covers the control’s extent, while the interact box marks the state-changing core region. This design enables clear attribution in a single interaction: missing the locate box indicates grounding failure; hitting locate but missing interact suggests insufficient point precision; and hitting interact but not reaching the goal state indicates a state-setting error beyond localization.
5
Experiments and Analysis
5.1
Baselines
We benchmark 8 representative GUI agents on FineState-Static. For a balanced comparison, we include 3 closed-source LVLMs (GPT-4o (OpenAI, 2024), Claude-3.5-Sonnet (Anthropic, 2024), Gemini-2.5-Flash (Google, 2024)) as strong general-purpose multimodal baselines, and 5 open-source GUI agents (OS-Atlas-7B (Wu et al., 2024), CogAgent-9B (Hong et al., 2024), UGround-7B (Gou et al., 2025), Jedi-7B1080p (Fu et al., 2024), ShowUI-2B (Lin et al., 2024)) that emphasize GUI grounding and action prediction. We evaluate all agents under the single-point protocol, using ES-SR@Int as the primary metric
5.2 Main Result
5.3 Component-Level Diagnosis of Interactable-Core Grounding As shown in Figure 6, we break down SR@Loc and SR@Int by UI component to diagnose which interaction types contribute most to the drop from coarse localization to interactable-core grounding. We find that precision-sensitive, continuous controls exhibit the largest gaps: on sliders, models often achieve reasonable SR@Loc but much lower SR@Int, and on seek bars SR@Int can even collapse despite strong localization. In contrast, discrete controls are more tractable, with steppers showing a much smaller gap between SR@Loc and SR@Int. Overall, these componentwise results suggest that the dominant bottleneck in broad fine-grained interactions is the transition 0 ) from coarse component localization (p0 ! ∈!Bloc 1 0 to interactable-core grounding (p ! ∈!Bint ); see Table 7 for full numbers.
🔴 red - locate 🟡 yellow - locatebbox 🟢 green - interact 🔵 blue - interactbbox A1-Slider
A2-Knob
Locate: Help me find where this music is playing right now. Interact: Please drag the slider to 74% of the maximum value for a rolling line. locate: [0.00,0.89,1.00,0.93], locatebbox: [0.00,0.89,0.02,0.92] interact: [0.00,0.89,1.00,0.93] interactbbox: [0.54,0.89,0.56,0.93]
A5-Chart Point Locate: Give the Samp2 cross-axis to 4 points. Interact: Drag to 40 vertical axis. locate: [0.0, 0.01, 0.97, 1.0] locatebbox: [0.5, 0.12, 0.54, 0.17] interact: [0.0, 0.01, 0.97, 1.0] interactbbox: [0.51, 0.32, 0.54, 0.36]
B4-Tabs
A3-Stepper
Locate: Position the current value of the B-rotton from left to right fourth sound adjuster. Interact: Reconciling the B-button of the fourth sound adjuster from left to right to right to right 105 degrees. locate: [0.58, 0.68, 0.60, 0.74] locatebbox: [0.58, 0.72, 0.59, 0.73] interact: [0.58, 0.68, 0.60, 0.74] interactbbox: [0.58, 0.70, 0.59, 0.71]
Interact: Click on the \" Layout Toolbox \" tab in the OfficePLUS panel" locate: [0.38, 0.41, 0.58, 0.48] locatebbox: [0.38, 0.41, 0.45, 0.48] interact: [0.38, 0.41, 0.58, 0.48] interactbbox: [0.45, 0.41, 0.58, 0.48]
C2-ColorPicker Locate: Position to Bottom Colour Selection. Interact: "Press the red button.
locate: [0.36, 0.94, 0.57, 0.98] locatebbox: [0.48, 0.94, 0.5, 0.97] interact: [0.36, 0.94, 0.57, 0.98] interactbbox: [0.44, 0.94, 0.45, 0.97]
D1-DragReorder Locate: Gives the position of the last icon in the left sidebar. Interact: Drag it to the fourth.
locate: [0.63,0.08,0.99,0.14] locatebbox: [0.93,0.09,0.95,0.12] interact: [0.63,0.08,0.99,0.14] interactbbox: [0.96,0.10,0.97,0.13]
B3-RadioGroup
Interact: Switch to Fourth Theme.
locate: [0.0, 0.14, 0.16, 0.24] locatebbox: [0.01, 0.17, 0.02, 0.20] interact: [0.0, 0.14, 0.16, 0.24] interactbbox: [0.01, 0.17, 0.02, 0.19]
locate: [0.05, 0.37, 0.9, 0.78], locatebbox: [0.05, 0.4, 0.21, 0.57], interact: [0.05, 0.37, 0.9, 0.78], interactbbox: [0.53, 0.39, 0.69, 0.58]
B6-Accordion
C1-Rating
Locate: Gives the position of the eighth slide in the left preview map.
Locate: Please find the button to withdraw the 'Text Direction' option. Interact: Please take back the list of 'Text Division' options.
Interact: Click on the 11th slide. locate: [0.03, 0.01, 0.22, 0.97] locatebbox: [0.02, 0.33, 0.21, 0.48] interact: [0.03, 0.01, 0.22, 0.97] interactbbox: [0.03, 0.81, 0.20, 0.97]
Locate: Please give the frame position for the first theme of the personalization device.
Interact: Check 'Find whole words only.
Interact: Turn on the switch in the automatic time zone. locate: [0.44, 0.11, 0.68, 0.15] locatebbox: [0.65, 0.11, 0.68, 0.15] interact: [0.44, 0.11, 0.68, 0.15] interactbbox: [0.65, 0.11, 0.68, 0.14]
locate: Please locate the star rating position. Interact: Rating the stars to five stars. locate: [0.5, 0.39, 0.62, 0.43] locatebbox: [0.5, 0.38, 0.62, 0.43] interact: [0.5, 0.39, 0.62, 0.43] interactbbox: [0.6, 0.38, 0.61, 0.42]
locate: [0.3, 0.04, 0.38, 0.31] locatebbox: [0.31, 0.04, 0.35, 0.06] interact: [0.3, 0.04, 0.38, 0.31] interactbbox: [0.31, 0.05, 0.35, 0.06]
C3-DatePicker
C5-ListboxDropdown
C4-TimePicker
Locate: The date frame on the right of the location date",
locate: [0.54, 0.17, 0.75, 0.58] locatebbox: [0.58, 0.32, 0.6, 0.35] interact: [0.54, 0.17, 0.75, 0.58] interactbbox: [0.65, 0.41, 0.68, 0.45]
Locate: Positioning to a drop-down menu with a character size. Interact: Remove the word size and pull the menu down.
Locate: Positioning end time selector.
Interact: Click 25 to select date.
Interact: Click finish time 3:30.
locate: [0.25, 0.43, 0.36, 0.65] locatebbox: [0.25, 0.44, 0.35, 0.49] interact: [0.25, 0.43, 0.36, 0.65] interactbbox: [0.25, 0.54, 0.35, 0.58]
locate: [0.36, 0.1, 0.42, 0.93] locatebbox: [0.35, 0.09, 0.4, 0.13] interact: [0.36, 0.1, 0.42, 0.93] interactbbox: [0.36, 0.1, 0.4, 0.14]
D3-ResizablePane
D2-ZoomPan Locate: Click the bottom right corner of the slide.
D6-Splitter
D4-Carousel
Interact: Reduction in length to 61 per cent and in height to 70 percent locate: [0.11, 0.24, 0.83, 0.73] locatebbox: [0.11, 0.69, 0.14, 0.73] interact: [0.39, 0.25, 0.83, 0.58] interactbbox: [0.39, 0.55, 0.42, 0.58]
locate: [0.0, 0.0, 1.0, 1.0] locatebbox: [0.0, 0.0, 1.0, 1.0] interact: [0.0, 0.44, 0.03, 0.54] interactbbox: [0.01, 0.45, 0.03, 0.54] Locate the currently selected item in the left vertical column.
D7-TableColumn locate: Select the D-E line boundary.
Locate: Select Left of the Comments Bar.
Interact: Move to 70% of column B horizontal direction.
Interact: Move to between AK and AL. locate: [0.85, 0.14, 0.86, 0.95] locatebbox: [0.85, 0.14, 0.86, 0.95] interact: [0.7, 0.14, 0.72, 0.94] interactbbox: [0.7, 0.14, 0.71, 0.94]
Locate: Please give the frame position for the change of the picture. Interact: Please switch to the previous picture."
Locate: Click the bottom left corner of the window
Interact: Reduce page lengths to 77 per cent and high to 71 percent.
D5-TreeView
locate: [0.03, 0.21, 0.21, 0.34] locatebbox: [0.05, 0.21, 0.06, 0.23] interact: [0.05, 0.21, 0.15, 0.23] interactbbox: [0.05, 0.21, 0.07, 0.23]
locate: [0.00,0.90,1.00,0.93] locatebbox:[0.00,0.89,0.02,0.93] interact: [0.00,0.90,1.0,0.93] interactbbox: [0.54,0.89,0.56,0.93]
Locate: Could you give us the frame position of the 'Find whole words only' option?
Locate: Please give the switch position for the automatic time zone.
locate: [0.16, 0.11, 0.87, 0.83] locatebbox: [0.83, 0.75, 0.87, 0.82] interact: [0.16, 0.11, 0.7, 0.62] interactbbox": [0.64, 0.53, 0.7, 0.63]
Interact: Please withdraw the 'level_grounding' options.
Interact: Help me point out where the markers in the progress bar are.
B2-Checkbox
B1-Switch
locate: [0.01, 0.04, 0.06, 0.89] locatebbox: [0.0, 0.79, 0.05, 0.89] interact: [0.0, 0.04, 0.05, 0.89] interactbbox: [0.0, 0.31, 0.05, 0.35]
Locate: Please find the button to withdraw the 'level_grounding'option.
Locate: Help me find where this music is playing right now.
Interact: Values of rows to be reduced by the stepr.
B5-Segmented
Locate: Click on the \" Layout Toolbox \" tab in the OfficePLUS area.
A4-Seek Bar
Locate: Gives the location of the current reconciliation column step forwards.
locate: [0.34, 0.28, 0.41, 1.0] locatebbox: [0.34, 0.27, 0.36, 1.0] interact: [0.19, 0.28, 0.41, 1.0] interactbbox: [0.18, 0.27, 0.21, 1.0]
Interaction Coordinates:[0.230, 0.220]. Interact: Click the 'BNB' item in the left vertical column. Interaction Coordinates: [0.207, 0.583].
Figure 4: Representative instances from FineState-Bench, covering all 23 UI component types. For each control, we annotate four normalized bounding boxes. Model Name
Mobile
Web
Desktop
AVG
20.6/2.2/4.4/2.2 22.0/3.4/5.4/3.4 12.7/0.7/3.8/0.7
24.8/4.3/6.8/4.3 23.0/5.6/7.9/5.6 36.4/10.0/13.9/10.0
Closed-source Models GPT-4o Claude-3.5-Sonnet Gemini-2.5-Flash
31.0/6.2/9.1/6.2 22.8/4.5/7.0/4.5 31.7/11.5/13.7/11.5 15.2/1.9/4.6/1.9 49.4/17.6/21.1/17.6 47.0/11.8/16.8/11.8 Open-source Models
OS-Atlas-7B (Wu et al., 2024) 47.5/12.8/18.7/12.8 33.2/7.5/9.2/7.5 45.3/9.8/15.2/9.8 42.0/10.0/14.4/10.0 CogAgent-9B (Hong et al., 2024) 17.7/1.8/2.5/1.8 24.1/6.4/13.7/6.4 29.4/2.3/3.5/1.2 23.7/3.5/6.6/3.1 UGround-7B (Gou et al., 2025) 50.7/19.6/22.4/19.6 62.0/32.8/62.0/32.8 46.3/16.0/25.4/16.0 53.0/22.8/36.6/22.8 13.3/1.6/3.1/1.5 12.7/8.3/12.7/8.3 12.2/0.8/1.5/0.8 12.7/3.6/5.8/3.5 Jedi-7B-1080p (Fu et al., 2024) ShowUI-2B (Lin et al., 2024) 20.3/5.2/6.7/5.2 26.7/5.3/26.7/5.3 30.3/3.2/9.1/3.2 25.8/4.6/14.2/4.6
Table 2: Baseline evaluation on FineState-Static. Under the single-point protocol, we report four diagnostic metrics: 0 0 1 1 SR@Loc (p0 ∈ Bloc ) / SR@Int (p1 ∈ Bint ) / ES-SR@Loc (p0 ∈ Bloc ) / ES-SR@Int (p1 ∈ Bint ) (%).
5.4
Diagnosis of Performance Bottlenecks
To attribute failures, we run a controlled comparison with VDA, which injects full VDA hints consisting of a Description and a Localization
Hint, while keeping the evaluated agent unchanged (same parameters/decoding and the same pointbased interaction interface). To attribute failures, we run a controlled comparison with VDA, which
60 GPT-4o Claude-3.5-Sonnet Gemini-2.5-Flash OS-Atlas-7B
Success Rate (%)
50 40
CogAgent-9B UGround-7B Jedi-7B-1080p ShowUI-2B
30
ant (w/ Description and w/ Localization Hint) (w/ Description and w/ Localization Hint) performs best, consistent with the description providing contextual disambiguation that improves the reliability of subsequent localization.
20 10 0 SR@Loc
SR@Int
ES-SR@Loc
ES-SR@Int
FineState Metrics
Figure 5: Performance degradation on FineState-Static.
injects full VDA hint, consisting of the Description and the Localization Hint, while keeping the evaluated agent unchanged (same parameters/decoding and the same point-based interaction interface). As shown in Table 3, VDA-generated localization hints substantially improve ES-SR@Int, with the largest gain observed on Gemini-2.5-Flash (+14.9% on average). These recoverable gains indicate that a major portion of failures stems primarily from inaccurate grounding/localization of the goal-directed operation region, with visual perception factors playing a secondary role, while overall accuracy still remains insufficient for broad finegrained interactions. Model
Mobile Web Desktop
Gemini-2.5-Flash ShowUI-2B OS-Atlas-7B
17.6 5.2 12.8
11.8 5.3 7.5
0.7 3.2 9.8
Gemini-2.5-Flash(VDA-Gemini-2.5-Flash) ShowUI-2B(VDA-Gemini-2.5-Flash) OS-Atlas-7B(VDA-Gemini-2.5-Flash)
29.8 5.8 18.9
27.2 12.3 18.2
15.4 7.5 19.1
Gemini-2.5-Flash(VDA-GPT4o) ShowUI-2B(VDA-GPT4o) OS-Atlas-7B(VDA-GPT4o)
29.8 7.3 15.9
26.6 7.2 14.2
20.9 9.5 13.1
Table 3: Overall impact of VDA on ES-SR@Int (%). We report baseline and VDA-augmented performance under identical agent settings; improvements reflect error recoverable by better visual grounding.
5.5
Ablation Study of VDA
We conduct an ablation study on Gemini-2.5-Flash in Table 4 to each VDA component / input cue / hint type under the single-point protocol. Using w/ Description but w/o localization hint yields no measurable improvement over w/o VDA, with ESSR@Int remaining nearly unchanged across platforms. By contrast, using w/ Localization Hint produces a pronounced increase in ES-SR@Int across all platforms, indicating that localization quality is a major driver of the overall gain. The the full vari-
VDA Configuration
Mobile Web Desktop
w/o VDA w/ Description, w/o Localization Hint w/o Description, w/ Localization Hint w/ Description, w/ Localization Hint
17.6 17.9 26.1 29.8
11.8 11.9 24.7 27.2
0.7 0.8 13.1 15.4
Table 4: Ablation study of VDA-Flash on Gemini-2.5Flash. We measure ES-SR@Int (%) under the singlepoint protocol.
We provide qualitative failure cases and w/ VDA comparisons in Appendix E of the Supplementary materials to illustrate typical error modes, such as missing the interactable core due to imprecise point placement.
6 Conclusion We present FineState-Bench, an open-source benchmark and diagnostic framework for finegrained, state-conditioned GUI state setting with exact goal-state verification across desktop, web, and mobile platforms. Our study highlights a key gap in current GUI agent evaluation: existing benchmarks rarely support exact goal-state verification and controlled, stage-wise diagnosis, making it difficult to attribute failures in fine-grained state-conditioned interactions. We hope FineStateBench, together with FineState-Metrics and VDAbased analysis, will enable precise evaluation and accelerate progress toward reliable, state-aware GUI agents, enabling attribution analysis.
Limitation Our study has several limitations. First, FineStateBench is designed to isolate fine-grained, stateconditioned state setting under static screenshots and a single-point interaction setting. This controlled setup prioritizes precise state verification and diagnostic clarity, rather than modeling longhorizon reasoning or multi-step corrective behaviors, which are complementary directions explored by existing interactive benchmarks. Second, VDA is designed as a diagnostic tool rather than a deployable component, and its localization hints rely on high-quality visual cues in the screenshot; extending this analysis to fully end-to-end or realtime interactive settings remains an open chal-
Δ86% Δ79%
80
SR@Loc SR@Int
Δ79% Δ68% Δ58% Δ57%
60
Δ50%
Δ47%
Δ46% Δ40% Δ40% Δ39%
40
Δ38% Δ38% Δ38% Δ37% Δ36% Δ36% Δ34% Δ34%
Δ32% Δ31% Δ31%
20 5.3%
5.1% 3.5% 3.7% 4.0%
0.3% 2.2%
4.3% 4.4%
2.3%
Kn D ob AT E_ IN P D U T at e Pi Ti m ck e er se le ct i T Ra on im di e o Pi Bu ck tt er on G ro Si up N ng av le ig at Bu io tt n on G ro up CO LO Ta R_ bs SE LE CT O R
el Bu tt on Te xt In pu t Se ek Ba r
Ra d
io
LD
Pa n
le
iz ab
Re s
IN PU
T_ F
IE
er
LD
ay
IE
Pl eo
T_ F
IN PU
Vi d
st
nd ar
Li
le Ca TE
XT _
nt
ay
Po i
ar t
D
Ch
D
is
pl
es gg
Su
CA
Se
ar ch
RO
U
SE
L
t
0
at a
Success Rate (%)
100
Component Type
Figure 6: Component-level localization vs. interactable-core grounding on FineState-Static.
lenge. Third, while the benchmark covers a broad range of platforms and UI components, it does not exhaustively represent all application domains or accessibility-driven interface variations, which may exhibit different grounding and state-control characteristics.
Acknowledgements We thank the anonymous reviewers and the area chair for their constructive comments. We also thank our mentors and colleagues from MBZUAI for their support and help.
References Adept AI. 2024. Apt: A general-purpose multimodal agent for vision-language-action tasks. arXiv preprint, arXiv:2407.01735. Mathieu Andreux, Breno Baldas Skuk, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Pierre-Louis Cedoz, Antoine Chassang, and 1 others. 2025. Surfer-H meets Holo-1: Cost-efficient web agent powered by open weights. arXiv preprint, arXiv:2506.02865. Anthropic. 2024. Introducing Claude 3.5 Sonnet. Jinghan Cao, Yu Ma, Xinjin Li, Qingyang Ren, and Xiangyun Chen. 2026. Task-specific efficiency analysis: When small language models outperform large language models. arXiv preprint, arXiv:2603.21389. Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. 2025. SPA-Bench: A comprehensive benchmark for smartphone agent evaluation. In Proceedings of the International Conference on Learning Representations. Wentong Chen, Junbo Cui, Jinyi Hu, Yujia Qin, Junjie Fang, Yue Zhao, Chongyi Wang, Jun Liu, Guirong Chen, Yupeng Huo, and 1 others. 2024a. GuiCourse: From general vision language models to versatile GUI agents. arXiv preprint, arXiv:2406.11317.
Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, and Deqing Yang. 2024b. EDGE: Enhanced grounded GUI understanding with enriched multi-granularity synthetic data. arXiv preprint, arXiv:2410.19461. Accepted at TheWebConf 2025. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI grounding for advanced visual GUI agents. arXiv preprint, arXiv:2401.10935. Tassnim Dardouri, Laura Minkova, Jessica López Espejel, Walid Dahhane, and El Hassane Ettifouri. 2024. Visual grounding for desktop graphical user interfaces. arXiv preprint, arXiv:2407.01558. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems. Bin Fu, Chen Wang, Xin Chen, Yucheng Han, Chi Zhang, Zebiao Huang, Yanda Li, Jiaxuan Liu, Zhao Yang, Furu Wei, and Gang Yu. 2024. Jedi: A generalist agent for desktop interface. arXiv preprint, arXiv:2410.19830. Google. 2024. Highlights from Google I/O 2024. Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In Proceedings of the International Conference on Learning Representations. Izzeddin Gur, Hao Zhu, Frank F. Xu, Shuyan Zhou, Hiroki Furuta, Po-Yu Huang, Yonatan Bisk, Daniel Fried, Ruslan Salakhutdinov, and Graham Neubig. 2024. AITW: A large-scale, time-aware, and real-world benchmark for web agents. arXiv preprint, arXiv:2406.13465. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and 1 others. 2024. CogAgent: A visual language model for GUI agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Wenjia Jiang, Yangyang Zhuang, Chenxi Song, Xu Yang, and Chi Zhang. 2025. AppAgentX: Evolving GUI agents as proficient smartphone users. arXiv preprint, arXiv:2503.02268.
Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. 2025. FerretUI 2: Mastering universal user interface understanding across platforms. In Proceedings of the International Conference on Learning Representations. Accepted paper; arXiv:2410.18967v2.
Dang Nguyen and 1 others. 2024. GUI agents: A survey. arXiv preprint, arXiv:2412.04538. Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, and Wenhao Xu. 2024. MobileFlow: A multimodal LLM for mobile GUI agent. arXiv preprint, arXiv:2407.04346. CC BY 4.0 License. OpenAI. 2024. GPT-4o. Technical report. Yijun Qian, Yujie Lu, Alexander G Hauptmann, and Oriana Riva. 2024. Visual grounding for user interfaces. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Junhong Liang and Bojun Zhang. 2025a. Vision language models are not (yet) spelling correctors. arXiv preprint, arXiv:2509.17418.
Yujia Qin, Yining Ye, Junjie Fang, and Haoming Wang. 2025. UI-TARS: Pioneering automated GUI interaction with native agents. arXiv preprint, arXiv:2501.12326.
Junhong Liang and Bojun Zhang. 2025b. Vision language models are not (yet) spelling correctors. arXiv preprint, arXiv:2509.17418.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shu-Tao Xia, Yong-Dong Zhang, and Jie Tang. 2023. Rethinking agent design: From top-down workflows to bottom-up skill evolution. arXiv preprint, arXiv:2307.07924.
Junhong Liang and Yu Zhou. 2025. RAIR: Retrievalaugmented iterative refinement for chinese spelling correction. arXiv preprint, arXiv:2504.18938. Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. ShowUI: One vision-languageaction model for GUI visual agent. arXiv preprint, arXiv:2411.17465. Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, and Fei Huang. 2025. PCAgent: A hierarchical multi-agent collaboration framework for complex task automation on PC. arXiv preprint, arXiv:2502.14282. Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. 2024a. VisualWebBench: How far have multimodal LLMs evolved in web page understanding and grounding? arXiv preprint, arXiv:2404.05955. Xingwei Liu, Zihan Ye, Jingfeng Zhang, Tianlin Li, Haoming Lu, Yuchen Zhou, Yuji Gao, Dongfang Liu, and DaCheng Tao. 2024b. Agent-Smith: A black-box attack on LLMbased agents. arXiv preprint, arXiv:2405.01957. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. OmniParser for pure vision-based GUI agent. arXiv preprint, arXiv:2408.00203. Siqi Ma, Jiajie Huang, Fan Zhang, Jinlin Wu, Yue Shen, Guohui Fan, Zhu Zhang, and Zelin Zang. 2026. MedLA: A logic-driven multi-agent framework for complex medical reasoning with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 845–853. Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, and 1 others. 2025. UI-Vision: A desktop-centric GUI benchmark for visual perception and interaction. arXiv preprint, arXiv:2503.15661. Anthony Nguyen. 2024. Improved GUI grounding via iterative narrowing. arXiv preprint, arXiv:2411.13591.
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Timothy Lillicrap, and Oriana Riva. 2024. AndroidWorld: A dynamic benchmarking environment for autonomous agents. arXiv preprint, arXiv:2405.14573. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Android in the wild: A largescale dataset for android device control. In Advances in Neural Information Processing Systems. Segev Shlomov, Ben Wiesel, Aviad Sela, Ido Levy, Liane Galanti, and Roy Abitbol. 2024. From grounding to planning: Benchmarking bottlenecks in web agents. arXiv preprint, arXiv:2409.01927. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024a. MobileAgent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint, arXiv:2401.16158. Philipp Wang, Mandi Wang, Yifan Jiang, Ari Holtzman, Caiming Xiong, and Victor Zhong. 2024b. ScreenAgent: A vision-language model-based agent for human-computer interaction. arXiv preprint, arXiv:2406.05459. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. AutoDroid: LLM-powered task automation in android. arXiv preprint, arXiv:2308.15272. Zhiwei Wu, Zekun Qi, Zhaofeng He, Yushi Hu, Junkai Wang, Zhaoyang Zhang, Yining-Gu, Hongcheng-Guo, HangyuLi, Zixuan-Chen, Yao-Mu, Yuzhong-Chen, Jiacheng-Liu, Wen-Guang, Chen, Yujia-Qin, Zhoujun-Cheng, YidongWang, Jindong-Wang, and 8 others. 2024. OS-ATLAS: A foundation action model for generalist GUI agents. arXiv preprint, arXiv:2410.23218. Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, Yiheng Xu, Junli Wang, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2025. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint, arXiv:2505.13227.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024. OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. 2025. An illusion of progress? assessing the current state of web agents. arXiv preprint, arXiv:2504.01382. Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in GPT-4v. arXiv preprint, arXiv:2310.11441. Jingpu Yang, Mingxuan Cui, Hang Zhang, Fengxian Ji, Zhengzhao Lai, and Yufeng Wang. 2025a. Agent-based anti-jamming techniques for UAV communications in adversarial environments: A comprehensive survey. arXiv preprint, arXiv:2508.11687. Jingpu Yang, Hang Zhang, Fengxian Ji, Yufeng Wang, Mingjie Wang, Yizhe Luo, and Wenrui Ding. 2025b. Frequency point game environment for UAVs via expert knowledge and large language model. arXiv preprint, arXiv:2508.02757. Jingpu Yang, Hang Zhang, Fengxian Ji, Yufeng Wang, Mingjie Wang, Yizhe Luo, and Wenrui Ding. 2026. Frequency point game environment for UAVs via expert knowledge and large language model. Drones, 10(2):147. Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2024. Ferret-UI: Grounded mobile UI understanding with multimodal LLMs. arXiv preprint, arXiv:2404.05719. Yiming Zeng, Wanhao Yu, Zexin Li, Tao Ren, Yu Ma, Jinghan Cao, Xiyan Chen, and Tingting Yu. 2025. Bridging the editing gap in LLMs: FineEdit for precise and targeted text modifications. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2193–2206, Suzhou, China. Association for Computational Linguistics. Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Guyue Liu, Qingwei Lin, and 1 others. 2024a. Large language model-brained GUI agents: A survey. arXiv preprint, arXiv:2411.18279. Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, and 1 others. 2025a. UFO: A UI-focused agent for windows OS interaction. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 597–622. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025b. AppAgent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. 2024b. LlamaTouch: A faithful and scalable testbed for mobile UI task automation. arXiv preprint, arXiv:2404.16054.
Yushi Zhang, Zhiwei Zhang, Zekun Qi, Yining Gu, Yining Li, Hongcheng Guo, Yujia Qin, Zhaofeng He, Yidong Wang, Zhoujun Cheng, Jindong Wang, Taro Watanabe, Yutaka Sasaki, Ruoyu Sun, Wei Xue, Tat-Seng Chua, and Xing Xie. 2024c. WorldGUI: A benchmark for evaluating generalist GUI agents on real-world and unseen tasks. arXiv preprint, arXiv:2407.13329. Zhiwei Zhang, Zekun Qi, Yining Gu, Zhaofeng He, Yushi Hu, Yining Li, Hongcheng Guo, Jiacheng Liu, Yujia Qin, Yidong Wang, Zhoujun Cheng, Jindong Wang, Gang Wang, Yutaka Sasaki, Taro Watanabe, Ruoyu Sun, Wei Xue, TatSeng Chua, Rui Zhao, and Xing Xie. 2024d. A-STAR: A benchmark for any-scale task automation on real-world software. arXiv preprint, arXiv:2407.14725. Jing-Yi Zhao, Hong-Quankreston Tran, Yining Li, Tianbao Xie, Zixuan Li, Jia-Qi Li, Xin-Yu Dai, Yujia Qin, RuiZhao, and Zhiyong-Wu. 2024. ScreenSpot-Pro: A benchmark for fine-grained GUI grounding in professional software. arXiv preprint, arXiv:2407.02078. Boyuan Zheng, Boyu Gou, Jinyi Zheng, Huan Wang, Cheng Wang, Weixin Yao, Mengjiao Wang, Kaixin Zheng, Huan Sun, and Yu Su. 2024a. GPT-4v(ision) is a generalist web agent, if grounded. In Proceedings of the International Conference on Machine Learning. Yatong Zheng, Zixuan Li, Ruixiang Zhang, HongQuankreston Tran, Haoxuan You, Xiao-Yong Wei, Yujia Qin, Xin-Yu Dai, Shwai He, Rui-Zhao, YidongWang, Xing-Xie, and Zhiyong-Wu. 2024b. GUI-Robust: A benchmark for evaluating GUI agents’ robustness. arXiv preprint, arXiv:2407.03901. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, and 1 others. 2023. WebArena: A realistic web environment for building autonomous agents. arXiv preprint, arXiv:2307.13854.
A Data Collection and Quality Control Benchmark Composition. FineState-Bench (FineState-Static) contains 2,209 instances across Desktop (810), Web (701), and Mobile (698). Each instance includes a screenshot, an instruction that specifies an exact target state, fine-grained state labels, and geometric annotations (dual bounding boxes). All coordinates are normalized to [0, 1]. Construction Pipeline. We curate the benchmark through LVLM-based pre-filtering and manual verification, following a five-step pipeline: (1) Filtering from dataset: use an LVLM to pre-filter candidates with non-trivial state changes from OS-Atlas; (2) Manual screenshot supplementation: add missing but representative interaction patterns to ensure coverage of all component types; (3) Human annotation: annotate dual bounding boxes and state labels; (4) LLM-assisted drafting: draft candidate instruction–state pairs, then refine them to enforce exact target states; (5) Manual verification:
validate instruction state consistency and bounding box quality, and remove ambiguous or noisy cases. Interaction Taxonomy: 23 Component Types. We group all tasks into four interaction families, covering 23 UI component subtypes (IDs follow the main paper taxonomy). A. Numerical and Range Adjustment (5). A1 Slider: drag along a track to reach a numeric value; A2 Knob: rotate to adjust a scalar value; A3 Stepper: click +/- to change a discrete value; A4 Seek Bar: scrub progress/time to a precise position; A5 Chart Point: select a specific point/value on a chart. B. Toggle and Option Selection (6). B1 Switch: binary on/off toggle; B2 Check Box: checked/unchecked (often multi-select); B3 Radio Group: choose exactly one option; B4 Tabs: switch active tab; B5 Segmented: choose a segment in a segmented control; B6 Accordion: expand/collapse a section. C. Specific Data-type Selection (5). C1 Rating: pick an ordinal rating (e.g., stars); C2 Color Picker: select an exact color (RGB/hex); C3 Date Picker: choose a specific date; C4 Time Picker: choose a specific time; C5 List Box: select an item from a list/dropdown. D. Content Organization and View Manipulation (7). D1 Drag Reorder: drag items to change ordering; D2 Zoom Pan: zoom/pan a canvas/map to a target view; D3 Resizable Pane: drag a pane edge to resize; D4 Carousel: switch the active card/page by swiping/scrolling; D5 Tree View: expand/collapse/select hierarchical nodes; D6 Splitter: drag a divider to adjust layout ratio; D7 Table Column: operate on columns (e.g., reorder/resize/sort).
A1Slider100 C5ListboxDropdown60 B1Switch40 A5ChartPoint40 C3DatePicker40 B2Checkbox40 B3RadioGroup40 C2ColorPicker40 D1DragReorder30 A3Stepper30 D2ZoomPan30 D3ResizablePane25 A2Knob30 B4Tabs30 C4TimePicker30 C1Rating30 D7TableColumn15 B6Accordion25 A4SeekBar40 D5TreeView20 D6Splitter15 B5Segmented25 D4Carousel25
138 129 115 115 112 108 97 91 89 89 89 87 86 81 80 78 77 60 57 53 47 40 0
50
100 150 200 250 300
Number of Tasks
Figure 7: Task distribution over the 23 UI component subtypes in FineState-Bench.
B System Prompts
Placeholders and Conventions. In all prompts below, placeholders in curly braces instruction are runtime-filled fields. IMAGE] denotes the screenshot input. All coordinates are normalized to [0, 1].
B.1 Base System Prompts
B.1.1 General GUI Agent Prompt Component Type Distribution. Figure 9 reports the number of tasks for each of the 23 component subtypes. The distribution is long-tailed: A1 Slider is the most frequent (291), followed by C5 List Box/Dropdown (138) and B1 Switch (129). Precise selection/adjustment interactions such as A5 Chart Point (115) and C3 Date Picker (115) are also well represented, along with B2 Checkbox (112) and B3 Radio Button (108). Meanwhile, rarer interactions such as D4 Carousel (40), B5 Segmented (47), and D6 Splitter (53) are intentionally included to ensure coverage across the full taxonomy.
291
You are a GUI automation agent. Input: one screenshot [IMAGE] and one instruction. Task: return the first interaction point on the target UI control. Rules: - Exact target-state matching (no approximation). - No iterative trial-and-error or multi-step refinement. Output only one point: [x, y] (normalized to [0, 1])
B.1.2
Enhanced Prompt with Component Information
You are a GUI agent for fine-grained state setting. Instruction:
{instruction}
Target component: {component_name} Component type: {component_type} Current state: {current_state} Target state: {target_state} Output only: [x, y]
B.2 B.2.1
Model-Specific Prompts Shared user template (default)
[IMAGE] Instruction: {instruction} Target: change {component_name} from {current_state} to {target_state} Return only: [x, y]
B.2.2
Closed-Source Models
GPT-4o. SYSTEM_PROMPT = """ You are GPT-4o for GUI automation. Given [IMAGE] and an instruction, return the first interaction point [x, y] in [0, 1]. Output only: [x, y] """
Claude-3.5-Sonnet. SYSTEM_PROMPT = """ You are Claude for precise GUI interaction. Given [IMAGE] and an instruction, return the first interaction point [x, y] in [0, 1]. Output only: [x, y] """
Gemini-2.5-Flash. SYSTEM_PROMPT = """ You are Gemini for GUI automation. Given [IMAGE] and an instruction, return the first interaction point [x, y] in [0, 1]. Constraint: exact target-state matching. Output only: [x, y] """
B.2.3
Open-Source Models
OS-Atlas-7B.
SYSTEM_PROMPT = """ You are OS-Atlas, a GUI-specialized vision-language model. Given [IMAGE] and an instruction, return the first interaction point [x, y] in [0, 1] to satisfy the exact state requirement. Output only: [x, y] """
ShowUI-2B. SYSTEM_PROMPT = """ You are ShowUI for GUI interaction. Focus: accurate UI grounding under a single-point protocol. Output only: [x, y] """
B.3 VDA-Enhanced Prompts and Additional Analyses VDA in FineState-Bench. VDA follows the twostep describe-then-localize procedure under the target instruction I 1 : The Describe step generates a structured Description of the target UI element for internal disambiguation; The Localize step predicts a tight Localization Hint B̂ 1 ∈ [0, 1]4 in normalized coordinates, and only B̂ 1 is appended to the agent input when predicting p1 . B.3.1 Describe Step: Description Generation VDA_DESCRIPTION_PROMPT = """ Analyze [IMAGE] and describe the target UI control. Instruction: {instruction} Include: 1) Functional role + visible state (if present) 2) Discriminative visual cues 3) Spatial relations to nearby anchors Return a short description. """
B.3.2 Localize Step: Localization Hint Prediction (BBox Output) VDA_LOCALIZATION_PROMPT = """ Predict a tight bounding box for the interactable core. Instruction: {instruction} Description: {description} Target state: {target_state} Output only: [x1, y1, x2, y2] """
(normalized to [0, 1])
B.3.3 Three Cross-Platform Examples Example 1 (Desktop, A1 Slider).
Instruction: Adjust the volume slider to 77.7%. Current -> Target: 45.2% -> 77.7% Stage-1 (description): Horizontal slider labeled "Volume"; knob on the track.
are comparatively more tractable and yield nontrivial Interact performance for several models. Notably, the near-zero Interact results on the Ctype data selection family (rating/color/date/time/dropdown) highlight a persistent weakness in exact data-type selection under exact-state verification.
Stage-2 (bbox_int): [0.727, 0.550, 0.749, 0.583] Final click point (bbox center): [0.738, 0.567]
Example 2 (Web, C3 Date Picker). Instruction: Select December 25, 2024 from the date picker. Current -> Target: 2024-11-30 -> 2024-12-25
Locate: Show the position on the line chart for September.
1. Component Description: A point on a line graph showing the value 48 in September, marked with an orange dot and yellow indicator 2. Interaction Coordinates: [0.792, 0.183]
Stage-1 (description): Calendar widget; target is the day cell "25" in Dec 2024.
3. Reasoning: The x-coordinate of 0.792 represents September's position on the timeline, while the y-coordinate of 0.183 represents the height of the 48-unit point relative to the
"Locate box": [0.27,0.19,0.91,0.93]
graph's scale."
Interact: Drag it to the vertical axis value of approximately 55. 1. Component Description: A line graph showing monthly water and electricity expenses,
Stage-2 (bbox_int): [0.456, 0.345, 0.478, 0.378]
with interactive data points along the line. 2. Interaction Coordinates: [0.750, 0.167] 3. Reasoning: I've calculated the position at approximately 0.750 horizontally (around
Final click point (bbox center): [0.467, 0.361]
Example 3 (Mobile, B1 Switch). Instruction: Turn on notifications. Current -> Target: OFF -> ON Stage-1 (description): A switch control on the right of the "Notifications" row. Stage-2 (bbox_int): [0.156, 0.478, 0.189, 0.512] Final click point (bbox center): [0.172, 0.495]
C
Supplementary Results for Fig. 6: Locate vs. IntLoc
Table 7 provides the component-level numbers underlying the aggregate trends in Fig. 6. For each UI component category, we report Locate (p1 ∈ Bloc ) and Interact ( p1 ∈ Bint and s1 (c∗ ) = sgoal ) success rates under the single-point protocol, enabling direct comparison of per-component difficulty and cross-model variation. Overall, Interact remains consistently lower than Locate across component types, with the largest gaps concentrated in precision-sensitive or continuous-value interactions (e.g., sliders/knobs/seek bars and view manipulation such as zoom/pan or splitters). Discrete selection primitives (e.g., steppers, switches, radio groups, and tabs)
September position) and 0.167 vertically (55 units from bottom on a 60-unit scale). "
"Interact box": [0.68,0.36,0.71,0.40]
Figure 8: Representative failure cases and diagnostic analysis on FineState-Static.
D Annotation Reliability To improve reproducibility, we provide additional details on how exact state is measured and recorded for each component family, as well as the quality control procedures used during annotation, including spot checks, re-annotation, and adjudication. To quantify annotation reliability, we conducted a small-scale double-annotation study. We randomly sampled a stratified subset of N = 200 SR@Loc SR@Int Diagnosis 0
0/1
1
0
1
1
Control grounding failure: the agent fails to identify the target control in the current UI. Interactable-core miss (precision failure): the agent finds the right control but places p1 outside the state-changing core. Check exact state: if ES-SR@Int= 0, it is a state-setting error beyond core localization; if ES-SR@Int= 1, the exact goal state is achieved.
Table 5: Rule-of-thumb diagnosis using FineStateMetrics.
Metric Result Box agreement (IoU, median) locate box 83.5% Box agreement (IoU, median) interactable-core box 68.0% Goal-state agreement (exact match) overall 97.5% Goal-state agreement categorical components 93.0% Goal-state agreement numeric components 88.5% Disagreement rate requiring adjudication 6.7%
Table 6: Double-annotation reliability results on a stratified subset of 200 instances.
instances across platforms and component types. Two annotators independently labeled the locate box, interactable-core box, and goal-state label. The results are summarized in Table 6. These results provide direct evidence that the fine-grained annotations are sufficiently reliable for evaluation and diagnostic analysis.
E
Failure Cases
Failures in FineState-Bench primarily occur during the transition from coarse localization to interactable-core grounding, where agents successfully identify the target component but miss the precise state-changing region. As illustrated in Fig. 9 and Fig. 8, even a minor coordinate offset on dense professional interfaces can prevent the agent from reaching the exact target state.
Locate: Locate the currently selected item in the left vertical column. 1. Component Description: The "SUMMARY" item in the left vertical column, which appears to be the currently selected item. 2. Interaction Coordinates: [0.230, 0.220]. 3. Reasoning: The instruction asks to locate the currently selected item in the left vertical column. "SUMMARY" is highlighted with a blue background, indicating it is the selected item. The coordinates point to the approximate center of this highlighted element.
“Locate box": [0.19,0.28,0.32,0.95]
Interact: Click the 'BNB' item in the left vertical column. 1. Component Description: The 'BNB' stock ticker item in the left-hand column. 2. Interaction Coordinates: [0.207, 0.583]. 3. Reasoning: This coordinate targets the center of the 'BNB' text and its associated data within the vertical list of suggested stocks.
“Interact box": [0.20,0.53,0.32,0.57]
Figure 9: Representative failure cases and diagnostic analysis on FineState-Static.
F
Metric Interpretation and Failure Attribution Rules
What each metric measures. SR@Loc measures component-level grounding: whether the
agent identifies the target control in the current UI by placing p0 inside the control’s visible ex0 . tent Bloc SR@Int measures interactable-core grounding and point precision: whether the goaldirected operation point p1 hits the state-changing 0 , which is particularly critical for core region Bint precision-sensitive continuous controls (e.g., sliders, seek bars, and pickers). ES-SR@Loc and ESSR@Int additionally require exact goal-state attainment (s1 (c∗ ) = sgoal ). Importantly, ES-SR@Loc conditions on the target-configuration locate box 1 ) to check control identity consistency un(Bloc der potential layout changes, whereas ES-SR@Int conditions on the target-configuration interact box 1 ) to check goal-directed core grounding for ex(Bint act state setting.
Claude-3.5-Sonnet ShowUI-2B UGround-V1-7B OS-Atlas-Base-7B SR@Loc SR@Int SR@Loc SR@Int SR@Loc SR@Int SR@Loc SR@Int A. Numerical and Range Adjustment A1 Slider 23.2 3.2 3.2 0.0 82.1 0.0 49.5 2.1 A2 Knob 31.0 0.0 55.2 0.0 65.5 0.0 69.0 6.9 A3 Stepper 16.7 0.0 20.0 6.7 96.7 80.0 76.7 33.3 0.0 0.0 0.0 0.0 25.0 0.0 62.5 0.0 A4 SeekBar A5 ChartPoint 84.2 0.0 68.4 0.0 73.7 0.0 78.9 2.6 B. Toggle and Option Selection B1 Switch 10.3 12.8 17.9 12.8 64.1 71.8 71.8 48.7 B2 Checkbox 25.0 0.0 36.1 0.0 72.2 30.6 66.7 13.9 67.6 23.5 35.3 14.7 67.6 61.8 70.6 52.9 B3 RadioGroup B4 Tabs 30.0 10.0 46.7 36.7 66.7 80.0 76.7 60.0 B5 Segmented 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 B6 Accordion 17.4 0.0 8.7 0.0 95.7 4.3 69.6 4.3 C. Specific Data-type Selection C1 Rating 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C2 ColorPicker 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C3 DatePicker 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C4 TimePicker 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C5 Dropdown 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 D. Content Organization and View Manipulation 46.7 16.7 33.3 0.0 73.3 6.7 80.0 0.0 D1 DragReorder D2 ZoomPan 25.0 14.3 17.9 21.4 35.7 35.7 64.3 17.9 36.0 0.0 28.0 0.0 28.0 0.0 8.0 0.0 D3 ResizablePane D4 Carousel 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 D5 TreeView 11.8 6.2 29.4 6.2 88.2 6.2 52.9 12.5 D6 Splitter 46.7 6.7 13.3 20.0 40.0 6.7 20.0 0.0 26.7 33.3 40.0 0.0 13.3 0.0 46.7 0.0 D7 TableColumn Component
(a) Closed-source / vision GUI baselines. Gemini-2.5-Flash GPT-4o CogAgent-9B MobileVLM-V2-7B SR@Loc SR@Int SR@Loc SR@Int SR@Loc SR@Int SR@Loc SR@Int A. Numerical and Range Adjustment 54.7 11.6 34.7 4.2 5.3 0.0 0.0 0.0 A1 Slider A2 Knob 65.5 0.0 34.5 0.0 44.8 0.0 44.8 0.0 A3 Stepper 80.0 70.0 16.7 0.0 10.0 0.0 16.7 0.0 68.8 0.0 18.8 0.0 0.0 0.0 0.0 0.0 A4 SeekBar A5 ChartPoint 81.6 2.6 78.9 0.0 57.9 0.0 76.3 0.0 B. Toggle and Option Selection B1 Switch 66.7 53.8 17.9 10.3 10.3 0.0 12.8 0.0 B2 Checkbox 69.4 22.2 41.7 8.3 2.8 0.0 11.1 0.0 B3 RadioGroup 91.2 67.6 64.7 11.8 50.0 2.9 32.4 5.9 B4 Tabs 93.3 56.7 93.3 40.0 0.0 0.0 0.0 0.0 B5 Segmented 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 B6 Accordion 52.2 4.3 30.4 8.7 13.0 0.0 4.3 0.0 C. Specific Data-type Selection C1 Rating 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C2 ColorPicker C3 DatePicker 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 C4 TimePicker C5 Dropdown 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 D. Content Organization and View Manipulation D1 DragReorder 70.0 3.3 46.7 10.0 40.0 3.3 43.3 0.0 D2 ZoomPan 42.9 32.1 28.6 17.9 17.9 14.3 10.7 25.0 D3 ResizablePane 32.0 0.0 32.0 0.0 52.0 0.0 44.0 0.0 D4 Carousel 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 D5 TreeView 41.2 6.2 35.3 12.5 0.0 0.0 0.0 0.0 46.7 0.0 33.3 6.7 46.7 0.0 26.7 20.0 D6 Splitter D7 TableColumn 40.0 0.0 0.0 0.0 66.7 6.7 53.3 0.0 Component
(b) Additional baselines.
Table 7: Component-wise SR@Loc vs. SR@Int success rates (%).