arXiv:2605.00642v1 [cs.AI] 1 May 2026
Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding
Yan Zhang1,3,∗ Daiqing Wu1,3,∗ Huawen Shen1,3 Yu Zhou2,† Can Ma1,3,† 1 Institute of Information Engineering, Chinese Academy of Sciences 2 VCIP & TMCC & DISSec, College of Computer Science, Nankai University 3 School of Cyber Security, University of Chinese Academy of Sciences [email protected]; [email protected]
Abstract Graphical User Interface (GUI) grounding maps natural language instructions to the visual coordinates of target elements and serves as a core capability for autonomous GUI agents. Recent reinforcement learning methods (e.g., GRPO) have achieved strong performance, but they rely on expensive multiple rollouts and suffer from sparse signals on hard samples. These limitations make on-policy self-distillation (OPSD), which provides dense token-level supervision from a single rollout, a promising alternative. However, its applicability to GUI grounding remains unexplored. In this paper, we present GUI-SD, the first OPSD framework tailored for GUI grounding. First, it constructs a visually enriched privileged context for the teacher using a target bounding box and a Gaussian soft mask, providing informative guidance without leaking exact coordinates. Second, it employs entropy-guided distillation, which adaptively weights tokens based on digit significance and teacher confidence, concentrating optimization on the most impactful and reliable positions. Extensive experiments on six representative GUI grounding benchmarks show that GUI-SD consistently outperforms GRPO-based methods and naive OPSD in both accuracy and training efficiency. Code and training data are available at https://zhangyan-ucas.github.io/GUI-SD/.
1
Introduction
Autonomous GUI agents have emerged as a promising direction for human-computer interaction, where GUI grounding serves as the fundamental capability of mapping natural language instructions to visual coordinates of target elements [4, 7]. To this end, a growing body of work [2, 6, 39, 51] has adopted reinforcement learning for GUI grounding, among which GRPO-based methods [19, 55] have become the dominant paradigm as shown in Figure 1(a). Specifically, given a user instruction, GRPO [8, 27] encourages the policy model to explore diverse solutions by sampling multiple rollouts, and evaluates each with a designed verifiable reward, such as binary [19], distance-constrained [46], or gaussian-based feedback [32]. The advantage of each rollout is then computed relative to the group reward distribution, steering the policy to reinforce successful explorations while discouraging unsuccessful ones [48]. Despite the advances, GRPO-based training for GUI grounding still depends on expensive multiple rollouts to estimate advantages and suffers from sparse signals on hard samples where all rollouts receive zero reward. These limitations call for a paradigm that can deliver dense supervision from fewer interactions. Recently emerging on-policy self-distillation (OPSD) [12, 26, 28, 30, 44] offers ∗ Equal contribution † Corresponding authors
Preprint.
✗
✗
Advantagebased Update
Student Logits
……
Reward=0
…
Reward=0
Policy
✗
Distillation-to-SFT Collapse.
Expensive Multiple Rollouts.
…
Reward=0
…
Reward=0
Sparse Signals on Hard Samples. (a). GRPO
… …
Textual Priviledged
+
Reverse KL divergence Visual Priviledged
…
Near-one-hot Teacher Logits
(b). Naive OPSD
+
Weighted Reverse KL divergence
…
Informative Soft Teacher Logits
Adaptive weight
Uniform weight Indiscriminate Optimization.
…
Student Logits
Policy
Policy
✗
Visual Privileged Guidance.
Entropy-Guided Optimization. (c). Ours
Figure 1: (a) GRPO requires expensive multiple rollouts and produces zero reward on hard samples. (b) Naive OPSD forwards the policy twice and distills via reverse KL between student and teacher logits with uniform per-token weight w = 1.0, yet suffers from distillation-to-SFT collapse and indiscriminate optimization. (c) Ours addresses both issues via visual privileged guidance and entropy-guided optimization.
such a possibility, providing token-level supervision from a single rollout by deploying the same model as both teacher and student under asymmetric contexts. Specifically, the asymmetry lies in the privileged information that is accessible to the teacher but hidden from the student, such as reference solutions [28], verifier signals [26], and environment feedback [12]. Guided by this privileged context, the teacher acts as a stronger model, yielding a more reliable output distribution whose per-token log-probabilities form a reverse Kullback-Leibler (KL) divergence loss that continuously refines the student. By replacing sparse outcome-level rewards with dense token-level guidance, OPSD provides an appealing alternative for improving both training efficiency and supervision quality. Motivated by these advantages, we explore for the first time the application of OPSD to GUI grounding. However, as illustrated in Figure 1(b), directly adapting OPSD to this setting encounters two critical bottlenecks: 1) Distillation-to-SFT Collapse. Naive OPSD paradigm directly appends the target coordinate as text to the teacher’s input, causing the teacher’s supervisory distribution to collapse into near-one-hot targets with near-zero entropy. In this regime, minimizing the KL divergence between teacher and student becomes equivalent to minimizing cross-entropy against hard labels, effectively reducing distillation to supervised fine-tuning (SFT) and erasing the dark knowledge [10] that makes soft-label supervision beneficial. 2) Indiscriminate Optimization. Naive OPSD applies reverse KL to distill all tokens uniformly, yet higher-order coordinate digits steer the optimization direction far more effectively than lower-order digits. Furthermore, the teacher’s confidence varies across tokens, and treating all tokens equally propagates unreliable signals from low-confidence positions, leading to sub-optimal gradients. To address these issues, we propose GUI-SD (GUI Grounding via Self-Distillation), an OPSD framework tailored for GUI grounding, which combines visually enriched privileged context with an entropy-guided loss to deliver rich token-level supervision for precise coordinate generation. Specifically, GUI-SD builds the teacher’s privileged context by highlighting the ground-truth region with a bounding box and applying a Gaussian soft mask that gradually fades the surrounding areas. Paired with an instructional hint, this visual prompt delivers informative yet constrained prior knowledge, guiding the teacher to the target without leaking the exact coordinates. Furthermore, GUI-SD introduces entropy-guided distillation, an adaptive objective that replaces uniform token weighting with targeted supervision. It prioritizes higher-order coordinate digits that dominate grounding accuracy while amplifying supervision from confident teacher predictions. Extensive experiments across six representative grounding benchmarks (ScreenSpot-v2 [40], ScreenSpot-Pro [17], UI-Vision [24], MMBench GUI L2 [37], OSWorld-G[41], and OSWorld-GRefine [41]) demonstrate that GUI-SD substantially outperforms GRPO-based methods [19, 32, 46] and naive OPSD [28] in both accuracy and training efficiency. Detailed ablation studies further validate that visually enriched privileged context provides effective teacher guidance, while entropyguided distillation concentrates optimization on the most impactful coordinate tokens. 2
Our main contributions are summarized as follows: • To the best of our knowledge, we present the first exploration of the OPSD framework in the GUI grounding domain, offering an appealing alternative to GRPO-based methods that suffer from expensive multiple rollouts and sparse signals on hard samples. • We propose GUI-SD, which integrates visually grounded teacher guidance with entropyaware distillation, enabling rich and reliable supervision that concentrates optimization on the most impactful coordinate tokens. • Extensive experiments verify the effectiveness of GUI-SD over naive OPSD and GRPObased methods across six representative GUI grounding benchmarks, demonstrating significant improvements in both accuracy and training efficiency, establishing OPSD as a promising paradigm for future GUI grounding research.
2
Preliminary
OPD. While GRPO-type reinforcement learning has driven significant progress in GUI grounding, its sparse sequence-level rewards provide no dense token-level guidance, offer little or even zero feedback on difficult samples, and require heavy online sampling [3]. On-Policy Distillation (OPD) [29] offers an alternative paradigm, where a separate, typically larger, teacher model πθ̂ provides token-level supervision along the student’s sampled trajectories. By distilling the teacher’s output distribution at each decoding step, OPD delivers continuous learning signals that enable more sampleefficient training and meaningful gradient updates even for samples that would otherwise receive no reward. OPSD. To remove the dependence on a separate teacher, On-Policy Self-Distillation (OPSD) [53] deploys the same model πθ as both teacher and student, with the two roles operating under asymmetric contexts. Specifically, the teacher is granted access to privileged information r (e.g., ground-truth answers [28] or verified reasoning traces [26]) that is unavailable to the student, yielding more informative token-level distributions along the student’s sampled trajectories. Formally, given the sample (x, r), where x denotes the input query and r the privileged context, the student generates an on-policy trajectory under x. Meanwhile, the teacher, conditioned on both (x, r), produces step-wise target distributions along the same trajectory. Training then minimizes the per-token divergence between the student and teacher distributions at each decoding step: Student:
PS (yt ) ≜ πθ (yt | x, y<t ),
PT (yt ) ≜ πθ (yt | x, r, y<t ), |y| X 1 DKL PS (yt ) PT (yt ) , L(θ) = Ey∼PS |y| t=1 Teacher:
(1)
where DKL measures the KL divergence, y<t denotes the generated trajectory up to step t, yt denotes the token generated at step t, and |y| is the total length of the trajectory.
3
Empirical Analysis of OPSD for GUI Grounding
To understand the failure of naive OPSD in GUI grounding, we analyze its teacher supervision signal through two complementary entropy-based views. Following prior distillation studies [10, 13, 15, 31, 35], we consider: (1) sample-level entropy, which assesses whether the overall teacher distribution remains informative or collapses toward near-one-hot targets [10, 15, 35]; and (2) token-level entropy, which evaluates the reliability of supervision across coordinate digits [13, 31]. 3.1
Distillation-to-SFT Collapse at Sample-level
Prior OPSD methods provide the teacher with privileged information in textual form, such as reference solutions, verifier signals, or environment feedback. Following this design, a naive adaptation to GUI grounding feeds the ground-truth coordinate directly into the teacher input as text. We evaluate the 3
Table 1: Comparison between naive OPSD teacher supervisory signal and SFT signal. SSP Acc denotes model accuracy on ScreenSpot-Pro.
SFT Signal Teacher Signal (Naive OPSD) Teacher Signal (GUI-SD)
Avg Entropy
Avg Top-1 Prob
SSP Acc
0.00 0.17 0.50
1.00 0.82 0.59
54.1 55.6 60.7
Figure 2: Per-token analysis of teacher and student predictions on incorrectly predicted tokens across digit positions (hundreds, tens, units). Top: teacher vs. student prediction entropy. Bottom: teacher vs. student ground-truth probability. Stars denote the mean teacher and student values for each digit position. resulting teacher signal at the sample level using two statistics over coordinate digits: average entropy and average top-1 probability. As shown in Table 1, the naive OPSD teacher produces a nearly deterministic distribution, with an average entropy of only 0.17 and an average top-1 probability of 0.82. Prior distillation studies [10, 15] suggest that such low-entropy targets behave similarly to hard labels, causing divergence minimization to degenerate toward standard cross-entropy training. This interpretation is consistent with downstream performance: naive OPSD improves over SFT by only 1.5% on ScreenSpot-Pro (55.6 vs. 54.1), indicating that textual privilege yields little benefit beyond hard-label supervision. In contrast, the optimized privileged context in GUI-SD produces a more informative teacher signal, reaching 60.7% on ScreenSpot-Pro, a 6.6% improvement over SFT, demonstrating that a well-designed privilege context can unlock substantial gains beyond hard-label supervision. Finding 1: In GUI grounding, textual privileged information drives the teacher distribution toward near-zero entropy, largely collapsing distillation into near-SFT behavior. We term this failure mode Distillation-to-SFT Collapse. 3.2
Indiscriminate Optimization at Token-level
A second question concerns the token level: under a well-designed privileged context, is the teacher’s supervision equally reliable across coordinate digits? To investigate this, we analyze incorrectly predicted tokens at each digit position, where teacher guidance is most needed. 4
Student Branch
1
5 Inst.
Policy
Original Context �
① Drawn Target Bbox ② Gaussian Soft-mask
Student Pred: [514, 623]
Copy
Inst.
Student Pred:
Hint
+ [514, 623]
Policy
Privileged Context ���� Teacher Branch
(a). Teacher Privileged Guidance
Student Logits:
6
High-order Logits
�� (�� |�<� , �) Reverse KL Divergence
4
w
2
3
Low-order Logits
Token Weight w
✕ digit
entropy
Teacher Logits:
�� (�� |�<� , ���� )
Low Entropy (High Confidence)
High Entropy (Low Confidence)
(b). Entropy-Guided Optimization
Figure 3: Overview of the GUI-SD framework. (a) The teacher branch receives a privileged context xpri , which augments the student’s original input x with a target bounding box, a Gaussian soft mask, and a hint prompt, while the student branch operates on the original context only. (b) GUI-SD trains the student with a weighted reverse-KL objective between teacher and student token distributions, where the token weight w(t) prioritizes high-order tokens via positional credit assignment and filters unreliable supervision via entropy-gated supervision. As shown in Figure 2 (Top), the teacher generally exhibits lower entropy than the student at each digit position, indicating that its token-level preferences are sharper and thus more likely to be amplified during distillation, even on positions where the student predicts incorrectly. Notably, entropy increases from hundreds to tens to units for both teacher and student, suggesting that supervision becomes progressively less certain on lower-order digits. This pattern is echoed in Figure 2 (Bottom), where the teacher assigns higher probability to the ground-truth token than the student at every digit position, with the largest gap observed at the hundreds digit. Finding 2: Teacher supervision in GUI grounding is inherently position-dependent: higher-order digits carry stronger and more reliable signals than lower-order digits. A uniform reverse-KL objective ignores this heterogeneity and therefore not only misallocates optimization budgets, but also risks amplifying erroneous token preferences on more uncertain digit positions. We term this failure mode Indiscriminate Optimization.
4
Method
Building on the above analysis, we propose GUI-SD as a targeted solution to the two failure modes of naive OPSD in GUI grounding. As illustrated in Figure 3, GUI-SD comprises two complementary components. (a) Visual Privileged Guidance replaces textual privilege with a visually enriched privileged context, enabling the teacher to maintain informative soft distributions rather than collapsing toward near-one-hot targets. (b) Entropy-Guided Optimization replaces uniform reverse KL with adaptive token weighting, so that optimization emphasizes high-value coordinate digits while downweighting uncertain supervision. 4.1
Visual Privileged Guidance
Finding 1 shows that textual privileged information collapses the teacher distribution toward nearone-hot targets, reducing distillation to near-SFT behavior. To address this issue, GUI-SD replaces textual coordinate privilege with visually grounded privileged context, allowing the teacher to receive target-aware guidance without direct access to the exact coordinate. Specifically, as shown in Figure 3(a), the teacher input is augmented with a red bounding box over the ground-truth region and a lightweight hint prompt, e.g., “Hint: The answer is located within the red rectangle.” This combination supplies informative yet constrained prior knowledge, guiding the teacher toward the target while preserving non-degenerate soft supervision. In addition, to improve 5
localization in high-resolution GUI scenes with dense layouts, we apply a Gaussian soft mask that progressively attenuates image regions farther from the annotated target, effectively creating a zoom-in effect around the privileged region. The modulation factor for each pixel is defined as: d2 (2) α(x, y) = exp − 2 , 2σ where d denotes the distance from pixel (x, y) to the nearest edge of the ground-truth bounding box (with d = 0 inside the box), and σ is scaled according to the target size with a minimum floor to prevent over-masking small objects. This design preserves full visibility of the target region while smoothly suppressing irrelevant surrounding content. 4.2
Entropy-Guided Optimization
In GUI-SD, the privileged information r in the general OPSD formulation (eq. (1)) is instantiated as the visually enriched context xpri . Conditioned on xpri , the teacher produces step-wise target distributions to provide supervisory signals along the student trajectory. Finding 2 shows that uniform per-token distillation is suboptimal for GUI grounding: coordinate digits differ substantially in both positional importance and supervision reliability. GUI-SD therefore replaces the uniform objective with an entropy-guided weighted reverse-KL loss: |y| X 1 L(θ) = Ey∼PS w(t) · DKL PS (yt ) PT (yt ) , (3) |y| t=1 where w(t) = wpos (t) · went (t).
(4)
Positional Credit Assignment. The first factor, wpos (t), captures the positional significance of coordinate digits. In decimal coordinate prediction, an error in a higher-order digit induces a much larger spatial deviation than an error in a lower-order digit. For example, an incorrect hundreds digit may introduce roughly 100 pixels of error, whereas an incorrect units digit affects only about 1 pixel. To reflect this positional asymmetry, we assign decaying weights from the most significant digit exponentially to the least significant digit: wpos (t) = α · kt ,
(5)
where t is the token index in the generated sequence, kt ∈ {1, 2, 3, 4} denotes the digit position of token t (1 for units, 2 for tens, 3 for hundreds, 4 for thousands), and α > 0 is a scaling factor. For non-numeric tokens, we set wpos (t) = 1. Entropy-Gated Supervision. The second factor, went (t), captures the reliability of teacher supervision. Since the teacher’s confidence varies across tokens, uniformly distilling all positions may over-amplify uncertain teacher preferences and propagate unreliable gradients. We therefore use the entropy of the teacher distribution to modulate supervision strength: H(pT (· | xpri , y<t )) went (t) = exp − , (6) τ where H(·) denotes the per-token entropy of the teacher distribution and τ is a scaling factor. Tokens with low entropy receive stronger supervision, whereas uncertain predictions are automatically down-weighted.
5
Experiment
5.1
Experimental Setup
We conduct experiments on top of Qwen3-VL-Instruct-8B [1], using approximately 7K samples drawn from the ScaleCUA GUI datasets [20] for training. We compare GUI-SD against the following baselines. GRPO-Binary is a standard RLVR method [19] that uses a sparse reward, yielding 1 when 6
Table 2: GUI grounding accuracy on six benchmarks including ScreenSpot-V2 (SS2), ScreenSpot-Pro (SSP), MMBenchGUI (MMG), UI-Vision (UIV), OSWorld-G (OSW-G), and OSWorld-G_R (OSWGR). Bold indicates the best results. Training time is measured in hours per epoch on 8×A800-80G. The detailed experimental results on each benchmark are in the Section C.1. Method UI-TARS-7B [25] GTA1-7B [45] GUI-Actor-7B [39] TongUI-7B [50] JEDI-7B [41] InfiGUI-R1-7B [19] SE-GUI-7B [46] LPO-7B [33] GUI-G2 -7B [32] HyperClick-7B [51] UI-Ins-7B [3] ZoomUI-7B [21] ZwZ-8B [38] MolmoWeb-Ground-8B [9] Propose-then-Critic-8B [36] Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32] + Ours
Time/epoch
SSP
SS2
UIV
OSW-G
OSW-GR
MMG
Avg.
-
35.7 50.1 44.6 24.7 39.5 35.7 47.3 47.5 48.2 52.2 52.8 56.8 58.7
91.6 92.4 89.5 88.7 91.7 90.3 90.5 93.3 93.7 94.0 91.8 91.3
17.6 18.0 25.2 25.7 27.1 28.9
55.1 54.2 60.0 59.6
67.7 69.0 -
79.6 83.1 72.8 78.4
-
16.9 16.7 16.8 4.2
53.6 56.8 56.6 57.4 60.7
93.2 94.6 93.8 94.0 95.1
25.2 27.6 27.5 28.2 33.3
58.7 61.2 62.1 61.9 64.0
67.4 68.6 69.9 70.0 70.9
83.0 84.3 83.3 83.7 86.7
63.5 65.5+2.0 65.5+2.0 65.9+2.4 68.4+4.9
the prediction falls inside the target bounding box and 0 otherwise. GRPO-Distance computes a dense reward based on the normalized distance between the click point and the center point of the target bounding box, following SE-GUI [47]. GRPO-Gaussian models GUI elements as continuous Gaussian distributions to provide dense reward signals, following GUI-G2 [32]. Details of the training setting of GUI-SD are provided in Appendix B. We comprehensively evaluate GUI-SD on six representative GUI grounding benchmarks: ScreenSpotv2 [40], ScreenSpot-Pro [17], UI-Vision [24], MMBench GUI L2 [37], OSWorld-G [41], and OSWorld-G-Refine [41]. Together, these benchmarks cover diverse application scenarios, hierarchical instruction following, and cross-platform generalization across different operating systems. More details on each benchmark are provided in Appendix A. 5.2
Main Results
Comparisons with Baselines. Table 2 presents the evaluation results across six representative GUI grounding benchmarks. GUI-SD achieves the highest average accuracy while demonstrating superior training efficiency compared to existing GRPO-based methods. We attribute this superiority to dense token-level credit assignment and requiring only a single rollout of the policy model instead of multiple. The former provides fine-grained supervision at every decoding step, particularly beneficial on ScreenSpot-Pro and UI-Vision, where small targets on high-resolution screenshots and diverse desktop layouts across 83 applications both demand precise spatial reasoning, achieving gains of +3.3% and +5.1% over GRPO-Gaussian, respectively. The latter substantially reduces training cost, achieving approximately 4× faster training per epoch compared to GRPO-based methods. Comparisons with SOTA Methods. GUI-SD surpasses existing GUI grounding methods in average accuracy across six representative benchmarks. Specifically, on ScreenSpot-Pro, GUI-SD achieves 60.7%, outperforming Propose-then-Critic [36] (58.7%), which relies on test-time scaling by generating multiple candidate actions over sub-regions and selecting via a critic model, incurring substantial inference overhead. On OSWorld-G-Refine, GUI-SD reaches 70.9%, surpassing ZwZ [38] (69.0%), which leverages large teacher models (e.g., Qwen3-VL-235B [1]) to distill zoom-in supervision into 7
Table 3: Ablation studies on teacher privileged context. We evaluate student performance on ScreenSpot-Pro (SSP) under varying teacher guidance settings, along with teacher supervised signal quality: sample accuracy (Acc), sample-averaged entropy (Ent.), and sample-averaged top-1 probability (Top-1). ∆ OPSD denotes the performance difference relative to Naive OPSD (Row 2). “Orig.” denotes Original, “Inst.” is Instruction, and “Gauss.” is Gaussian. Teacher’s Guidance Setting Visual Context
Student
Teacher Signal
Text Context
SSP
∆ OPSD
Acc
Ent.
Top-1
Orig. Image
Inst.
53.0
-2.6
52.0
0.59
0.53
Orig. Image
Inst. + Text BBox
55.6
0
93.0
0.17
0.82
Orig. Image + Drawn BBox
Inst. + Drawn Hint
59.9
+4.3
89.8
0.53
0.61
Gauss. Zoom + Drawn BBox
Inst. + Drawn Hint
60.7
+5.1
99.6
0.50
0.59
Table 4: Ablation study of entropy-guided distillation components. We report ScreenSpot-Pro overall accuracy, hundreds-digit accuracy, and performance on the hard subset, which consists of samples that the base model fails to predict correctly across all 8 rollouts. Entropy-gated and Positional Credit denote entropy-gated supervision and positional credit assignment, respectively. Entropy-gated
Positional Credit
Screenspot-Pro
Hundreds-digit Accuracy
Hard Subset
✗ ✗ ✓ ✓
✗ ✓ ✗ ✓
58.3 59.6 59.2 60.7
77.0 78.7 78.0 79.7
17.5 19.2 19.9 21.1
training, requiring significantly more computational resources for data synthesis. Notably, GUI-SD achieves these improvements through a self-distillation framework that uses only the policy model itself as the teacher, without relying on test-time scaling or external large-scale models, demonstrating that dense token-level supervision from a well-designed privileged context can be more effective. 5.3
Ablation Studies
Effectiveness of Teacher Visual Context. As shown in Table 3, we evaluate student performance on ScreenSpot-Pro under varying teacher guidance settings, along with teacher signal quality (sample accuracy, entropy, and top-1 probability). In Row 1, the teacher and student receive identical inputs without any privilege, yielding a teacher accuracy of only 52.0% and the weakest student score (53.0%), confirming that asymmetric context between teacher and student is critical for effective self-distillation. Row 2 presents Naive OPSD, which appends the ground-truth coordinate as text to the teacher’s input, achieving a student score of 55.6% on ScreenSpot-Pro. We further analyze the teacher’s supervisory signal and find that, although the teacher accuracy reaches 93.0%, the average entropy drops to merely 0.17 with a top-1 probability of 0.82, indicating a near-collapsed distribution that reduces soft-label distillation to hard-label SFT. To address the distribution collapse caused by textual privilege, Row 3 delivers the ground-truth information through the visual channel instead — drawing a bounding box on the teacher’s input image and pairing it with an instructional hint. This forces the teacher to reason over the image rather than simply copying the answer, raising the entropy to 0.53 and producing a substantially softer distribution that enables richer gradient signals, lifting the student to 59.9% (+4.3% over Naive OPSD). Row 4 further introduces a Gaussian soft-mask zoom-in that gradually suppresses surrounding content, enabling the teacher to better focus on the target region especially in high-resolution GUIs with dense layouts. This pushes teacher accuracy to 99.6% while maintaining a healthy entropy of 0.50, achieving the best student performance of 60.7% (+5.1% over Naive OPSD). Effectiveness of Entropy-guided Optimization As shown in Table 4, we ablate the individual components of entropy-guided distillation and report performance on ScreenSpot-Pro, its corresponding hard subset, and hundreds-digit accuracy. The first row serves as the baseline optimized solely with the standard reverse KL loss, yielding the lowest performance across all metrics. Introducing the digit-position weighting (Row 2) directly improves the hundreds-digit accuracy from 77.0% to 78.7%, which consequently lifts the overall ScreenSpot-Pro score to 59.6%, confirming that the 8
(a). Hundreds-digit accuracy
(b). Overall sample accuracy
Figure 4: Training dynamics of GUI-SD, Standard Reverse KL, and GRPO-Gaussian over optimization steps within a single epoch. (a) Hundreds-digit accuracy. (b) Overall sample accuracy. targeted optimization of leading digits is highly effective for GUI grounding. Incorporating only the entropy-guided weighting (Row 3) steadily improves the overall score to 59.2% and yields substantial gains on the hard subset (from 17.5% to 19.9%). Finally, combining both components (Row 4) yields complementary gains, achieving the best results across the board: 60.7% on ScreenSpot-Pro, 79.7% on hundreds-digit accuracy, and a substantial jump to 21.1% on the hard subset. 5.4
Training Dynamics
Figure 4 illustrates the training dynamics over the optimization steps within a single epoch, comparing GUI-SD against two baselines: GRPO-Gaussian and an ablated variant without entropy-guided optimization. As shown in Figure 4(a), GUI-SD achieves significantly higher hundreds-digit accuracy than both GRPO-Gaussian and the ablated variant, which we attribute to positional credit assignment that concentrates optimization on leading digits and entropy-gated supervision that amplifies high-confidence teacher signals. As shown in Figure 4(b), this improved precision on leading digits translates directly into superior overall sample accuracy. Notably, GUI-SD reaches higher performance in substantially fewer training steps than GRPO-Gaussian, validating the efficiency of dense token-level supervision over sequence-level rewards for GUI grounding.
6
Related Work
6.1
On-Policy Self Distillation
OPD has recently emerged as an effective on-policy training paradigm for delivering rich, token-level feedback [18, 29]. Specifically, the student first generates a rollout, which is subsequently fed into a stronger teacher model to provide step-by-step guidance. OPSD eliminates the need for an external teacher by having the same model serve both roles, conditioned solely on different input contexts, such as reference solutions, verifier signals, and environment feedback. A representative work, SDPO [12], exemplifies this by casting the feedback-conditioned model as a self-teacher, distilling its enriched next-token predictions back into the student policy. RLVR [44] extends this by leveraging self-distillation to obtain token-level supervision for fine-grained update magnitudes, concurrently deriving reliable update directions from environmental feedback. However, previous explorations of OPSD are predominantly confined to the natural language domain. When directly applied to visual tasks, especially GUI grounding, OPSD encounters critical issues such as distillation-to-SFT collapse, as detailed in Section 3. 6.2
GUI Grounding via Verifiable Reinforcement Learning
Reinforcement learning with verifiable rewards (RLVR) methods, such as GRPO, have become an effective paradigm for post-training reasoning models by enabling them to autonomously explore solution trajectories under verifiable feedback [8, 16, 43]. Recent efforts have successfully extended this paradigm to GUI grounding [5, 11, 14, 49, 54]. As pioneering efforts, GUI-R1 [23] and UIR1 [22] adopt a binary reward that yields 1 when the predicted coordinate falls inside the target 9
bounding box and 0 otherwise. To mitigate the limitations of such binary feedback, GUI-G1 [55] introduces dense rewards derived from bounding boxes with size-based difficulty coefficients. GUIG2 [32] further refines this by modeling GUI elements as continuous Gaussian distributions for precise spatial alignment. Despite these improvements, GRPO-based training in GUI grounding remains hindered by expensive multiple rollouts, sparse signals on hard samples, and heavy reliance on manually designed rewards.
7
Conclusion and Limitations
In this paper, we present GUI-SD, the first exploration of on-policy self-distillation for GUI grounding, which integrates visually grounded teacher guidance with entropy-guided optimization to deliver targeted token-level supervision for precise coordinate generation. Extensive evaluations across six benchmarks demonstrate that GUI-SD substantially outperforms both GRPO-based methods [19, 32, 46] and naive OPSD [28, 42, 52] in accuracy while achieving approximately 4× faster training. One limitation of our current work is that we have not yet explored scaling to larger model sizes or evaluating compatibility with other model families beyond Qwen3-VL. A promising avenue for future work is extending GUI-SD to long-horizon GUI agent tasks, where multi-step planning and sequential interactions introduce additional challenges for token-level supervision.
References [1] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [2] Chen Chen, Jiawei Shao, Dakuan Lu, Haoyi Hu, Xiangcheng Liu, Hantao Yao, and Wu Liu. Gui-eyes: Tool-augmented perception for visual grounding in gui agents. arXiv preprint arXiv:2601.09770, 2026. [3] Liangyu Chen, Hanzhang Zhou, Chenglin Cai, Jianan Zhang, Panrong Tong, Quyu Kong, Xu Zhang, Chen Liu, Yuqi Liu, Wenxuan Wang, et al. Ui-ins: Enhancing gui grounding with multi-perspective instruction-as-reasoning. arXiv preprint arXiv:2510.20286, 2025. [4] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Li YanTao, Jianbing Zhang, and Zhiyong Wu. Seeclick: Harnessing gui grounding for advanced visual gui agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9313–9332, 2024. [5] Sicheng Fan, Qingyun Shi, Shengze Xu, Shengbo Cai, Tieyong Zeng, Li Ling, Yanyi Shang, and Dehan Kong. Webfactory: Automated compression of foundational language intelligence into grounded web agents. arXiv preprint arXiv:2603.05044, 2026. [6] Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding to novel environments via autonomous exploration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33249–33266, 2025. [7] Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05243, 2024. [8] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [9] Tanmay Gupta, Piper Wolters, Zixian Ma, Peter Sushko, Rock Yuren Pang, Diego Llanes, Yue Yang, Taira Anderson, Boyuan Zheng, Zhongzheng Ren, et al. Molmoweb: Open visual web agent and open data for the open web. arXiv preprint arXiv:2604.08516, 2026. [10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 10
[11] Kun Huang, Weikai Xu, Yuxuan Liu, Quandong Wang, Pengzhi Gao, Wei Liu, Jian Luan, Bin Wang, and Bo An. Mobileipl: Enhancing mobile agents thinking process via iterative preference learning. arXiv preprint arXiv:2505.12299, 2025. [12] Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learning via self-distillation. arXiv preprint arXiv:2601.20802, 2026. [13] Seongryong Jung, Suwan Yoon, DongGeon Kim, and Hwanhee Lee. Todi: Token-wise distillation via fine-grained divergence control. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8089–8102, 2025. [14] Weitai Kang, Bin Lei, Gaowen Liu, Caiwen Ding, and Yan Yan. Guirlvg: Incentivize gui visual grounding via empirical exploration on reinforcement learning. arXiv preprint arXiv:2508.04389, 2025. [15] Jeonghyun Kim, SooKyung Kim, Richeng Xuan, and Hyunsoo Cho. Trust the uncertain teacher: distilling dark knowledge via calibrated uncertainty. arXiv preprint arXiv:2602.12687, 2026. [16] Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer use agents. arXiv preprint arXiv:2508.14040, 2025. [17] Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. Screenspot-pro: Gui grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778– 8786, 2025. [18] Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huan-ang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. [19] Yuhang Liu, Pengxiang Li, Congkai Xie, Xavier Hu, Xiaotian Han, Shengyu Zhang, Hongxia Yang, and Fei Wu. Infigui-r1: Advancing multimodal gui agents from reactive actors to deliberative reasoners. arXiv preprint arXiv:2504.14239, 2025. [20] Zhaoyang Liu, JingJing Xie, Zichen Ding, Zehao Li, Bowen Yang, Zhenyu Wu, Xuehui Wang, Qiushi Sun, Shi Liu, Weiyun Wang, et al. Scalecua: Scaling open-source computer use agents with cross-platform data. arXiv preprint arXiv:2509.15221, 2025. [21] Ziwei Liu, Tao Feng, Borui Kang, Yanbing Yang, and Jun Luo. Zoom to essence: Trainless gui grounding by inferring upon interface elements. arXiv preprint arXiv:2603.14448, 2026. [22] Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Pengxiang Zhao, Guangyi Liu, et al. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 17608–17616, 2026. [23] Run Luo, Lu Wang, Wanwei He, Longze Chen, Jiaming Li, and Xiaobo Xia. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458, 2025. [24] Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M Tamer Özsu, Aishwarya Agrawal, David Vazquez, et al. Uivision: A desktop-centric gui benchmark for visual perception and interaction. arXiv preprint arXiv:2503.15661, 2025. [25] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. [26] Yuxiao Qu, Amrith Setlur, Virginia Smith, Ruslan Salakhutdinov, and Aviral Kumar. Pope: Learning to reason on hard problems via privileged on-policy exploration. arXiv preprint arXiv:2601.18779, 2026. 11
[27] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [28] Idan Shenfeld, Mehul Damani, Jonas Hübotter, and Pulkit Agrawal. Self-distillation enables continual learning. arXiv preprint arXiv:2601.19897, 2026. [29] Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models. arXiv preprint arXiv:2604.00626, 2026. [30] Yuda Song, Lili Chen, Fahim Tajwar, Remi Munos, Deepak Pathak, J Andrew Bagnell, Aarti Singh, and Andrea Zanette. Expanding the capabilities of reinforcement learning via text feedback. arXiv preprint arXiv:2602.02482, 2026. [31] Chi-Ping Su, Ching-Hsun Tseng, Bin Pu, Lei Zhao, Jiewen Yang, Zhuangzhuang Chen, and Shin-Jye Lee. Ea-kd: Entropy-based adaptive knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 731–740, 2025. [32] Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, et al. Gui-g2 : Gaussian reward modeling for gui grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 33214–33222, 2026. [33] Jiaqi Tang, Yu Xia, Yi-Feng Wu, Yuwei Hu, Yuhui Chen, Qing-Guo Chen, Xiaogang Xu, Xiangyu Wu, Hao Lu, Yanqing Ma, et al. Lpo: Towards accurate gui agent interaction via location preference optimization. arXiv preprint arXiv:2506.09373, 2025. [34] Venus Team, Changlong Gao, Zhangxuan Gu, Yulin Liu, Xinyu Qiu, Shuheng Shen, Yue Wen, Tianyu Xia, Zhenyu Xu, Zhengwen Zeng, et al. Ui-venus-1.5 technical report. arXiv preprint arXiv:2602.09082, 2026. [35] Hao Wang, Hao Gu, Hongming Piao, Kaixiong Gong, Yuxiao Ye, Xiangyu Yue, Sirui Han, Yike Guo, and Dapeng Wu. Learning while staying curious: Entropy-preserving supervised finetuning via adaptive self-distillation for large reasoning models. arXiv preprint arXiv:2602.02244, 2026. [36] Wenkai Wang, Xiyun Li, Hongcan Guo, Wenhao Yu, Tianqing Fang, Haitao Mi, Dong Yu, and Shengyu Zhang. Measure twice, click once: Co-evolving proposer and visual critic via reinforcement learning for gui grounding. arXiv preprint arXiv:2604.21268, 2026. [37] Xuehui Wang, Zhenyu Wu, JingJing Xie, Zichen Ding, Bowen Yang, Zehao Li, Zhaoyang Liu, Qingyun Li, Xuan Dong, Zhe Chen, et al. Mmbench-gui: Hierarchical multi-platform evaluation framework for gui agents. arXiv preprint arXiv:2507.19478, 2025. [38] Lai Wei, Liangbo He, Jun Lan, Lingzhong Dong, Yutong Cai, Siyuan Li, Huijia Zhu, Weiqiang Wang, Linghe Kong, Yue Wang, et al. Zooming without zooming: Region-to-image distillation for fine-grained multimodal perception. arXiv preprint arXiv:2602.11858, 2026. [39] Qianhui Wu, Kanzhi Cheng, Rui Yang, Chaoyun Zhang, Jianwei Yang, Huiqiang Jiang, Jian Mu, Baolin Peng, Bo Qiao, Reuben Tan, et al. Gui-actor: Coordinate-free visual grounding for gui agents. arXiv preprint arXiv:2506.03143, 2025. [40] Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv. org/abs/2410.23218. [41] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint arXiv:2505.13227, 2025. [42] Tianbao Xie, Jiaqi Deng, Xiaochuan Li, Junlin Yang, Haoyuan Wu, Jixuan Chen, Wenjing Hu, Xinyuan Wang, Yuhui Xu, Zekun Wang, et al. Scaling computer-use grounding via user interface decomposition and synthesis. arXiv preprint arXiv:2505.13227, 2025. 12
[43] Yifan Xu, Xiao Liu, Xinghan Liu, Jiaqi Fu, Hanchen Zhang, Bohao Jing, Shudan Zhang, Yuting Wang, Wenyi Zhao, and Yuxiao Dong. Mobilerl: Online agentic reinforcement learning for mobile gui agents. arXiv preprint arXiv:2509.18119, 2025. [44] Chenxu Yang, Chuanyu Qin, Qingyi Si, Minghui Chen, Naibin Gu, Dingyu Yao, Zheng Lin, Weiping Wang, Jiaqi Wang, and Nan Duan. Self-distilled rlvr. arXiv preprint arXiv:2604.03128, 2026. [45] Yan Yang, Dongxu Li, Yutong Dai, Yuhao Yang, Ziyang Luo, Zirui Zhao, Zhiyuan Hu, Junzhe Huang, Amrita Saha, Zeyuan Chen, et al. Gta1: Gui test-time scaling agent. arXiv preprint arXiv:2507.05791, 2025. [46] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505.12370, 2025. [47] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505.12370, 2025. [48] Xinbin Yuan, Jian Zhang, Kaixin Li, Zhuoxuan Cai, Lujian Yao, Jie Chen, Enguang Wang, Qibin Hou, Jinwei Chen, Peng-Tao Jiang, et al. Enhancing visual grounding for gui agents via self-evolutionary reinforcement learning. arXiv preprint arXiv:2505.12370, 2025. [49] Xiangjian Zeng, Wenjing Li, Qingqiang Wu, and Liang Zhang. Fdc-ground: Improving grpo for gui grounding via exponential rewards and fact-aligned pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 28122–28130, 2026. [50] Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, Song-Chun Zhu, and Qing Li. Tongui: Building generalized gui agents by learning from multimodal web tutorials. arXiv e-prints, pages arXiv–2504, 2025. [51] Shaojie Zhang, Pei Fu, Ruoceng Zhang, Jiahui Yang, Anan Du, Xiuwen Xi, Shaokang Wang, Ying Huang, Bin Qin, Zhenbo Luo, et al. Hyperclick: Advancing reliable gui grounding via uncertainty calibration. arXiv preprint arXiv:2510.27266, 2025. [52] Shaojie Zhang, Ruoceng Zhang, Pei Fu, Shaokang Wang, Jiahui Yang, Xin Du, Shiqi Cui, Bin Qin, Ying Huang, Zhenbo Luo, et al. Btl-ui: Blink-think-link reasoning model for gui agent. arXiv preprint arXiv:2509.15566, 2025. [53] Xinsen Zhang, Zhenkai Ding, Tianjun Pan, Run Yang, Chun Kang, Xue Xiong, and Jingnan Gu. Opsdl: On-policy self-distillation for long-context language models. arXiv preprint arXiv:2604.17535, 2026. [54] Yuan Zhao, Hualei Zhu, Tingyu Jiang, Shen Li, Xiaohang Xu, and Hao Henry Wang. Coepg: A framework for co-evolution of planning and grounding in autonomous gui agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 36582–36590, 2026. [55] Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents. arXiv preprint arXiv:2505.15810, 2025.
13
A
Evaluation Benchmarks
ScreenSpot-v2 [40]. ScreenSpot-v2 is a refined version of the original ScreenSpot benchmark [4], designed to address annotation ambiguities in earlier versions. It covers mobile, desktop, and web platforms, with each sample consisting of a GUI screenshot paired with a natural language instruction and a ground-truth bounding box. ScreenSpot-v2 is widely adopted as a standard benchmark for general-purpose GUI grounding evaluation. ScreenSpot-Pro [17]. ScreenSpot-Pro focuses on the under-explored challenge of grounding in professional, high-resolution software environments. It comprises 1,581 instructions captured from 23 real-world applications spanning five professional industries, including development tools (e.g., VSCode, PyCharm), creative applications (e.g., Photoshop, Premiere), CAD/engineering software (e.g., AutoCAD, SolidWorks), scientific tools (e.g., MATLAB, Stata), and office software (e.g., Word, Excel), across three operating systems (Windows, macOS, Linux). The central challenge of ScreenSpot-Pro is the extremely small target size: UI elements occupy on average only 0.07% of the high-resolution screenshot area. Dense and complex interface layouts further increase the difficulty of precise localization, making ScreenSpot-Pro one of the most demanding grounding benchmarks available. UI-Vision [24]. UI-Vision is the largest desktop-centric GUI benchmark to date, spanning 83 opensource desktop applications across six domains: Productivity, Development, Creativity, Education, Browsers, and Entertainment. We evaluate on its Element Grounding benchmark, which contains over 8,200 query-label pairs with high-quality human-annotated bounding boxes. A distinguishing aspect of UI-Vision is its cross-application diversity, requiring GUI agents to generalize across highly varied software interfaces and interaction patterns, exposing limitations in spatial reasoning and professional software understanding. OSWorld-G and OSWorld-G-Refine [41]. OSWorld-G is a comprehensive GUI grounding benchmark set in the Linux environment, comprising 564 finely annotated samples that cover 32 distinct UI element types. OSWorld-G captures diverse real-world computer-use interactions, requiring software knowledge, layout understanding, and fine-grained operations. The benchmark organizes tasks into four categories: text matching, element recognition, layout understanding, and precise operation. OSWorld-G-Refine is a refined version that rewrites instructions to remove domain-specific knowledge, isolating the model’s pure spatial grounding ability from its software understanding. MMBench GUI L2 [37]. MMBench-GUI is a hierarchical, cross-platform benchmark for evaluating GUI automation agents across six platforms: Windows, macOS, Linux, iOS, Android, and Web. The benchmark is organized into four ascending capability levels: L1-Content Understanding, L2-Element Grounding, L3-Task Automation, and L4-Task Collaboration. In our evaluation, we adopt the L2-Element Grounding level, which assesses the model’s ability to localize target GUI elements from natural language instructions. L2 includes both basic and advanced difficulty tiers with diverse instruction styles (e.g., action descriptions, target element descriptions, and refusal cases). Its unique contribution lies in enabling consistent cross-platform comparison under a unified evaluation protocol, revealing how models handle the varying interface designs and visual layouts across different operating systems.
B
Training Details.
Training Data. Our training data is sourced entirely from the grounding training subset of ScaleCUA [20]. Since the original annotations contain labeling errors and lack instruction diversity, we apply a two-stage data curation pipeline. First, we leverage UI-Venus1.5-8B [34] to filter the dataset, retaining only samples where its prediction agrees with the original annotation, removing noisy labels. Second, we employ Qwen3-VL-8B [1] to rewrite the original instructions into diverse paraphrases, enriching instruction variety. After filtering and rewriting, approximately 7K samples remain for training. Hyperparameters.
The hyper-parameter details for GUI-SD are provided in Table 5 14
Table 5: Hyperparameter settings for training. Hyperparameter
Value
Learning Rate Per Device Train Batch Size Gradient Accumulation Steps Number of Training Epochs Warmup Ratio Maximum Sequence Length Maximum Completion Length EMA Decay Coefficient DeepSpeed (Student) DeepSpeed (Teacher) σ Scale Factor Minimum σ Floor
from 2.5e-6 to 0 1 16 1 0.05 20000 128 0.95 ZeRO-2 ZeRO-3 1.5 √ 0.1 · min(W, H)
Table 6: Performance comparison on the ScreenSpot-Pro benchmark [17]. Methods
CAD
Development
Creative
Scientific
Office
OS
Avg.
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Text
Icon
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
58.38 63.45 62.44 62.44
14.06 20.31 18.75 21.88
79.22 80.52 82.47 84.42
24.83 31.72 29.66 30.34
69.70 70.71 70.71 71.21
17.48 18.88 19.58 20.28
76.39 79.17 78.47 79.86
27.27 31.82 33.64 34.55
80.79 84.18 83.62 84.18
35.85 37.74 39.62 39.62
71.03 72.90 71.03 71.96
26.97 30.34 30.34 29.21
53.57 56.80 56.61 57.37
+ Ours
67.01
31.25
83.77
35.86
73.74
22.38
82.64
37.27
84.75
52.83
72.90
37.08
60.72
C
Additional Experiments and Ablations
C.1
Detailed Benchmark Results
We provide per-category results for each evaluation benchmark. Table 6 reports ScreenSpot-Pro results across six professional domains (CAD, Development, Creative, Scientific, Office, OS), split by Text and Icon targets. Table 7 reports ScreenSpot-v2 results across Mobile, Desktop, and Web platforms. Table 8 reports UI-Vision results across Basic, Functional, and Spatial grounding tasks. Table 9 reports MMBench GUI L2 results across six operating systems (Windows, macOS, Linux, iOS, Android, Web), split by Basic and Advanced difficulty. Table 10 and Table 11 report OSWorld-G and OSWorld-G-Refine results across four task types (Text Matching, Element Recognition, Layout Understanding, Fine-Grained Manipulation). GUI-SD consistently outperforms all GRPO baselines across the majority of sub-categories. C.2
Ablation on Visual Privilege Design
Beyond the main ablation in Table 3, we further compare different visual privilege designs in Table 12. The first row is the Naive OPSD baseline that appends the target coordinate as text. The second row replaces textual privilege with a standard zoom that masks all regions outside a fixed-size area centered on the target while drawing a bounding box on it. The third row uses our adaptive zoom with a hard mask that adjusts the visible region proportionally to the ground-truth bounding box size. Both visual privilege variants substantially outperform Naive OPSD (+4.5 and +4.7), confirming that delivering ground-truth information through the visual channel is critical. The adaptive zoom achieves a further gain over standard zoom, as its flexible masking better adapts to varying target sizes across different GUI layouts. C.3
The Self-teacher Improves during Training
A key design choice in GUI-SD is how the teacher model is maintained during training. Unlike off-policy distillation where the teacher is typically frozen, on policy self-distillation allows the teacher to evolve alongside the student. As shown in Table 13, using the current policy qθ directly as the teacher yields 59.4%. In this setting, the teacher updates simultaneously with the student at every 15
Table 7: Performance comparison on the Screenspotv2 benchmark [40]. Mobile
Methods
Desktop
Web
Avg.
Text
Icon
Text
Icon
Text
Icon
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
97.93 99.66 99.66 99.66
88.63 89.57 88.63 89.10
96.91 98.97 97.94 97.42
89.29 89.29 89.29 89.29
95.30 96.15 96.15 95.73
87.68 90.15 86.70 88.67
93.16 94.58 93.71 93.95
+ Ours
99.66
89.57
97.42
92.86
97.01
91.13
95.05
Table 8: Performance comparison on the UI-Vision benchmark [24]. Methods
Basic
Functional
Spatial
Avg.
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
30.53 34.03 34.09 35.16
31.88 34.59 33.97 34.76
14.16 15.35 15.45 15.76
25.19 27.61 27.47 28.18
+ Ours
41.87
39.39
19.79
33.27
step, resulting in minimal divergence between their distributions and weakening the distillation signal. Freezing the teacher at initialization (qθref ) performs similarly at 59.6%, as the fixed teacher quickly becomes outdated once the student improves beyond its initial capacity. Exponential moving average (EMA) with a decay of 0.95 achieves the best result (60.7%), balancing stability and adaptability by allowing the teacher to gradually absorb the student’s improving policy while maintaining a smoother, more reliable distribution. A lower decay of 0.90 updates the teacher more slowly, causing it to lag behind the student’s progress, reducing performance to 59.8%.
16
Table 9: Performance comparison on the MMBench-GUI L2 benchmark [37]. Windows
Methods
MacOS
Linux
iOS
Android
Web
Avg.
Bas.
Adv.
Bas.
Adv.
Bas.
Adv.
Bas.
Adv.
Bas.
Adv.
Bas.
Adv.
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
89.30 90.04 91.88 91.88
65.07 67.65 69.12 68.38
85.51 86.67 83.19 83.19
70.81 71.97 69.36 69.65
76.96 76.44 75.39 75.39
58.16 61.73 58.16 58.67
95.86 96.18 96.18 96.50
84.24 86.36 84.85 86.06
96.35 96.35 96.07 96.07
85.63 87.89 87.61 88.17
95.48 95.48 94.52 95.16
77.60 80.52 78.25 80.52
82.99 84.32 83.27 83.71
+ Ours
91.14
72.06
90.72
76.30
78.01
66.33
97.13
86.67
96.91
91.27
95.81
83.44
86.65
Table 10: Performance comparison on the OSWorld-G benchmark [41]. Text Matching
Element Recognition
Layout Understanding
Fine-Grained Manipulation
Avg.
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
47.37 47.37 42.11 42.11
67.16 70.90 74.63 73.13
66.67 70.22 70.67 70.67
62.12 62.88 62.88 63.64
58.69 61.17 62.06 61.88
+ Ours
52.63
76.12
71.56
62.12
62.94
Methods
Table 11: Performance comparison on the OSWorld-G-Refine benchmark [41]. Text Matching
Element Recognition
Layout Understanding
Fine-Grained Manipulation
Avg.
Qwen3-VL-Instruct [1] + GRPO-Binary [19] + GRPO-Distance [46] + GRPO-Gaussian [32]
52.63 47.37 47.37 52.63
78.36 79.85 82.09 82.09
79.56 79.11 80.00 80.00
65.15 70.45 71.97 71.97
67.38 68.62 69.86 70.04
+ Ours
52.63
82.09
83.56
69.70
70.92
Methods
Table 12: Ablation on visual privilege design. SSP denotes ScreenSpot-Pro accuracy. ∆ OPSD denotes the performance difference relative to Naive OPSD. Teacher’s Guidance Setting Visual Context Text Context
Student SSP ∆ OPSD
Orig. Image Standard Zoom + Drawn BBox Adaptive Zoom + Drawn BBox
55.6 60.1 60.3
Inst. + Text BBox Inst. + Drawn Hint Inst. + Drawn Hint
0 +4.5 +4.7
Table 13: Ablation on teacher update strategy. qθ denotes using the current policy as the teacher (updated every step), qθref denotes using a frozen copy of the initial policy as the teacher, and EMA denotes the exponential moving average teacher with the specified decay coefficient. Teacher
ScreenSpot-Pro
qθ qθref EMA = 0.90 EMA = 0.95
59.4 59.6 59.8 60.7
17