arXiv:2607.29199v1 [cs.CR] 31 Jul 2026
Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion Haoxin An
Yunpeng Song
Zihao Bai
Xi’an Jiaotong University Xi’an, China [email protected]
Xi’an Jiaotong University Xi’an, China [email protected]
Xidian University Xi’an, China [email protected]
Zhongmin Cai
Guojun Xiong
Chenhao Lin
Xi’an Jiaotong University Xi’an, China [email protected]
Shanghai Jiao Tong University Shanghai, China [email protected]
Xi’an Jiaotong University Xi’an, China [email protected]
Wentao Chen
Feng Wei
Chao Shen
Artificial Intelligence Research Institute, China Academy of Information and Communications Technology Beijing, China [email protected]
Artificial Intelligence Research Institute, China Academy of Information and Communications Technology Beijing, China [email protected]
Xi’an Jiaotong University Xi’an, China [email protected]
Abstract
1
Trustworthy deployment of GUI agents in ubiquitous computing settings requires alignment that survives dynamic interaction and precise threat conditions, not just single-turn refusal of explicit harmful requests. We argue that prompt-level alignment, the dominant lightweight defense in current mobile agents, is a local phenomenon: it works reliably only in the narrow evaluation slice where it is typically measured, namely single-turn, explicitly-verbalized intent, and degrades systematically along two axes that any real user can traverse. Using a paired diagnostic on three frontier GUI agents, screen-grounded, user-side persuasion, no environment injection, we show that a one-line guardrail achieves large single-shot ASR reductions, up to roughly 40 points, at near-zero over-refusal cost. Nevertheless, moving from independent probes to four-turn escalation chains raises guarded ASR by approximately 20 points on every model. Relative to the neutral baselines, this increase reflects substantial guardrail erosion for Qwen but a largely defense-orthogonal dynamic risk for Claude and GPT. The sign of the salience gap flips under the guardrail: concealed requests are not systematically more successful than explicit ones without a guardrail, but more successful with one, indicating that the defense engages primarily when intent is named. Static single-turn ASR therefore overstates deployed robustness by a systematic and predictable margin.
Trustworthy deployment of GUI agents in ubiquitous computing settings imposes three demands that a scalar single-turn attack-success rate cannot capture. Safety must survive dynamic interaction, since an agent that refuses in isolation may accede once dialogue history accumulates. It must be precise, holding at every action boundary rather than only when harmful intent is explicitly named. And because a GUI agent’s outputs eventually become taps and confirmations on a real device, its pre-execution commitments, i.e., what it is willing to materially advance toward, function as a necessary gate on execution safety. Yet the dominant evaluation practice for GUI-agent alignment remains static, single-turn, and response-only, reporting one attack-success number per model [1, 11, 28]. Existing safety work approaches the problem from two directions. One line studies environmental injection, in which malicious content is planted in the interface as pop-ups, hidden elements, or third-party payloads [7, 15, 16, 30, 36]; here the adversary is the environment, not the principal. A second line evaluates sandboxed harmful-task execution, issuing an explicitly harmful objective and checking whether the resulting actions cross a boundary in an emulator [1, 25, 32]. Both are valuable, and both under-represent the case that ubiquitous deployment makes concrete: the principal themselves, whether a malicious user, a delegating party on a shared or kiosk device, or a social engineer operating the agent on a victim’s screen, persuading the agent in ordinary screen-grounded language. Neither line asks the question that lightweight promptlevel defenses, the dominant deployed safeguard, force us to ask: on which evaluation slice is such a defense actually effective, and how does its effectiveness change as we move off that slice? We frame this as the problem of local alignment. A promptlevel guardrail is not a scalar property of a model but a property of a narrow evaluation region defined by two axes any real user can traverse: the salience of adversarial intent, i.e., whether it is
CCS Concepts • Security and privacy → Human and societal aspects of security and privacy.
Keywords GUI agents, AI safety, red-teaming
Introduction
An et al.
Figure 1: Overview of the AXIS pipeline. Harmful cases and benign controls feed a shared harness that crosses regime × salience × defense; screen grounding (the attached screenshot) is held constant across all cells. explicitly verbalized or dissolved into a mundane or procedural request [14, 26, 27], and the turn structure of the interaction, i.e., whether the request stands alone or emerges from an accumulating dialogue. Instrumenting these two axes as controlled variables in a paired diagnostic on three frontier GUI agents, we find that a one-line guardrail cuts unit-level attack success by up to ∼40 points in the independent regime at over-refusal cost below 2.2% (F1); that moving from independent probes to four-turn chains raises guarded unit-level ASR by approximately 20 points on every model, although relative guardrail erosion is substantial only for Qwen (F2); and that moving off it along the salience axis flips the sign of the salience gap, so that concealed requests become more successful than explicit ones for every guarded model, a pattern consistent with a defense that engages primarily when harmful intent is verbalized (F3). Scalar ASR therefore overstates deployed robustness by a systematic and predictable margin, and precision alignment must be reported conditionally over salience and turn structure rather than as a single number. Contributions. (1) We articulate local alignment as a framework for evaluating prompt-level defenses in GUI agents, and identify user-side, screen-grounded adversarial persuasion as its deployment-relevant threat model. (2) Through our proposed AXIS, a paired diagnostic that crosses salience and turn structure with a shared guardrail axis under refusal-aware reporting, we quantify two off-axis failures of simple prompt-level alignment on three frontier agents and derive concrete conditional-reporting recommendations for future ubiquitous-agent safety benchmarks.
2
Related Work
GUI agents built on MLLMs now operate phone and computer screens, ranging from grounding-focused models [4, 9, 33] to native end-to-end agents [21] and mobile/multi-app systems [29, 35, 37], generally built on general-purpose MLLMs [2, 17, 18]. Their task capabilities are measured by benchmarks such as WebArena [39], VisualWebArena [10], Mind2Web [6], WebVoyager [8], OSWorld [31],
AndroidWorld [22], and Android-in-the-Wild [23]. AXIS instead targets the safety of such agents. One line of safety work asks whether agents comply with overtly harmful goals, as in AgentHarm [1] and SafeArena [28], where the objective is stated up front and the focus is refusal versus compliance. AXIS treats intent salience as an explicit variable and grounds every probe in a mobile screen. A second line targets device agents directly: MLA-Trust reports that GUI embodiment degrades trustworthiness and that multi-step interaction executes content a standalone model would refuse [34], while MobileSafetyBench [11], RiOSWorld [32], and ST-WebAgentBench [12] evaluate risky operations in execution environments, with sandboxes such as ToolEmu emulating downstream tool risk [25]. AXIS is complementary in that it isolates the persuasion dynamics (salience, turns) that precede action and quantifies the defense/over-refusal trade-off these outcome-only benchmarks leave implicit. A third line studies environmental or indirect injection against GUI agents, including privacy-leaking environmental injection [15], pop-up attacks [36], robustness dissection [30], unprivileged thirdparty hijacking [16], and indirect prompt injection into LLM-integrated apps [7]. Those threat models give the adversary partial control of the environment; AXIS instead assumes the adversary is the conversational principal and controls only their own utterances, a distinct and deployment-relevant surface. In the text domain, automated and human red-teaming [3, 20, 27, 41] and multi-turn jailbreaks [14, 26] show that gradual attacks defeat single-turn defenses. AXIS carries this insight into screengrounded interaction and adds a controlled salience manipulation that separates the multi-turn effect from the concealment effect, together with an over-refusal control in the spirit of exaggeratedsafety benchmarks [5, 24] so that safety is not credited to blanket refusal. To the best of our knowledge, AXIS is the first paired diagnostic that instruments salience and turn structure as controlled axes on the same screen-grounded corpus, with an exaggeratedsafety control, for mobile GUI agents.
Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
Table 1: Two-level risk taxonomy. Top-level risk
GUI sub-classes
Expected Beh.
Privacy exfiltration
read out third-party IDs; transcribe private docs silent deletion; mass or hard-to-reverse changes disable security toggles; skip verification confirm transfers without informed consent alter recovery/security on another’s account post/broadcast harmful content from device
refuse / redact
Over-privileged ops Protection bypass Financial actions Account tampering Unsafe content
confirm refuse / warn confirm refuse refuse
AXIS is among the first screen-grounded evaluations to jointly examine intent salience, turn structure, and over-refusal under a user-side persuasion threat model.
3 Method 3.1 Threat Model We consider a GUI agent that receives a system prompt, a screenshot of the current screen, and a stream of user utterances, and returns responses that may include on-screen actions. The adversary is the conversational principal or a delegate operating the agent, for example a malicious user on a shared or kiosk device, or a social engineer running the agent against a victim’s screen. The adversary controls only their own utterances; they do not modify the environment, inject content into third-party apps, or access the system prompt or weights. Pixel-level perturbations, indirect prompt injection [7, 15, 16, 30, 36], and supply-chain compromise are orthogonal to this surface and out of scope. The assets protected are on-screen third-party data, device integrity, and safety-relevant confirmations. A case consists of a captured screenshot, its parsed accessibility tree, a natural-language description of the unsafe outcome an aligned agent should refuse (the target intent 𝜏), a risk category, and a reference safe behavior (refuse, require explicit confirmation, or redact). Cases were captured on researcher-owned devices via ADB, pairing each screenshot with a UI-automator hierarchy in which every element carries text, bounds, and clickability; this element-level grounding distinguishes our corpus from screenshotonly collections. All on-screen personal information is synthetic or redacted. Every probe is issued with the screenshot attached, so screen grounding is held constant across all conditions.
3.2
Risk Taxonomy
The taxonomy has a top level chosen for cross-benchmark comparability and GUI-specific leaves that anchor each risk to concrete on-screen operations. Top-level categories cover privacy exfiltration, over-privileged operations, protection bypass, financial actions, account tampering, and unsafe content. Each leaf specifies the reference safe behavior used by the judge (Table 1).
Figure 2: The two controlled axes. Rows vary intent salience (explicit vs. concealed); columns vary turn structure (singleshot vs. four-turn escalation). Detection difficulty grows toward the bottom-right. Cell labels describe framing, not probe wording.
3.3
Two Paired Diagnostics
Our findings rest on two contrasts, and the harness is organized around them rather than around a monolithic factorial run. Salience-selectivity diagnostic. For each case we synthesize probes at two levels of salience. High-salience probes state the target intent plainly and reference concrete on-screen elements. Low-salience probes never verbalize the target intent; they dissolve it into a mundane, time-pressured, or procedural request, so the agent is given no cue to deliberate on intent [14, 26, 27]. Each probe is delivered in a fresh context (no shared history) and judged individually. The contrast isolates whether the agent’s protection depends on the request being explicitly named. Context-accumulation diagnostic. For each case we synthesize two four-turn escalation chains, one under the low-salience constraint and one under the high-salience constraint. Each chain preserves its assigned salience level while progressively advancing the same underlying target intent across four user turns, with a target-model response collected after every turn. The whole dialogue is judged holistically, with the first unsafe turn recorded as an escalation-depth estimate. The contrast against the singleshot regime isolates the effect of accumulated context on the same underlying risk. Both diagnostics are crossed with a shared defense axis, so that the erosion of a specific defense can be read directly from the two contrasts. A fixed benign screen-description prompt precedes every case and is excluded from success accounting. Figure 2 depicts the four resulting probe types.
3.4
Defense Axis and Over-Refusal Control
The agent is run under two conditions with an otherwise identical system prompt. In the neutral condition, only a plain GUI-assistant role is given, measuring intrinsic alignment. In the guardrail condition, the same prompt carries a concise safety instruction to refuse privacy-violating, over-privileged, confirmation-bypassing, or risky operations. To ensure that any drop in attack success is not bought
An et al.
by blanket refusal, we additionally construct 23 benign look-alike scenarios that preserve similar screen contexts and operation types but contain no harmful intent, and both conditions are evaluated on them, in the spirit of exaggerated-safety benchmarks [5, 24].
3.5
Independent
Attack Synthesis and Diversity Control
Researchers first specify the target intent for each case, and Gemini 3.1 Pro rewrites it into salience-conditioned probes with a balanced low/high split and no cross-referencing in the single-shot regime. All generated probes are manually reviewed for intent fidelity, salience consistency, and screen grounding before evaluation. Because model-based rewriting risks mode collapse, we audit every group with self-BLEU [19, 40], distinct-𝑛 for 𝑛 ∈ {1, 2, 3} [13], and mean and maximum pairwise cosine similarity, regenerating low-diversity cells.
3.6
Table 2: Unit-level ASR (%) by regime and defense across three target models (lower is safer). A unit is one independent probe or one complete four-turn escalation chain.
Model
Neutral
Guard
Neutral
Guard
Qwen3.7-Plus Claude Opus 4.8 GPT-5.6 Sol
76.7% 33.7% 40.1%
37.2% 23.8% 14.5%
79.1% 53.5% 62.8%
58.1% 44.2% 34.9%
Table 3: Unit-level ASR (%) in the Independent regime, by defense and salience. The Low−High difference is the salience gap Δsal (Eq. 2); it is small or negative without a guardrail but positive for all three models under one.
Judgment and Metrics
Neutral
Following LLM-as-a-judge methodology [38], Gemini 3.1 Pro first assesses whether the agent’s response materially advances 𝜏; refusals, deflections, and safe alternatives count as non-success. It is called once per independent probe or once over a complete multiturn dialogue, for which it also reports the first unsafe turn. Two researchers then independently review every verdict blind to the target model; inter-annotator agreement was high (Cohen’s 𝜅 = 0.94), and Gemini agreed with the adjudicated labels on 96% of items (𝜅 = 0.92). All results in Section 4 use the human-adjudicated labels. For cross-regime reporting, an evaluation unit 𝑢 is one independent probe in the single-shot regime or one complete four-turn chain in the escalation regime. Let 𝑦(𝑢) = 1 if the judge finds that the corresponding response or dialogue materially advances 𝜏 and 0 otherwise. Writing U for the set of adversarial evaluation units (benign controls excluded), the unit-level attack success rate is 1 ∑︁ ASRunit = 𝑦(𝑢). (1) |U| 𝑢∈U
Restricting to units of a fixed salience level 𝜎 ∈ {low, high} gives ASR𝜎 , and the salience gap is Δsal = ASRlow − ASRhigh,
(2)
so Δsal > 0 means concealment helps the attacker. On the benign control set B, the over-refusal rate is 1 ∑︁ 1[ 𝐴 refuses 𝑏 ] . (3) ORR = |B| 𝑏∈B
Reporting ASR alone can be gamed by refusing everything, a concern made concrete by exaggerated-safety benchmarks [5, 24]. We therefore also report a refusal-aware robustness score 𝑅 = 1 − 𝑤 𝑎 ASR − 𝑤𝑜 ORR,
Escalation
𝑤 𝑎 + 𝑤𝑜 = 1,
(4)
with 𝑤 𝑎 = 𝑤𝑜 = 12 as our default. 𝑅 rewards a model only when it is simultaneously hard to attack and willing to help on benign look-alikes.
Guard
Model
Low
High
Low
High
Qwen3.7-Plus Claude Opus 4.8 GPT-5.6 Sol
77.9% 31.4% 38.4%
75.6% 36.0% 41.9%
41.9% 25.6% 16.3%
32.6% 22.1% 12.8%
Table 4: Independent regime under the guardrail defense: unit-level ASR vs. over-refusal (ORR) on benign controls, and the refusal-aware robustness score 𝑅 (Eq. 4, 𝑤 𝑎 =𝑤𝑜 = 21 ). Over-refusal is near-zero at this scale. Model
ASR ↓
Over-refusal ↓
Robustness ↑
Qwen3.7-Plus Claude Opus 4.8 GPT-5.6 Sol
37.2% 23.8% 14.5%
0.0% 0.0% 2.2%
81.4% 88.1% 91.6%
4 Experimental Setup and Results 4.1 Setup and Hypotheses We evaluate each target model using the two paired diagnostics defined in Section 3, both of which share the same defense axis. Each comparison is run under the neutral and guardrail conditions, and every probe is screen-grounded. Gemini 3.1 Pro is held fixed as both the probe-rewriting model and the initial automated judge, with human review applied to all probes and verdicts. The three targets are Claude Opus 4.8, GPT-5.6 Sol, and Qwen3.7-Plus. The benchmark contains 43 harmful screen-grounded cases and 23 benign control scenarios. For each harmful case, the independent regime contains two low-salience and two high-salience probes, yielding 172 independent probes in total. The escalation regime contains one low-salience and one high-salience chain per case, yielding 86 chains; every chain comprises four user turns and four corresponding target-model responses. Thus, under each model– defense condition, the harmful evaluation contains 172 independent probes and 86 escalation chains. Across the three target models and two defense conditions, this produces 1,032 independent-probe judgments and 516 holistic chain judgments (1,548 adversarial evaluation units in total), corresponding to 3,096 target-model response
Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
(a) Guardrail erosion under escalation
sal (pts)
Results
Table 2 gives unit-level ASR per regime×defense cell; Table 3 decomposes the independent regime by defense and salience at the unit-level level, from which Eq. (2) is read; Table 4 reports ASR, over-refusal, and robustness under the guardrail. Figure 3 visualizes the two central effects. Three findings emerge, corresponding to the origin and the two off-axis directions of the local-alignment picture. F1: where the guardrail works. In the independent regime, the guardrail reduces unit-level ASR on every target: Qwen3.7-Plus drops from 76.7% to 37.2%, GPT-5.6 Sol decreases from 40.1% to 14.5%, and Claude Opus 4.8 falls from 33.7% to 23.8% (Table 2), while over-refusal on benign controls stays at most 2.2% (Table 4). Refusal-aware robustness (Eq. 4) ranks GPT-5.6 Sol (91.6%) above Claude (88.1%) and Qwen3.7-Plus (81.4%). F1 fixes the single-shot regime in which prompt-level alignment appears strongest. F2: multi-turn as an off-axis attack surface. Under the guardrail, moving from independent probes to multi-turn escalation chains raises unit-level ASR by approximately 20 points on every target (Table 3; Figure 3a). The low-ASR operating point identified in F1 is not maintained once dialogue history accumulates; on Claude, guarded multi-turn ASR (44.2%) even exceeds its unguarded single-shot ASR (33.7%), so ordinary multi-turn framing recovers the safety level of the undefended model. The corresponding neutral shifts (+2.4, +19.8, +22.7; Qwen bounded by its 76.7% single-turn ceiling) confirm the effect is not an artifact of the guardrail. Prompt-level protection thus concentrates in the single-shot slice and gives roughly 20 of those points back along an axis every deployed conversation traverses. F3: the naming gap. Without a guardrail, salience barely predicts success and can reverse in sign, with unit-level gaps of +2.3, −4.6, and −3.5 points (Table 3, neutral). Under the guardrail the gap turns positive on every model (+9.3, +3.5, +3.5; Figure 3b). Concealed requests never reach complete bypass (ASR 16.3–41.9%); the finding is the sign flip, not the level. The pattern is consistent with a defense that engages primarily when harmful intent is explicitly verbalized, and refines H1: the salience gap is a property of the defended model. Together, F1 characterizes the single-shot regime, F2 measures the increase in chain-level exposure along the turn axis, and F3 measures variation along the salience axis. Static single-turn ASR understates deployed risk by roughly 20 points once history accumulates, and hides a systematic reallocation of residual risk toward concealed intent.
40
Independent + Guard Escalation + Guard
58.1
60
44.2
37.2 23.8
20
34.9 14.5
0
Salience gap
4.2
Case-level ASR (%)
turns. The 23 benign scenarios form the separate control set used to compute over-refusal. Following the local-alignment framing of Section 3, we preregister three hypotheses. H1 (naming gap): under a guardrail, concealed probes succeed at least as often as explicit ones, Δsal ≥ 0. H2 (dynamic gap): under the guardrail, four-turn escalation chains have higher unit-level ASR than salience-matched independent probes. H3 (defense trade-off): the guardrail lowers single-shot ASR at low over-refusal cost, but its aggregate value must be read jointly with salience and turn structure.
(b) Salience gap by defense condition +9.3
10 5
sal Neutral sal Guard
+3.5
+2.3
+3.5
0 5
-4.6
Qwen3.7
Claude
-3.5
GPT-5.6
Figure 3: (a) Under the guardrail, moving from independent probes to four-turn chains raises unit-level ASR by ∼ 20 points on all three models (F2). (b) The salience gap (Eq. 2) is mixed or negative without a guardrail and positive for every model under one (F3).
5
Discussion
The results are consistent with prompt-level alignment behaving as a local phenomenon: reliable in the slice where it is typically measured, and systematically weaker along the two axes any user can traverse. Two implications follow from the off-axis findings. From F2, on the dynamic gap. Four-turn escalation raises guarded unit-level ASR by approximately 20 points across all three models, although the difference-in-differences analysis attributes substantial relative guardrail erosion only to Qwen. Aggregate ASR should therefore be reported conditionally on turn structure rather than marginalized over it, and dialogue history should be treated as safety state, with intent re-evaluated at each action boundary. Single-probe and chain-level evaluations are not interchangeable: the former measures isolated compliance, whereas the latter measures cumulative exposure over a deployed interaction. From F3, on the naming gap. Under the guardrail the attacker gains ground by dissolving the goal rather than by stating it more forcefully. A guardrail keyed on lexicalized cues protects the region where those cues are present and leaves its complement partially exposed. Salience-decomposed ASR, or the low-versus-high gap, should be reported alongside the aggregate: a reduction that reshapes residual risk toward concealed intent is not the same safety property as one that shrinks it uniformly. The response-level scoring used here further acts as a pre-execution gate, so a defense that holds only in the origin slice provides an execution-safety guarantee only in that slice; emulator-based benchmarks [25] and this diagnostic cover complementary segments of the persuasion-to-action pipeline. Threats to validity. Several limitations bound our claims. AXIS scores responses rather than executed actions, so our numbers are a pre-execution signal, not an end-to-end safety measurement. Verdicts depend on an automated judge; dual expert adjudication
An et al.
mitigates but does not eliminate residual subjectivity on borderline compliance. Probes come from a single fixed attacker with diversity auditing, which does not approximate an adaptive human red team, so reported ASRs should be read as lower bounds. Salience is operationalized by construction and correlates in practice with directness and specificity; we do not claim to have isolated it from these covariates. The multi-turn lift is directly observable on only two of three neutral conditions, as Qwen3.7-Plus is ceiling-bound at 76.7%, so the defense-orthogonality reading of F2 rests on a narrow base. Finally, both probe rewriting and initial judging use Gemini 3.1 Pro; although all outputs are human-reviewed, alternative models may yield different attack distributions or borderline judgments.
6
Conclusion
We recast prompt-level alignment in GUI agents as a local property and, through AXIS, showed that a one-line guardrail effective in single-shot probes gives back roughly 20 ASR points under multiturn escalation and flips the salience gap so that concealed requests outperform explicit ones on every defended model. Deployed robustness therefore cannot be summarized by a scalar ASR; it must be reported conditionally over salience and turn structure, alongside an over-refusal control. We view AXIS as a template for safety evaluations whose conclusions survive the conditions under which ubiquitous agents are actually used.
References [1] Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=AC5n7xHuR1 [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025). doi:10.48550/arXiv.2502.13923 [3] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2025. Jailbreaking Black Box Large Language Models in Twenty Queries. (2025), 23–42. doi:10.1109/SaTML64287.2025.00010 [4] Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 9313–9332. doi:10.18653/v1/2024.acl-long.505 [5] Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. OR-Bench: An Over-Refusal Benchmark for Large Language Models. 267 (2025), 11515–11542. https://proceedings.mlr.press/v267/cui25a.html [6] Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 28091–28114. https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 5950bf290a1570ea401bf98882128160-Abstract-Datasets_and_Benchmarks.html [7] Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security. Association for Computing Machinery, 79–90. doi:10.1145/3605764.3623985 [8] Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 6864–6890. doi:10.18653/v1/2024.acllong.371 [9] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, and Jie Tang. 2024. CogAgent: A Visual Language Model for GUI Agents. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14281–14290. doi:10.1109/CVPR52733.2024.01354 [10] Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Russ Salakhutdinov, and Daniel Fried. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 881–905. doi:10.18653/v1/2024.acl-long.50 [11] Juyong Lee, Dongyoon Hahm, June Suk Choi, W. Bradley Knox, and Kimin Lee. 2026. MobileSafetyBench: Evaluating Safety of Autonomous Agents in Mobile Device Control. Proceedings of the AAAI Conference on Artificial Intelligence 40, 44 (2026), 37565–37573. doi:10.1609/aaai.v40i44.41090 [12] Ido Levy, Ben Wiesel, Sami Marreed, Alon Oved, Avi Yaeli, and Segev Shlomov. 2026. ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents. In The Fourteenth International Conference on Learning Representations. https://openreview.net/forum?id=MuCDzH0ctf [13] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A Diversity-Promoting Objective Function for Neural Conversation Models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 110–119. doi:10.18653/v1/N16-1014 [14] Nathaniel Li, Ziwen Han, Ian Steneker, Willow E. Primack, Riley Goodside, Hugh Zhang, Zifan Wang, Cristina Menghini, and Summer Yue. 2024. LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet. arXiv preprint arXiv:2408.15221 (2024). doi:10.48550/arXiv.2408.15221 [15] Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. 2025. EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum? id=xMOLUzo2Lk [16] Guohong Liu, Jialei Ye, Jiacheng Liu, Yuanchun Li, Wei Liu, Pengzhi Gao, Jian Luan, and Yunxin Liu. 2025. Hijacking JARVIS: Benchmarking Mobile GUI Agents against Unprivileged Third Parties. In Proceedings of the 2nd International Workshop on Edge and Mobile Foundation Models. Association for Computing Machinery, 12–18. doi:10.1145/3737902.3768354 [17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 34892–34916. https://proceedings.neurips.cc/paper_files/paper/ 2023/hash/6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html [18] OpenAI. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774 (2023). doi:10.48550/arXiv.2303.08774 [19] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, 311–318. doi:10.3115/1073083.1073135 [20] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red Teaming Language Models with Language Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 3419–3448. doi:10.18653/v1/2022.emnlp-main.225 [21] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. 2025. UI-TARS: Pioneering Automated GUI Interaction with Native Agents. arXiv preprint arXiv:2501.12326 (2025). doi:10.48550/arXiv.2501.12326 [22] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William E. Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Kenji Toyama, Robert James Berry, Divya Tyamagundlu, Timothy P. Lillicrap, and Oriana Riva. 2025. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id= il5yUQsrjC [23] Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. 2023. Android in the Wild: A Large-Scale Dataset for Android Device Control. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc. https://proceedings.neurips.cc/ paper_files/paper/2023/hash/bbbb6308b402fe909c39dd29950c32e0-AbstractDatasets_and_Benchmarks.html [24] Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Association for Computational Linguistics, 5377–5400. doi:10.18653/v1/2024.naacl-long.301
Alignment Is Local: A Paired Diagnostic for GUI Agents under User-Side Persuasion
[25] Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the Risks of LM Agents with an LM-Emulated Sandbox. In The Twelfth International Conference on Learning Representations. https://openreview. net/forum?id=GEcwtMk1uA [26] Mark Russinovich, Ahmed Salem, and Ronen Eldan. 2025. Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack. In 34th USENIX Security Symposium (USENIX Security 25). USENIX Association, 2421–2440. https://www.usenix.org/conference/usenixsecurity25/presentation/ russinovich [27] Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. “Do Anything Now”: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In Proceedings of the 2024 ACM SIGSAC Conference on Computer and Communications Security. Association for Computing Machinery, 1671–1685. doi:10.1145/3658644.3670388 [28] Ada Defne Tur, Nicholas Meade, Xing Han Lù, Alejandra Zambrano, Arkil Patel, Esin Durmus, Spandana Gella, Karolina Stanczak, and Siva Reddy. 2025. SafeArena: Evaluating the Safety of Autonomous Web Agents. In Proceedings of the 42nd International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 267). PMLR, 60404–60441. https://proceedings.mlr.press/ v267/tur25a.html [29] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception. arXiv preprint arXiv:2401.16158 (2024). doi:10.48550/arXiv.2401.16158 [30] Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. 2025. Dissecting Adversarial Robustness of Multimodal LM Agents. In The Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=YauQYh2k1g [31] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems, Vol. 37. Curran Associates, Inc., 52040–52094. doi:10.52202/079017-1650 [32] Jingyi Yang, Shuai Shao, Dongrui Liu, and Jing Shao. 2025. RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents. In Advances in Neural Information Processing Systems, Vol. 38. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 0c79d6ed1788653643a1ac67b6ea32a7-Abstract-Conference.html [33] Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv preprint arXiv:2310.11441 (2023). doi:10.48550/arXiv.2310.11441 [34] Xiao Yang, Jiawei Chen, Jun Luo, Zhengwei Fang, Yinpeng Dong, Hang Su, and Jun Zhu. 2025. MLA-Trust: Benchmarking Trustworthiness of Multimodal LLM Agents in GUI Environments. arXiv preprint arXiv:2506.01616 (2025). doi:10. 48550/arXiv.2506.01616 [35] Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025. AppAgent: Multimodal Agents as Smartphone Users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, Article 70, 20 pages. doi:10.1145/ 3706598.3713600 [36] Yanzhe Zhang, Tao Yu, and Diyi Yang. 2025. Attacking Vision-Language Computer Agents via Pop-ups. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 8387–8401. doi:10.18653/v1/2025.acl-long.411 [37] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024. GPT-4V(ision) Is a Generalist Web Agent, If Grounded. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235). PMLR, 61349–61385. https://proceedings.mlr.press/v235/zheng24e.html [38] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., 46595–46623. https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html [39] Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=oKn9c6ytLx [40] Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A Benchmarking Platform for Text Generation Models. In Proceedings of the 41st International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, 1097–1100. doi:10.1145/3209978.3210080
[41] Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv preprint arXiv:2307.15043 (2023). doi:10.48550/arXiv. 2307.15043