GUI agent: Guided Exploration of User-Sensitive Screens ∗
arXiv:2606.25705v1 [cs.AI] 24 Jun 2026
Aradhana Nayak 1 , Mussadiq Nazeer1 , Wang Peng 2 and Feng Liu1 1 Huawei Heisenberg Research Center (Munich) 2 Huawei Technologies Co., Ltd {aradhana.nayak, mussadiq.mussadiq, wangpeng394, feng.liu1}@huawei.com
Abstract LLM agents are increasingly being used to automate tasks for users within an open GUI environment. They inevitably encounter screens containing user-sensitive information, for which takeover of task execution by the user is highly desirable or even necessary. State-of-the-art LLM-driven agents are usually fine-tuned to complete tasks regardless of the safety implications of their actions. This makes their real-world deployment difficult and adversely affects the reliability. Therefore, it is crucial to identify and categorize usersensitive states and define user-sensitive queries. This dataset would be to engineers to recognize and request handover to the user in critical scenarios. This short paper develops an explorer agent that systematically explores the query space starting from one demonstrated task to identify queries that, if executed, would lead to user-sensitive states in a GUI environment.
1
Introduction
The rapid development of the thinking and reasoning capabilities of LLMs, combined with their ability to match the performance of older models with an increasingly smaller number of parameters, has led to their widespread application to edge devices ([Ye et al., 2025], [Qin et al., 2025], [Lee et al., 2024]). LLMs can perform a wide range of tasks on web and GUI interfaces, from simple tasks such as adding calendar events to long-term tasks such as shopping for items and cross-application tasks. This has necessitated the endowment of LLM-based GUI agents with a high degree of autonomy, which may sometimes infringe users’ privacy and conflict with their preferences ([Zhang et al., 2024], [He et al., 2025]). Furthermore, the actions executed by a GUI agent are often irreversible, such as sending emails, deleting files and completing transactions ([Hua et al., 2024]). The workflow of GUI agents can easily become uncontrollable, as they operate in closed loops where each action modifies the environment, and a single incorrect click can lead to unexpected ∗
Presented at the IJCAI-ECAI 2026 RobustifAI workshop
screens and cascading errors ([Wu et al., 2024]). To safely integrate LLMs as decision makers in GUI environments, it is therefore necessary to identify and isolate screens that could potentially require user feedback.
2
Related Work
Exploration in GUI Environment Recent works address GUI agent generalization by leveraging exploration-based priors. GUI-Xplore [Sun et al., 2025] uses pre-recorded videos to build transition graphs for cross-app generalization, while GUI-Explorer [Xie et al., 2025] constructs unsupervised knowledge graphs for dynamic guidance. Guardian [Ran et al., 2024] offloads computation to external engines to refine action spaces. Reinforcement Learning (RL) methods for LLM agents RL enables active, autonomous exploration, reducing reliance on static datasets. This allows agents to dynamically adapt to novel GUI environments. Their application however, is challenging as sparse rewards in dynamic interfaces often fail to provide sufficient guidance for decision making. [Zhai et al., 2025] and [Putta et al., 2024] integrate Monte Carlo Tree Search (MCTS) with RL to decompose sparse rewards into dense Q-values, addressing credit assignment; [Putta et al., 2024] further uses self-critique as intermediate feedback. [Lin et al., 2025] propagates future utility backward to prune inefficient behaviors, while Search-Agent [Koh et al., 2024] employs A* search with alternative paths for explicit back-tracking. [Fan et al., 2025] treats GUI elements as tokens for robust visual grounding, and [Shi et al., 2025] uses dynamic memory-guided MCTS to adjust action evaluation. [Zhou et al., 2023] adapts MCTS via Language Agent Tree Search, expanding ReAct into a reasoning search. MCTS-based Exploration for GUI Environments While policy improvement through RL-based methods has received significant attention in literature, their use in dataset curation and out-of-domain generalization remains less explored. By optimizing exploration policies iteratively, RL-based methods could potentially prioritize unvisited states, significantly improving screen diversity coverage and to selectively design training datasets for application specific purposes. While reasoning capabilities of current Reasoning LLMs (RLLMs) are generally desirable for systematic
Algorithm 1 Learning Framework for Explorer LM
Figure 1: The explorer LM is trained with an MCTS-like approach. The native LM (M3A [Rawles et al., 2024]) proceeds to modify user-sensitive data during roll-out.
problem solving, [Lu et al., 2025] challenges this assumption. The authors argue that RLLMs often engage in “structureless wandering” rather than rigorous exploration. The lack of rigorous exploration of the solution space leads to exponential performance deterioration as problem complexity increases, suggesting that mere computational scaling is insufficient. The solution-space exploration policies in literature are either through instruction-tuning ([Murty et al., 2024]) or purely algorithmic ([Zhao et al., 2024]). [Chi et al., 2025] propose ScreenExplorer to maximize exploration efficiency through a reward shaping mechanism. It prioritizes all unvisited screen states to discover novel interaction paths with the aim of eventually finding a solution trajectory to every possible user query. However, the performance of the model after several rounds of training is comparable to an untrained SOTA model and, a coverage argument for exploration of solution-space is not provided. In this short paper, we propose an algorithm to isolate user-sensitive query (or tasks) and screens starting from a single user-defined trajectory in a GUI application. The method is based on MCTS and relies on experience distillation from previous iterations (training rounds) to generate novel user-query. We obtain evidences of coverage of query space 1) within a single training round through instruction finetuning and a proposed saturation algorithm 2) within consecutive rounds through rewards employed in the RL training.
3
Methodology
Our framework consists of two models: the native language (or vision language) model (LM), which determines actions, and the explorer LM, which is the main contribution of this work. The explorer agent is iteratively trained with the objective of pure exploration, i.e. the policy determines which node to explore with rewards based on novelty and user sensitivity of the node. Our exploration objective is dual as the search space comprises both the base query and the sensitive nodes branching out from it.
3.1
Query Selection, Expansion and Saturation
In order to select a base query in the query search space, we train the policy Πθ (s0 ) where s0 is the home page screen in the GUI environment. The policy function is the log-likelihood of the output sequence of the explorer LM
Require: Exploration model Πθ , Algorithms Asel (Query Selection), Asat (Saturation Check), Minimum number of novel queries per iteration Nmin 1: Generate and Select Q ∼ Πθ with Asat , Asel 2: while len(Q) > Nmin do: 3: Perform roll-out for all queries in Q 4: Back propagate gradients from roll-out queries 5: Update Πθ 6: Generate and Select Q ∼ Πθ with Asat , Asel 7: end while under it’s distribution. The explorer LM generates N × M episodes (queries and their steps) in N batches of size M each up to saturation (Algorithm 3). After generation of queries, selection method (in Algorithm 2) is used which relies on a single execution trajectory T0 = (u0 , s0j , a0j ); j = 1 . . . T of T screens as input along with an instruction I (details in Appendix). Essentially we simplify the following UCT formula as N (ui ) = 0 i.e. repeated queries are dropped by Algorithm 2. s ! log N (ui ) ∗ u = arg max V (ui , s0 ) + 1 + log N (ui ) i∈{1,...,M } = arg max V (ui , s0 ) i∈{1,...,M }
where V (ui , s0 ) := P ΠM k=1 Πp=1 {Sim(ui , uk ) − τ ) ∗ Sim(ui , up ) − τ } and Sim(ui , uj ) is defined in Algorithm 2 (Asel ). Instead of a single query u∗ in each iteration of MCTS loop, we select a batch of queries {ui }i∈index retained in order to prevent overfitting during training of the explorer LM. This is also the reason we choose GRPO for training the LLM and experience distillation. The selected batch of queries is: u∗ ∈ {ui : V (ui , s0 ) > 0} Asel enforces that indices retained = {i : V (ui , s0 ) > 0} wherein all generated queries in a batch are compared with themselves as well as the pool (initialized with T0 and pool P = 1). Queries retained in each batch are added to P . The number of batches N is decided according to a Algorithm 3 which identifies the inflection point after which few new queries are generated by the model. Each query ui is generated along with steps. In particular, actions taken called “actionPurpose” aij , screen descriptions called “uiSummary” and “category” of screens for steps j ∈ {0, . . . , T − 1} are generated. The number of unique queries is less than or equal to N × M and the number of steps per query T is dependent on the query itself.
3.2
Roll-out with Emulator
Our set-up for roll-out is that of SPABench ([Chen et al., 2024]). The agent is connected to an Android emulator providing the screen observation to the agent. The actions (generated by the agent) are executed within the emulator which outputs a screenshot after each action. The execution
Algorithm 2 Query Selection Algorithm (Asel ) Require: Model Πθ , Instruction I, Pool Size P , Threshold τ , execution trajectory T0 = (u0 , s0j , a0j ); j = 1 . . . T Ensure: Vectorized Pool {up,vec }P p=1 1: for i = 1 to M do 2: Generate query (and steps): ui ∼ Πθ (·|I, T0 ) 3: end for 4: Preprocessing: 5: Compute vectorized representations for all queries: ui,vec ← Embed(ui ) for each ui ∈ {u1 , . . . , uM } 6: index retained ← ∅ query retained ← ∅ 7: for i = 1 to M do 8: for k = 1 to i do 9: Compute similarity of query in batch to query in batch: Sim(ui , uk ) ← CosineSimilarity(ui,vec , uk,vec ) 10: for p = 1 to P do 11: Compute similarity of query in batch to query in pool: Sim(ui , up ) ← CosineSimilarity(ui,vec , up,vec ) 12: if Sim(ui , up ) < τ & Sim(ui , uk ) < τ then 13: Add i to index retained 14: Add ui to query retained 15: end if 16: end for 17: end for 18: end for 19: return index retained, query retained trajectory containing screenshots (from emulator) and action logs (from agent) are saved in the worker machine which manages communication between emulator and agent and, oversees the task completion process. The native LM can be freely chosen in SPABench such as SeeAct ([Zheng et al., 2024]), M3A ([Rawles et al., 2024]), MobileAgent ([Wang et al., 2024b], [Wang et al., 2024a]) etc.. We choose M3A agent for the experiments in Section 4.
3.3
Figure 3: Saturation in Round 1 after 160 queries, in Round 2 after 100 queries and in Round 3 after 70 queries are generated. Red star denotes satisfaction of slope condition (line 13) in Algorithm 3
objective is achieved through Group Relative Policy Optimization (GRPO) and reward modeling. We first construct a roll-out buffer of episodes called as memory bank in order to prepare the training data. The query generated by the explorer LM are rolled-out, stored in the memory bank and updated iteratively (Figure 1). It consists of queries ui and steps aij ; i ∈ {1, . . . I} and j ∈ {0, . . . T − 1}. Besides this, memory bank contains text embeddings of ui and aij generated by Qwen3-Embedding-0.6B Model. We design following novelty score to compose the reward: Query Novelty Score: • Cluster queries in memory bank and generated query with kmeans for number of clusters ranging up to total number of queries to be clustered. Find kmeans inertia for each of them • Find optimal no. of cluster (= argmin of first derivative of inertia) and same cluster indices query • query novelty score = exp{−len(queries[j])}; j ∈ same cluster indices query Step Novelty Score: • Compute the cosine similarity matrix Sim[i, j] for each step ‘actionPurpose” with other steps in memory bank • step novelty score = Σi [1 − maxj (Sim[i, j])]/len(steps); i <= len(steps), j <= len(steps memory bank)
Training
The backpropagation step consists of training the model to generate 1) accurate steps and, 2) Novel query leading to novel user-sensitive states. The first objective is achieved through supervised finetuning (SFT) of the model with prompt-response pairs from roll-out episodes. The second
Category Novelty Score: • Count number of times each category has been generated in memory bank • Define ci as count of category in step i of generated query. ci = 0 if category is not critic and ci = 1 if it is critic • category novelty score = Σi 1/len(steps) ∗ (1 + ci ) The total reward for queries ui and steps aij is computed as: R(ui , aij ) := (1) query novelty score(ui ) ∗ [step novelty score(aij ) + category novelty score(aij )]
4 Figure 2: Backpropagation step in MCTS is replaced by GRPO training over a batch of selected queries for Explorer LM
Experiments and Results
We set Nmin = 70 in Algorithm 1, τ = 0.85, batch size M = 10 in Algorithm 2, Nmax = 200 in Algorithm 3. Llama3.1-8B and Qwen2.5-3B-Instruct models were unable
Novelty Score
Queries Selected
Query
Step (Predicted)
Step (Rolled-out)
10 11
0.01892 0.00285
0.1485 0.1154
0.1056 0.0821
Table 3: Novelty scores across different sets in round 3
Figure 4: Total rewards over training steps across 3 training rounds
Novelty Score
Queries Selected
Query
Step (Predicted)
Step (Rolled-out)
13 9 9
0.0299 0.0211 0.0217
0.2195 0.2190 0.1790
0.1785 0.1635 0.1357
Table 1: Novelty scores across different aggregated sets in round 1
to proceed beyond initial screen and hence produced very low novelty scores in the first round. The mean query novelty score for Qwen2.5-32B-Instruct in round 1 is approximately 0.0519 compared to approximately 0.0042 for Llama 3.1 8B. We therefore choose the explorer LMs as Qwen-2.5-32B-Instruct. We performed SFT using Llama Factory with LoRA (rank=64, all linear layers) and a cosine learning rate scheduler (key hyperparameters: learning rate: 1e-4, batch size: 1, max steps: 60, infrastructure details: DeepSpeed ZeRO-3, fp16, 8 NPUs). We fine-tuned the SFT model using Group Relative Policy Optimization (GRPO) on a cluster of 4 nodes with 8 NPUs (8GB each) each, leveraging the VerL framework with reward function described in Section 3.3 and the total reward in (1). Training employed a fixed learning rate of 5 × 10−7 , a batch size of 64, and 2 epochs, with model merging performed post-training to consolidate FSDP shards. We conducted 3 training rounds and observed that: (i) The total reward reduces progressively in orders of magnitude from 10−2 to 10−5 in subsequent training rounds (or MCTS iterations) implying a shrinking of query and screen space in each iteration (Figure 4). This is also evidenced by progressive drop in number of queries up to saturation (n in Algorithm 3) over subsequent training rounds as seen in Figure 3. (ii) The standard deviation of step novelty score in Round 1 is approximately 0.0686 compared to 0.00240 and 0.0068 in Round 2 and Round 3 respectively, showing that step Novelty Score
Queries 10 20 30
Query
Step (Predicted)
Step (Rolled-out)
0.0123 0.0096 0.0093
0.1197 0.1187 0.1232
0.1097 0.1127 0.1129
Table 2: Novelty scores across different aggregated sets in round 2
prediction accuracy improves due to SFT. (iii) Instruction tuning enables novel query generation in subsequent batches within the same training round as novelty score reduces with increase of batch number (as seen in Tables 1, 2, 3). Algorithm 3 Saturation Check Algorithm Asat Require: Maximum number of batches to be generated Nmax , Linear fit Function flinear , Polynomial fit Function fpoly , execution trajectory T0 1: Initialize batch counter n ← 1, Saturation Set Ssat = 0 2: Initialize query set Q ← T0 , batch number x ← ∅ 3: Number of queries per batch y ← ∅ 4: IsSaturated(x, y) = False 5: while n ≤ Nmax and not IsSaturated(x, y) do 6: Obtain query retained, index retained from Asel (Algorithm 2) with P = Q 7: Q ← Q ∪ {query retained from Batch x} 8: y ← y ∪ len(index retained from Batch x) 9: xelbow ← 0, xcoarse ← linspace(min(x), max(x), 3) 10: yapprox ← flinear (x, fpoly (y))(xcoarse ) y [1]−yapprox [0] 11: slopestart ← xapprox coarse [1]−xcoarse [0] y
[2]−y
[1]
approx slopeend ← xapprox coarse [2]−xcoarse [1] if slopestart < slopeend and 2 ∈ y then Set elbow to middle point: xelbow ← xcoarse [1] nsat = nsat + 1, Ssat ∪ {n} if nsat == 2 and ∃k, k + 1 ∈ Ssat ; k < n − 1 then: 17: IsSaturated(x, y) = True 18: end if 19: end if 20: n ← n + 1, x ← x ∪ {n} 21: end while 22: return n, Q
12: 13: 14: 15: 16:
5
Conclusion
This work proposes a method for systematically and autonomously exploring the query space of a GUI application. It was demonstrated that the space of user-sensitive queries shrinks following subsequent training rounds. We hypothesize that a more aggressive search could be achieved by rejecting queries with a novelty score below the total rewards for the previous round besides the repeating ones in Algorithm 2. Furthermore, step-level exploration of sensitive screens from a single base query could enable a comprehensive coverage of screen space and include queries with high cosine similarity (in query) such as ‘turn on
Figure 5: Frequency of ‘Not-Critic’ screens across 3 training rounds
Figure 6: category novelty score across 3 training rounds
notification’ and ‘turn off notification’. These topics require further investigation and are left as future work. Appendices Given an example task trajectory T0 , the instruction I for explorer LM is in Listing 1. The frequency of screens which are categorized as ‘not critic’ increases in subsequent training rounds (in Figure 5) and as a result of this, the rewards for screen category for rolled out queries (in Figure 6) also reduces progressively.
References [Chen et al., 2024] Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, et al. Spa-bench: A comprehensive benchmark for smartphone agent
evaluation. In NeurIPS 2024 Workshop on Open-World Agents, 2024. [Chi et al., 2025] Haotian Chi, Zhaogeng Liu, Xing Chen, Bohao Qu, Jifeng Hu, Yuan Jiang, Hechang Chen, and Yi Chang. State transition difference prediction for deep reinforcement learning. Pattern Recognition, page 112824, 2025. [Fan et al., 2025] Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. Gui-bee: Align gui action grounding to novel environments via autonomous exploration. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33249–33266, 2025. [He et al., 2025] Feng He, Tianqing Zhu, Dayong Ye, Bo Liu, Wanlei Zhou, and Philip S Yu. The emerged security and privacy of llm agent: A survey with case studies. ACM Computing Surveys, 58(6):1–36, 2025. [Hua et al., 2024] Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. Trustagent: Towards safe and trustworthy llm-based agents through agent constitution. In Trustworthy Multi-modal Foundation Models and AI Agents (TiFA), 2024. [Koh et al., 2024] Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents. arXiv preprint arXiv:2407.01476, 2024. [Lee et al., 2024] Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting llm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pages 1119–1133, 2024. [Lin et al., 2025] Zongyu Lin, Yao Tang, Xingcheng Yao, Da Yin, Ziniu Hu, Yizhou Sun, and Kai-Wei Chang. Qlass: Boosting language agent inference via q-guided stepwise search. arXiv preprint arXiv:2502.02584, 2025. [Lu et al., 2025] Jiahao Lu, Ziwei Xu, and Mohan Kankanhalli. Reasoning llms are wandering solution explorers. arXiv preprint arXiv:2505.20296, 2025. [Murty et al., 2024] Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D Manning. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild. arXiv preprint arXiv:2410.02907, 2024. [Putta et al., 2024] Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint arXiv:2408.07199, 2024. [Qin et al., 2025] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025.
[Ran et al., 2024] Dezhi Ran, Hao Wang, Zihe Song, Mengzhou Wu, Yuan Cao, Ying Zhang, Wei Yang, and Tao Xie. Guardian: A runtime framework for llm-based ui exploration. In proceedings of the 33rd ACM SIGSOFT international symposium on software testing and analysis, pages 958–970, 2024. [Rawles et al., 2024] Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024. [Shi et al., 2025] Zijing Shi, Meng Fang, and Ling Chen. Monte carlo planning with large language model for text-based game agents. arXiv preprint arXiv:2504.16855, 2025. [Sun et al., 2025] Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. Gui-xplore: Empowering generalizable gui agents with one exploration. In Proceedings of the computer vision and pattern recognition conference, pages 19477–19486, 2025. [Wang et al., 2024a] Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. Advances in Neural Information Processing Systems, 37:2686–2710, 2024. [Wang et al., 2024b] Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. arXiv preprint arXiv:2401.16158, 2024. [Wu et al., 2024] Chen Henry Wu, Rishi Shah, Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried, and Aditi Raghunathan. Dissecting adversarial robustness of multimodal lm agents. arXiv preprint arXiv:2406.12814, 2024. [Xie et al., 2025] Bin Xie, Rui Shao, Gongwei Chen, Kaiwen Zhou, Yinchuan Li, Jie Liu, Min Zhang, and Liqiang Nie. Gui-explorer: Autonomous exploration and mining of transition-aware knowledge for gui agent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5650–5667, 2025. [Ye et al., 2025] Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, et al. Mobile-agent-v3: Fundamental agents for gui automation. arXiv preprint arXiv:2508.15144, 2025. [Zhai et al., 2025] Yuanzhao Zhai, Tingkai Yang, Kele Xu, Dawei Feng, Cheng Yang, Bo Ding, and Huaimin Wang. Enhancing decision-making for llm agents via step-level q-value models. In Proceedings of the AAAI conference on artificial intelligence, volume 39, pages 27161–27169, 2025.
[Zhang et al., 2024] Xinyu Zhang, Huiyu Xu, Zhongjie Ba, Zhibo Wang, Yuan Hong, Jian Liu, Zhan Qin, and Kui Ren. Privacyasst: Safeguarding user privacy in tool-using large language model agents. IEEE Transactions on Dependable and Secure Computing, 21(6):5242–5258, 2024. [Zhao et al., 2024] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. Expel: Llm agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024. [Zheng et al., 2024] Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. Gpt-4v (ision) is a generalist web agent, if grounded. arXiv preprint arXiv:2401.01614, 2024. [Zhou et al., 2023] Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023.
You are an assistant acting as a proxy user, and you need to create a new task description based on the example tasks provided by the user. 2 The task will be completed by an intelligent mobile GUI agent in a mobile environment, which will determine the specific actions for each step of the task. 3 Below is an example task that includes steps: 4 [Insert Example Task Trajectory Here] 5 As a proxy user, follow the steps shown above to learn about the application and create a single new task. 6 For example if ’Adjust shipping preference and update contact information’ is NOT a single task 7 whereas ’Adjust shipping preferences for future orders’ IS a single task. The app is Ebay (ECommerce/Shopping App). 8 Create diverse tasks and explore the possibilities of the application. Task with different parameters will be considered the same task: 9 In the Ebay app, for example, buying a mobile phone and buying clothes are not different tasks because they both involve similar steps, 10 such as opening app, searching for products, selecting specifications, and checkout. Similarly, in the Ctrip app, book hotels in Shenzhen and book hotels in Beijing. 11 It’s not a different mission, for the same reason. As a proxy user, your main task is to think about operational tasks that involve functions of the application. 1
12
(1) Do not generate task like those in the list: {instructions} (2) Think of tasks which can be completed assuming the past instructions and the example tasks are already completed in the app. 15 For example, if an instruction was ’add a product to the cart’, you can assume the product has been added and think of critic tasks based on this information. 16 (3) You need to create task that may involve key operations and steps. Key steps involve authentication, personal data, 17 Account management, purchasing products and privacy/location access. The categories of these screens should be in the list: {categories} 18 (4) The Task Goal should not exceed 15 words and must contain the application name Ebay. 19 (5) You should give all the steps of the task along with uiSummary, actionPurpose, category. 20 ’actionPurpose’ can contain a maximum of 50 words, in the same format as given in the example. 21 If there are critical actions on the screen, assign a category to them. It can be one of the following list: {categories} 22 If there is no critical action on the screen, give ’category’:’not_critic’. If no suitable category is found, give ’category’:’critic’. 23 (6) Output a python dictionary list in the following format. 24 [{ "taskGoal": "Task proposed by you as a proxy user", 25 "step1": {"uiSummary": "screen seen in step 1", "actionPurpose": "action along with its purpose", "category": "category, if it is a critic step; Otherwise, give not_critic"}, 26 "step2": {"uiSummary": "screen seen in step 2", "actionPurpose": "action along with its purpose", "category": "category, if it is a critic step; Otherwise, give not_critic"}, 27 #Add more steps as needed 28 "stepN": {"uiSummary": "screen seen in step N", "actionPurpose": "action along with its purpose", "category": "category, if it is a critic step; Otherwise, give not_critic"} }] 29 Ensure that the output complies with the Python syntax dictionary list. 13 14
30
Listing 1: Instruction I used for the Explorer LM to generate new taskGoal (query) and steps