UI-Copilot: Advancing Long-Horizon GUI Automation via Tool-Integrated Policy Optimization Zhengxi Lu1 , Fei Tang1 , Guangyi Liu1 , Kaitao Song2 , Xu Tan3 , Jin Ma3 Wenqi Zhang1 , Weiming Lu1 , Jun Xiao1 , Yueting Zhuang1 , Yongliang Shen1 * 1 2 3 Zhejiang University Apple Tencent {zhengxilu, syl}@zju.edu.cn
1
Introduction
Graphical User Interface (GUI) agents are designed to interact with digital environments in a humanlike manner (Hu et al., 2025; Zhang et al., 2025a; Wang et al., 2025a; Tang et al., 2025b; Liu et al., 2025b). Recent multimodal large language model (MLLM)-based GUI agents (Ye et al., 2025; Gu et al., 2025; Qin et al., 2025), trained via supervised fine-tuning (SFT) and reinforcement learning (RL), have demonstrated strong capability in * Corresponding author
Math
5.2% Memory 6.1%
40 30 20 10
88.7%
0
o B 7B 7B M 7B T-4 otCP UI- RS-7 I-S1GP pil A ent 8B eG U Co Ag-GUI- obil UI-T I U M
MemGUI-Bench
arXiv:2604.13822v1 [cs.LG] 15 Apr 2026
MLLM-based GUI agents have demonstrated strong capabilities in complex user interface interaction tasks. However, long-horizon scenarios remain challenging, as these agents are burdened with tasks beyond their intrinsic capabilities, suffering from memory degradation, progress confusion, and math hallucination. To address these challenges, we present UICopilot, a collaborative framework where the GUI agent focuses on task execution while a lightweight copilot provides on-demand assistance for memory retrieval and numerical computation. We introduce memory decoupling to separate persistent observations from transient execution context, and train the policy agent to selectively invoke the copilot as Retriever or Calculator based on task demands. To enable effective tool invocation learning, we propose Tool-Integrated Policy Optimization (TIPO), which separately optimizes tool selection through single-turn prediction and task execution through on-policy multiturn rollouts. Experimental results show that UI-Copilot-7B achieves state-of-the-art performance on challenging MemGUI-Bench, outperforming strong 7B-scale GUI agents such as GUI-Owl-7B and UI-TARS-1.5-7B. Moreover, UI-Copilot-7B delivers a 17.1% absolute improvement on AndroidWorld over the base Qwen model, highlighting UI-Copilot’s strong generalization to real-world GUI tasks.
AndroidWorld
Abstract
15 10
SOTA performance Average performance drop by 90.9%
Other Other
Math
10.2%
19.5%
5
70.3%
0
Memory
B B B B B 7B t-7 L-7 -1.5-7 -S1-7 nus-7 wlilo .5V I-O -Cop UI Ve n2 -TARS U I e G I U U Qw UI
Figure 1: Left: Performance on dynamic GUI benchmarks. Right: Task distribution of these benchmarks.
solving short-horizon tasks, which typically require fewer than 10 interaction steps (Rawles et al., 2024; Zhao et al., 2025; Chen et al., 2026), as shown in Figure 1 and 17). However, deploying these agents in longhorizon, memory-intensive scenarios exposes fundamental limitations. As shown in Figure 1, existing 7B models suffer an average performance drop of 90.90% on MemGUI-Bench (Liu et al., 2026). We identify three interconnected challenges underlying these failures (as illustrated in Figure 2): 1) Memory Degradation: overloaded context causes agents to lose or misrecall critical information from earlier steps; 2) Progress Confusion: interleaving reasoning traces with action histories obscures task state, leading to redundant actions, disordered subtask execution, or premature termination; and 3) Math Hallucination: numerical reasoning errors compound as incorrect intermediate results propagate through subsequent computations.
Screenshots
MemGUI-Bench Inference Case
Instruction: In Bing App, �ind the current stock prices for NVIDIA (NVDA) and Apple (AAPL). Then, calculate and answer combined value of 50 NVDA and 75 AAPL. Ground Truth: 50*169.70+75*226.7 6=25492 UI-TARS-1.5-7B: Memory Degradation
(step-15) Thought: I just checked the stock price on Bing, and it's $143.92. So, I'll go ahead and type in 143.92 to calculate ...
Qwen2.5VL-7B: Progress Confusion (step-9) Thought: I have gathered all the stock prices. The next step is to calculate...
GUI-Qwl-7B: Math Hallucination (Step-16) Thought: ... '169.70' for NVDA and 226.76’ for Apple. Now, I need to calculate product: \\(50 \\times 169.70 \\plus 75 \\times 226.76 = 29540\\). The next ...
UI-Copilot-7B: Complete the task (Step-9) Tool: None Thought: Current stock price for Apple is $226.76. I need ... (Step-15) Tool: Retriever Result: The price for Apple is $226.76. Thought: I've �inally gathered all the necessary information! Apple's stock is currently at $226.67, and NVIDIA's is at $169.70. (Step-16) Tool: Calculator Result: 25492 Thought: The total combined value of both holdings is 25492. So ... Action: type: answer, text: 25492
Qwen3-4B: Copilot Model (Step-15) Role: Retriver Thought: My goal is to... Given the interaction history, I have obtained the price... Answer: The price for Apple is $226.76.
(Step-16) Role: Calculator Thought: My task is to calculate the comibined value of... Python: def product(x1, n1,x2, n2): ret = x1*n1+x2*n2...
Figure 2: MemGUI-Bench Inference Case. Our method successfully completes the task by invoking Copilot Model, whereas other models fail due to memory degradation, progress confusion, and math hallucinations.
Existing approaches address these limitations through multi-agent workflows (Agashe et al., 2025; Wang et al., 2025b, 2024a) or retrieval augmentation (Liu et al., 2025a; Li et al., 2025b; Xu et al., 2025). However, multi-agent workflows rely on predefined pipelines that invoke external modules regardless of actual necessity, resulting in prohibitive inference costs. Retrieval-augmented methods depend heavily on retrieval quality and fail to resolve progress confusion. We attribute these limitations to a shared underlying cause:
single-turn supervision, while action generation learns through multi-turn rollouts conditioned on self-generated histories, aligning training dynamics with deployment conditions. Extensive experiments demonstrate that UI-Copilot-7B achieves SOTA performance on MemGUI-Bench, where over 90% of tasks require persistent memory, and attains 39.1% accuracy on AndroidWorld, validating the generalization of the proposed framework. In summary, our contributions are: • We propose UI-Copilot, a collaborative framework where the GUI agent selectively invokes a lightweight copilot for memory retrieval and numerical computation, enabling efficient long-horizon GUI navigation.
Agents are burdened with challenges beyond its capabilities, leading to confusion under increasingly overloaded context. Our key insight is that GUI agents should focus on task execution with lightweight context, while memory and computation are decoupled and invoked only when needed. Building on this insight, we introduce UI-Copilot, a collaborative framework that decouples persistent observations from transient execution context. Detailed reasoning traces are stored locally while only concise progress summaries remain in the dialogue history, keeping the context window focused and enabling on-demand information retrieval. The policy agent selectively invokes a lightweight copilot model as Retriever or Calculator, enabling adaptive tool usage that responds to actual task demands. To enable more effective tool invocation, we propose Tool-Integrated Policy Optimization (TIPO), which decouples tool prediction and task execution during training. Tool selection is trained via
• We introduce memory decoupling to separate persistent observations from transient context, effectively mitigating context overload. • We develop TIPO, a reinforcement learning algorithm that separately trains GUI agents’ tool invocation and action generation.
2
Related Work
2.1
Reinforcement Learning for GUI Agent
Recent advances in multimodal models have catalyzed significant progress in GUI automation (Hu et al., 2025; Zhang et al., 2025a; Wang et al., 2025a; Tang et al., 2025b; Liu et al., 2025b; Ye et al., 2025; Wu et al., 2026). Inspired by DeepSeek-R1 (Guo et al., 2025), recent work (Lu et al., 2025b; Luo et al., 2025a; Qin et al., 2025; Lu et al., 2025a; Gu
et al., 2025; Tang et al., 2025a; Du et al., 2025) has begun applying Group Relative Policy Optimization (GRPO) (Shao et al., 2024) to GUI automation. However, external tool calling for GUI agent training remain un-explored. 2.2
Memory for GUI Agent
Memory remains a fundamental challenge for GUI agents due to the limited context windows. Recent attempts, including multi-agent workflows (Wang et al., 2024a, 2025b; Agashe et al., 2025) and fewshot Retrieval-Augmented Generation (RAG) (Liu et al., 2025a; Li et al., 2025b; Xu et al., 2025), aim to enhance agent performance without model finetuning. While effective in certain scenarios, these approaches often suffer from limited scalability and high deployment costs. Additional works incorporate history-aware training mechanisms (Zhou et al., 2025; Liu et al., 2025c; Wang et al., 2025c; Lu et al., 2026), but still face memory degradation for memory-intensive, long-horizon GUI tasks.
3
Method
3.1
UI-Copilot
Problem Definition. We formulate GUI automation as a sequential decision-making problem. Given a task instruction I and initial screen state S0 , the agent generates a sequence of actions {a1 , a2 , . . . , aT } to complete the task. At each step t, the agent observes the current screenshot state St and samples an action at from policy πθ (at |I, St , Ht ), where θ denotes model parameters and Ht represents action history. The action space A includes coordinate-based operations (click, swipe, long_press), text-based operations (type, answer) and system-based operations (system_button, open, wait, terminate), as shown in Table 3. The environment E transitions to the next state according to St+1 = E(St , at ) and rollout continues until task finish or failure. Rollout Paradigm. Given a policy agent M (initialized from Qwen2.5VL-7B) and a copilot model Mc (Qwen3-4B), we define a tool-integrated multiturn summary rollout paradigm for M: <tool> T : tool call </tool> ↘ call copilot model <result> R: tool return </result> <think> thought </think> <action> a </action> <summary> summary </summary>
where the prompt is illustrated in Figure 27. The sampling policy is then written as at , summaryt , Tt , thoughtt ∼ πθ (·|I, St , Ht ) (1) where Tt ∈ {Calculator, Retriever, none} denotes the role assigned to the copilot model Mc at step t. For memory-intensive tasks, the Retriever is activated using promptR in Figure 28. It takes as input the history knowledge K (stored as a JSON file), the task instruction I, and progress summaries summary<t , and returns tool results to the policy agent in textual form: Rt = Mc (promptR , K, I, summary<t ). For numerical calculation tasks, the Calculator is invoked with promptC in Figure 29 to generate executable Python code. The generated code is then executed by a Python interpreter, and the resulting output is returned to the rollout process: Rt = PythonExecutor(Mc (promptC , I, summary<t )). Memory Decoupling. Existing agents (Lu et al., 2025c; Ye et al., 2025) maintain the full reasoning content during multi-step rollouts, updating the history as Ht = Ht−1 ∪ {at−1 , thoughtt−1 }, which we refer to as a multi-turn context (MC). However, it could lead to progress confusion (e.g., redundant steps, disordered sub-task execution, or premature termination) due to content overload, as illustrated in Figure 2 and 9. To mitigate this, we propose a multi-turn summary (MS) paradigm that decouples progress tracking from detailed reasoning. In the dialogue history, we maintain only a concise summary reporting current completion status (e.g., "I have finished sub-task A"): Ht = Ht−1 ∪ {at−1 , summaryt−1 }. The full reasoning content thoughtt−1 , which includes the agent’s planning and explicit history observations (e.g., "The stock price is 45 dollars"), is stored locally in a text file K: K = K ∪ thoughtt−1 . This decoupling reduces context overload in the multiturn dialogue while preserving detailed information for retrieval or numerical reasoning. Conclusion. We formalize a multi-turn summary interaction paradigm that leverages a copilot model as tools. This offers several advantages: 1) Lightweight Context Window. By maintaining only concise progress summaries in the dialogue, the agent can focus on task execution, reducing the risk of planning hallucinations. 2) Decoupled Memory. Detailed observations are stored locally and retrieved on demand, mitigating information
Instruction ~ I
Rollout
Reward
... Tn1 ...
rformat
I
Multi-turn Rollout
a11
a21
...
a2
... a N t
a1
N
Tool Call Learning
rformat
N
Ref Model
KL Divergence
A0 A1
Group
...
rtool
at1
Reward
...
...
... ... T N ... m
Step-level Advantage
Instruction ~
Rfuture
Policy Model
Policy update
Vision Encoder
AN
Language Model
Action Prediction Learning
Figure 3: Overview of TIPO Pipeline. Policy model jointly learns tool invocations and multi-turn action prediction.
loss and memory hallucination. 3) Efficient Inference. Unlike previous agent workflows (Wang et al., 2024a), the agent selectively invokes external models, simplifying the execution pipeline. 3.2
Dataset Curation
We collect N diverse, human-annotated trajectories τ ∗ = {(S1∗ , a∗1 ), . . . , (ST∗ , a∗T )} from AndroidControl (Li et al., 2024). Further, we use GPT4o to synthesize tool call content tool∗t , reasoning content thought∗t and summary summary∗t for each step t in each trajectory, forming the expert dataset Dexpert = {τi∗ }N i=1 . For tool call learning, we use GPT-4o (Hurst et al., 2024) to form memoryintensive or calculation-needed queries based on Dexpert , named as Dtool . Then we merge them as D0 = Dexpert ∪ Dtool , then randomly split D0 into RL and D RL DSFT , Dtool action respectively for SFT and RL training. This process is detailed in Figure 4. Vanilla queries
Expert Trajectory
Screenshot: Summay: I have opened ... Task: In the any.do app, create a reminder HR meeting GT Tool: None
Memory-intensive queries Screenshot: Summay: I have ... Task: What’s the price of the Nike Fly.By Mid 3 shoe? GT Tool: Retriver
Calculation-needed queries
Screenshot: Summay: I have ... Task: How much is 5 pairs of Nike Fly.By Mid 3 shoe and 4 pieces of... ? GT Tool: Calculator
Figure 4: Training Dataset Curation Pipeline.
3.3
Tool-Integrated Policy Optimization
Our GUI agent M is efficiently trained on the pseudo-labeled D0 , during which the copilot model Mc is not involved.
Cold Start. We initialize the policy via SFT on Qwen2.5VL-7B using DSFT . Training is performed with a standard cross-entropy loss for next-token prediction, which enables both format learning and behavior cloning from expert trajectories. LSFT (θ) = −E(I,{St },{at })∼DSFT " T # X log πθ (at | I, St , Ht )
(2)
t=1
Decoupled Sampling. We model the agentic reasoning process in Equation 7 as explicit tool invocation, then our sampling is written as: T Y Pθ (T , a | I; T) = Pθ (Tt | Ht , I; T) · |t=1 T Y
{z
Tool Calling
}
(3)
Pθ (at | a<t , Ht , I; T))
t=1
|
{z
Action Generation
}
where T denotes the set of available tools. Accordingly, we adopt a decoupled strategy that separates (i) tool-calling rollouts from actiongeneration rollouts, and (ii) tool-call learning from action learning, depending on the source of instrucRL , we only conduct single-turn tion I. For I ∼ Dtool tool prediction, conditioned on off-policy history RL , we follow Ht∗ = summary∗<t . For I ∼ Daction Lu et al. (2025c) and conduct multi-turn action prediction, conditioned on self-generated history Htπ = summaryπ<t . Equation 3 is modified as: T Y ∗ RL Pθ (Tt | Ht , I; T) , if I ∼ Dtool , Pθ ≈ t=1 T Y RL Pθ (at | a<t , Htπ , I) , if I ∼ Daction t=1
Tool Call Learning. Unlike agentic reasoning that provides tool feedback from environment, we compute rule-based reward. For i-th rollout, Rti = 0.1 · rformat + 0.9 · I[rformat =1] · rtool
(4)
Models
Type
CogAgent AO Qwen2.5VL-7B* AT UI-Venus-7B AT UI-S1-7B* MC UI-TARS-1.5-7B MC GUI-Owl-7B MC UI-Copilot-7B* TC Mobile-Agent-V2 MW SeeAct MW AppAgent MW Mobile-Agent-E MW T3A MW M3A MW Agent-S2 MW
1 App p@1 p@3 0.0 0.0 14.3 17.9 21.4 28.6 17.9 21.4 14.3 21.4 21.4 35.7 42.9 50.0 14.3 17.9 10.7 25.0 14.3 42.9 25.0 42.9 42.9 60.7 46.4 64.3 50.0 78.6
#Cross App 2 App 3 App p@1 p@3 p@1 p@3 0.0 0.0 0.0 0.0 1.8 1.8 0.0 0.0 1.8 1.8 0.0 2.9 3.6 3.6 0.0 0.0 0.0 1.8 0.0 2.9 1.8 1.8 2.9 5.9 12.5 16.1 5.9 8.8 0.0 0.0 0.0 0.0 0.0 0 0.0 0 0.0 0.0 0.0 0.0 0.0 1.8 0.0 0.0 16.1 37.5 23.5 38.2 28.6 41.1 29.4 44.1 19.6 35.7 26.5 52.9
4 App p@1 p@3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.0 0.0 0.0 0.0 0 0.0 0.0 0.0 0.0 0.0 30.0 30.0 50.0 10.0 30.0
Easy p@1 p@3 0.0 0.0 10.4 12.5 14.6 20.8 12.5 14.6 8.3 16.7 14.6 22.9 29.2 33.3 8.3 10.4 6.2 12.5 8.3 22.9 12.5 22.9 31.2 45.8 39.6 47.9 41.7 64.6
Difficulty Level Med Hard p@1 p@3 p@1 p@3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.6 2.6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.4 2.6 2.6 13.2 18.4 4.8 7.1 0.0 0.0 0.0 0.0 0.0 2.4 0.0 0.0 0.0 2.4 0.0 0.0 2.4 2.4 0.0 2.6 16.7 45.2 18.4 34.2 35.7 50.0 21.1 44.7 19.0 42.9 18.4 36.8
Avg p@1 p@3 0.0 0.0 3.9 4.7 5.5 7.8 5.5 6.2 3.1 6.2 6.2 10.2 16.4 20.3 3.1 3.9 2.3 5.5 3.1 9.4 5.5 10.2 22.7 42.2 32.8 47.7 27.3 49.2
Table 1: Results on MemGUI-Bench. p@k denotes pass@k. Results marked with * are evaluated by ourselves with tool usage, while the remaining results are reported from the benchmark paper, where pass@3 is tested with long-term memory. The best performance in each column is highlighted in bold, and the second best is underlined. AO denotes Action-Only rollout without history. AT denotes Action-Thought history management. MC denotes Multi-turn Context. TC denotes our Tool-Integrated multi-turn Context rollout. MW denotes Multi-agent Workflow, with Gemini-2.5 Pro (Comanici et al., 2025) as planning agents.
Action Prediction Learning. For multi-turn execution learning, we train on a dataset without tool calling. First, we compute the step-wise reward as: rti = 0.1 · rformat + 0.4 · I[rformat =1] · rtype + 0.5 · I[rformat ·rtype =1] · racc
(5)
where all the rewards are defined in Appendix B. Then introduce discounted future reward Rti = Ptendwe,k−t rki and compute the advantage for the k=t γ individual tokens using the normalized reward Ri : Ri −mean({Ri }G ) Ai,t = t std({Ri }Gt i=1 , where G is the total numt i=1 ) ber of samples within a group, which is set as 8. The training objective of TIPO is: G T |oi,t | 1 XX X JTIPO (θ) = EI∼DRL ,{oi,t }G,T 1,1 K i=1 t=1 k=1 ∼πθold (·|I) (6) min(ρ(θ)Ai,t , clip(ρ(θ), 1 ± ϵ)Ai,t )) − β DKL (πθ ∥ πref ) RL ∪ D RL where DRL = Dtool action denotes the RL dataset, K is the total number of tokens, ρ(θ) = πθ (oi,t,k |I,oi,t,<k ) πθold (oi,t,k |I,oi,t,<k ) is the importance sampling ratio, and β controls the KL penalty strength. To ensure effective learning, we enforce minimum advantage variance: σ({Ai,t }) > η (η set as 0.3), performing dynamic sampling until this threshold is met.
4
Experiment
4.1
Experiment Setup
Baselines. To comprehensively assess the performance of UI-Copilot, we include three kinds of
baselines: (1) advanced proprietary models, including GPT-4o (Hurst et al., 2024) and Claude (Anthropic, 2024); (2) SOTA open-source models, such as AgentCPM-GUI (Zhang et al., 2025b), GUIOwl (Ye et al., 2025), UI-TARS-1.5 (Qin et al., 2025) and UI-Venus (Gu et al., 2025); (3) multiagent workflows, such as Mobile-Agent-E (Wang et al., 2025b), Mobile-Agent-V2 (Wang et al., 2024a) and Agent-S2 (Agashe et al., 2025). Benchmarks. To highlight UI-Copilot-7B’s strengths in memory- and math-intensive tasks, we first evaluate all models on the challenging MemGUI-Bench (Liu et al., 2026), which consists of 70.3% memory-intensive and 19.5% mathintensive tasks, with an average of 36 golden steps. We then assess UI-Copilot on the widely used dynamic benchmarks AndroidWorld and MiniWob++ (Rawles et al., 2024) to validate its improvements in multi-turn performance. We additionally include AC-Real (referred to as SOP in UI-S1 (Lu et al., 2025c)), reporting both progress (PG) and task success rate (TSR). We also adopt the static GUI navigation benchmarks AndroidControl (Li et al., 2024) and GUI Odyssey (Lu et al., 2024) to evaluate comprehensive GUI understanding under high-level instructions, with action type match accuracy (TM), grounding accuracy rate (GR) and step success rate (SR) reported. To further evaluate the generalization ability of UI-Copilot-7B, we assess grounding and low-level interaction capabilities, as reported in
Models TM Closed-source Models Claude-CU (SoM) (Anthropic, 2024) GPT-4o (SoM) (Hurst et al., 2024) Open-source Models Qwen2VL-2B (Wang et al., 2024b) ShowUI-2B (Lin et al., 2024) OS-Genesis-7B (Sun et al., 2024) OS-Atlas-7B (Wu et al., 2024) Qwen2.5VL-3B (Bai et al., 2025) Qwen2.5VL-7B (Bai et al., 2025) UI-R1-3B (Lu et al., 2025b) UI-R1-7B (Lu et al., 2025b) AgentCPM-GUI-8B (Zhang et al., 2025b) UI-S1-7B (Lu et al., 2025c) UI-TARS-7B (Qin et al., 2025) Ours Models UI-Copilot-3B UI-Copilot-7B
AC-High GR SR
GUI Odyssey TM GR SR
AC-Real PG TSR
Wob SR
AW SR
Avg SR
63.7 66.3
0.0 0.0
12.5 20.8
60.9 34.3
0.0 0.0
3.1 3.3
– –
– –
– 62.0
27.9 34.5
– –
42.3 41.8 65.9 57.4 47.8 62.2 57.9 72.4 77.7 79.9 83.7
18.7 32.8 – 54.9 46.5 72.5 55.7 62.8 – 73.4 80.5
13.6 19.7 44.4 29.8 38.9 52.7 45.4 54.2 69.2 68.2 72.5
24.4 34.8 11.7 60.4 37.4 67.4 52.2 67.1 90.8 76.3 94.6
12.2 24.6 – 39.7 26.5 56.3 34.5 41.3 – 61.7 90.1
12.6 21.4 3.6 27.0 26.7 52.4 32.5 43.5 75.0 59.5 87.0
2.0 6.8 7.6 14.3 3.4 17.4 8.4 16.9 17.1 32.4 28.1
1.0 2.6 3.0 8.6 1.4 9.8 4.1 10.8 10.6 16.3 14.0
20.8 27.1 19.8 35.2 24.1 54.0 26.1 45.2 37.8 60.9 58.7
0.0 7.0 17.4 12.1 5.0 22.0 8.2 15.1 16.4 34.0 33.0
7.3 11.7 13.4 18.6 10.2 28.6 12.8 23.7 21.6 37.1 35.2
64.3 82.9
54.5 72.2
50.3 71.8
52.4 74.5
36.8 63.8
35.8 57.2
15.6 31.5
6.9 15.8
25.9 61.2
15.7 39.1*
16.2 38.7
Table 2: Results on Other GUI Benchmarks. * shows the result with tool calling, and UI-Copilot-7B achieves 32.2% accuracy without tool usage. Wob denotes MiniWob++ and AW denotes AndroidWorld. Average SR is computed as average of AC-Real-TSR, Wob-SR and AW-SR. The highest value is in bold, the second is underlined.
Table 4. 4.2
Main Results
Model Comparison. As shown in Table 1, UICopilot-7B achieves SOTA performance among 7B models on the challenging MemGUI-Bench. It attains a pass@1 accuracy of 16.4% and a pass@3 accuracy of 20.3%, substantially outperforming strong baselines like GUI-Owl-7B and UITARS-1.5-7B, which achieve up to 10.2% accuracy. Moreover, UI-Copilot-7B achieves performance comparable to agentic workflows, including Mobile-Agent-E (5.5%), AppAgent (3.1%), and T3A (22.7%), highlighting the efficiency of our UI-Copilot paradigm. Notably, UI-Copilot-7B successfully solves some Hard tasks that require over 40 steps or 4 Apps, underscoring its potential for long-horizon, memory-intensive GUI tasks.
namic benchmarks such as MiniWob++ (61.2%) and AndroidWorld (39.1%), with results comparable to closed-source models like GPT-4o. Taken together, these results indicate that our model serves as a comprehensive GUI agent, excelling not only in long-horizon tasks but also in general scenarios. 4.3
Training Dynamics
The training dynamics in Figure 5 reveal critical insights. [Insight 1] Accuracy dynamics: Model accuracy steadily improves as training proceeds, and converges after approximately 40 training steps, indicating sufficient policy optimization. [Insight 2] Tool-calling dynamics: The frequency of tool in-
Training Effect. Under Tool-Integrated setting, TIPO yields substantial improvements over the base model, Qwen2.5VL-7B, demonstrating the effectiveness of our rollout and training strategy. A detailed analysis of the improvements introduced by TIPO is provided in Figure 9. General Performance. As shown in Table 2, both our 3B and 7B models achieve substantial improvements over their base models on AC-High and GUI Odyssey, demonstrating the effectiveness of TIPO for GUI grounding and high-level understanding. Furthermore, UI-Copilot-7B attains advancing performance among 7B models on dy-
Figure 5: Training Dynamics of UI-Copilot-7B on our selected AndroidWorld-Verified (60 tasks), MemGUIBench-Verified (55 tasks) subsets, Tool-call-Test (1000 tasks from D0 ) and AC-Real-Test (1536 tasks).
Mc
Method
MemGUI-Bench* AndroidWorld*
✓
MW ✓
Avg
Avg
SFT RL
step↓
Acc(%)↑
step↓
Acc(%)↑
step↓
Acc↑
PG↑
TSR↑
Acc↑
Tool
25.5
33.4
68.3
27.8
46.9
30.6
73.6
17.4
9.88
33.6
Cal Ret Acc(%)↑
Tool
AC-Real
RL Action
21.8
35.2
53.3
25.0
37.5
30.1
86.4
17.6
10.1
38.0
✓
AT
9.1
20.3
35.0
13.0
22.1
16.6
94.4
16.8
9.95
40.4
✓
MC
10.9
19.5
58.3
14.4
34.6
16.9
91.2
28.6
14.1
44.6
MS
10.9
18.7
65.0
13.1
38.0
15.9
83.6
32.4
16.5
44.2
✓
On-policy
20.0
19.4
66.7
13.4
43.4
16.4
86.2
22.6
10.3
39.7
✓
Off-policy
✓
21.8
20.1
67.3
14.0
44.5
17.1
95.6
21.5
10.2
42.4
✓
✓
Off-policy
✓
36.4
19.3
66.7
13.8
51.5
16.6
95.0
31.0
16.1
47.4
✓
✓
On-policy
MW
MS
✓
MS MS
✓
best
✓ ✓
On-policy
RL | : |D RL | |Daction tool
Copilot Model UI-Copilot-7B
23.6
20.8
51.7
15.3
37.7
18.0
91.0
30.8
15.8
45.9
200:600
Qwen2.5VL-7B
30.9
21.2
53.3
14.2
42.1
17.7
91.2
31.5
16.3
46.3
300:1000
Qwen3-0.6B
27.3
19.9
63.3
13.9
45.3
16.9
95.6
29.8
15.2
46.9
600:1000
Qwen3-1.7B
30.9
19.5
61.7
13.6
46.3
16.6
95.0
31.0
16.1
47.4
600:2000
Qwen3-4B
36.4
19.3
66.7
13.8
51.6
16.6
93.4
31.3
16.2
47.0
600:2400
worst
Figure 6: Ablation Study on Inference Strategies. * de- Figure 7: Ablations on Training Paradigms notes the verified subset. Cal and Ret denote Calculator and Dataset. Tool calling and multi-turn perand Retriever, respectively. All models M are fine-tuned formance are tested on Tool-call-Test (1000 on DRL . Multi-agent workflow (MW) includes UI-Copilot- tasks) and AC-Real (1536 tasks). On/Offpolicy depends on the history summary. 7B and Qwen3-4B. The step includes tool invocations.
vocations consistently decreases during training, indicating the model’s improving ability to use external tools. Notably, compared to AndroidWorld (approximately 6% tool usage), more complex tasks such as MemGUI-Bench demand significantly higher tool utilization (nearly 13%) and require a longer training phase for tool invocation to stabilize. [Insight 3] Execution efficiency dynamics: As training progresses, the average execution steps decrease, suggesting that RL effectively reduces redundant actions and mitigates progress confusion, leading to more efficient completion. 4.4
Ablation Analysis
Ablations on Rollout Paradigm. As shown in Figure 6 (upper part), under the setting without tool calling, ours rollout paradigm, Multi-turn Summary (MS), consistently achieves higher accuracy with fewer execution steps than Action–Thought (AT) and Multi-turn Context (MC). This indicates that MS effectively mitigates redundant actions and mitigates progress confusion during multi-turn execution. Based on the MS paradigm, we further conduct tool-set ablations. The results show that both the Calculator and Retriever contribute to improved performance on MemGUI-Bench. Notably, the full collaborative tool set achieves the best overall performance, with 51.5% average accuracy and 16.6 average steps. It also attains competitive performance against the multi-agent workflow which invokes the copilot model at every step, demon-
strating the efficiency of UI-Copilot and TIPO. Ablations on Copilot Model. We compare different copilot models in Figure 6 (bottom part). Among all models, Qwen3-4B achieves the best performance, outperforming Qwen3-0.6B, Qwen1.7B, and MLLMs such as Qwen2.5VL-7B. This result highlights the strong capability of Qwen34B in context understanding and summarization, which is crucial for effective copilot assistance. Ablations on TIPO. We conduct ablations on training paradigms (TIPO) in Figure 7 (upper part). The results indicate that SFT plays a critical role as a cold start, providing a reliable initialization for subsequent RL training. Furthermore, both tool call RL and action prediction RL are essential for effective tool calling (see Tool-call-Test) and stable multi-turn execution (see AC-Real), respectively. For action prediction learning, on-policy (multiturn self-generated) histories consistently outperform off-policy (expert-collected) histories, due to the better alignment with multi-turn evaluation. Ablations on Training Dataset. Figure 7 (bottom part) demonstrates 600:2000 as an optimal data RL | : |D RL |), which achieves a favorratio for (|Daction tool able balance between tool-call learning and action RL generation learning. Increasing the size of Daction RL or Dtool does not yield further improvements, indicating diminishing returns from additional actionlevel or tool-level learning samples.
Figure 8: Case Study. UI-Copilot-7B successfully completes a math-related task from AndroidWorld (top) and a memory-related task from MemGUI-Bench (bottom).
4.5
Case Study and Analysis
Case Study. Figure 8 presents two challenging cases from AndroidWorld and MemGUI-Bench, which require numerical computation and information retrieval, respectively. In both scenarios, UICopilot-7B successfully invokes the Copilot model as the appropriate external tool. Specifically, in the AndroidWorld example, the Calculator generates executable code at step 9 and returns the computed results to UI-Copilot-7B. In the MemGUI-Bench Qwen2.5VL-7B + Qwen3-4B Memory Other 13.3% 13.3% (8) (8) 33.3% (20)
36.7% (22) Math
Memory 31.5% (23) Other 2.7%
8.2%
Progress
66.7% (40)
AndroidWorld-Veri�ied
Qwen2.5VL-7B+ Qwen3-4B
43.8% (32)
11.7% (7)
15.0% (9)
Correct
3.3%
Progress
UI-Copilot-7B + Qwen3-4B Memory Other Progress 6.7%
13.7% Correct (10)
Math
Correct
UI-Copilot-7B + Qwen3-4B Other Memory
10.9% 21.8% (12) 23.6% (13) Progress
5.5%
MemGUI-Bench-Veri�ied Math
38.2% (21)
Correct
Figure 9: Error Type Analysis with Tool Usage. Errors are categorized into Memory Degradation, Progress Confusion, Math Hallucination and Other Fault.
case, the Retriever gathers critical information at steps 5, 12, and 18, enabling UI-Copilot-7B to correctly complete the form-filling task at step 23. Additional successful and failed cases are provided in Appendix F, further demonstrating the effectiveness of our tool-invocation and training method. Error Type Analysis. Figure 9 illustrates several representative error types categorized using GPT4o between TIPO (MS with tool) and Qwen2.5VL7B (AT with tool). Among all error categories, Progress Confusion emerges as the dominant failure mode on both benchmarks. For the more challenging MemGUI-Bench (only 8.2% success rate orignally), MLLMs further suffer from more reasoning limitations, including Memory Degradation and Math Hallucination. As results demonstrate, our UI-Copilot-7B achieves substantial improvements over the base Qwen2.5VL-7B (+33.4% on AndroidWorld and +30.0% on MemGUI-Bench). From the error-type perspective, Progress Confusion is almost halved, while Memory Degradation and Math Hallucination are significantly mitigated, demonstrating the effectiveness of our TIPO. Error analysis of tool invocation is shown in Figure 20.
5
Conclusion
We incorporate Copilot Model as external tools and propose TIPO for tool-integrated learning, aiming to solve complex and long-horizon GUI tasks. Our UI-Copilot-7B achieves consistently strong performance across several challenging GUI benchmarks.
Limitations Currently, our tool set is limited to Calculator and Retriever. However, real-world GUI scenarios often require a broader spectrum of tools, such as web search and visual cropping. Extending our framework to support more diverse tool integrations remains an important direction for future work.
References Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, and Xin Eric Wang. 2025. Agent s2: A compositional generalist-specialist framework for computer use agents. Preprint, arXiv:2504.00906. Anthropic. 2024. Developing a computer use model. https://www.anthropic.com/news/ developing-computer-use. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Tongbo Chen, Zhengxi Lu, Zhan Xu, Guocheng Shao, Shaohan Zhao, Fei Tang, Yong Du, Kaitao Song, Yizhou Liu, Yuchen Yan, Wenqi Zhang, Xu Tan, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026. Knowu-bench: Towards interactive, proactive, and personalized mobile agent evaluation. Preprint, arXiv:2604.08455. Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing gui grounding for advanced visual gui agents. arXiv preprint arXiv:2401.10935. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, and 1 others. 2025. Agentic reinforced policy optimization. arXiv preprint arXiv:2507.19849.
International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, and 1 others. 2025. Ui-venus technical report: Building high-performance ui agents with rft. arXiv preprint arXiv:2508.10833. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948. Xueyu Hu, Tao Xiong, Biao Yi, Zishu Wei, Ruixuan Xiao, Yurun Chen, Jiasheng Ye, Meiling Tao, Xiangxin Zhou, Ziyu Zhao, and 1 others. 2025. Os agents: A survey on mllm-based agents for general computing devices use. arXiv preprint arXiv:2508.04482. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025a. Screenspot-pro: Gui grounding for professional high-resolution computer use. arXiv preprint arXiv:2504.07981. Runze Li, Yuwen Zhai, Bo Xu, LiWu Xu, Nian Shi, Wei Zhang, Ran Lin, and Liang Wang. 2025b. Echotrailgui: Building actionable memory for gui agents via critic-guided self-exploration. arXiv preprint arXiv:2512.19396. Wei Li, William Bishop, Alice Li, Chris Rawles, Folawiyo Campbell-Ajala, Divya Tyamagundlu, and Oriana Riva. 2024. On the effects of data scale on computer control agents. arXiv e-prints, pages arXiv– 2406. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025c. Torl: Scaling tool-integrated RL. CoRR, abs/2503.23383.
Yong Du, Yuchen Yan, Fei Tang, Zhengxi Lu, Chang Zong, Weiming Lu, Shengpei Jiang, and Yongliang Shen. 2025. Test-time reinforcement learning for gui grounding via region consistency. arXiv preprint arXiv:2508.05615.
Shuquan Lian, Yuhang Wu, Jia Ma, Yifan Ding, Zihan Song, Bingqi Chen, Xiawu Zheng, and Hui Li. 2025. Ui-agile: Advancing gui agents with effective reinforcement learning and precise inference-time grounding. arXiv preprint arXiv:2507.22025.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth
Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2024. Showui: One vision-language-action model for gui visual agent.
Guangyi Liu, Pengxiang Zhao, Yaozhen Liang, Qinyi Luo, Shunye Tang, Yuxiang Chai, Weifeng Lin, Han Xiao, WenHao Wang, Siheng Chen, and 1 others. 2026. Memgui-bench: Benchmarking memory of mobile gui agents in dynamic environments. arXiv preprint arXiv:2602.06075. Guangyi Liu, Pengxiang Zhao, Liang Liu, Zhiming Chen, Yuxiang Chai, Shuai Ren, Hao Wang, Shibo He, and Wenchao Meng. 2025a. Learnact: Fewshot mobile gui agent with a unified demonstration benchmark. arXiv preprint arXiv:2504.13805. Guangyi Liu, Pengxiang Zhao, Liang Liu, Yaxuan Guo, Han Xiao, Weifeng Lin, Yuxiang Chai, Yue Han, Shuai Ren, Hao Wang, and 1 others. 2025b. Llm-powered gui agents in phone automation: Surveying progress and prospects. arXiv preprint arXiv:2504.19838. Tao Liu, Chongyu Wang, Rongjie Li, Yingchen Yu, Xuming He, and Bai Song. 2025c. Gui-rise: Structured reasoning and history summarization for gui navigation. arXiv preprint arXiv:2510.27210. Zikang Liu, Junyi Li, Wayne Xin Zhao, Dawei Gao, Yaliang Li, and Ji-rong Wen. 2025d. Pal-ui: Planning with active look-back for vision-based gui agents. arXiv preprint arXiv:2510.00413. Fanbin Lu, Zhisheng Zhong, Shu Liu, Chi-Wing Fu, and Jiaya Jia. 2025a. Arpo: End-to-end policy optimization for gui agents with experience replay. arXiv preprint arXiv:2505.16282. Quanfeng Lu, Wenqi Shao, Zitao Liu, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Gui odyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. arXiv preprint arXiv:2406.08451. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. 2025b. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620. Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026. Skill0: In-context agentic reinforcement learning for skill internalization. arXiv preprint arXiv:2604.02268. Zhengxi Lu, Jiabo Ye, Fei Tang, Yongliang Shen, Haiyang Xu, Ziwei Zheng, Weiming Lu, Ming Yan, Fei Huang, Jun Xiao, and 1 others. 2025c. Ui-s1: Advancing gui automation via semi-online reinforcement learning. arXiv preprint arXiv:2509.11543. Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. 2025a. Gui-r1: A generalist r1-style visionlanguage action model for gui agents. arXiv preprint arXiv:2504.10458.
Zhihao Luo, Wentao Yan, Jingyu Gong, Min Wang, Zhizhong Zhang, Xuhong Wang, Yuan Xie, and Xin Tan. 2025b. Navimaster: Learning a unified policy for gui and embodied navigation tasks. arXiv preprint arXiv:2508.02046. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Uitars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo CampbellAjala, and 1 others. 2024. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, and 1 others. 2024. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. arXiv preprint arXiv:2412.19723. Fei Tang, Zhangxuan Gu, Zhengxi Lu, Xuyang Liu, Shuheng Shen, Changhua Meng, Wen Wang, Wenqi Zhang, Yongliang Shen, Weiming Lu, and 1 others. 2025a. Gui-g2 : Gaussian reward modeling for gui grounding. arXiv preprint arXiv:2507.15846. Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025b. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865. Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024a. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multi-agent collaboration. arXiv preprint arXiv:2406.01014. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, and 1 others. 2024b. Qwen2vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. Xinyuan Wang, Bowen Wang, Dunjie Lu, Junlin Yang, Tianbao Xie, Junli Wang, Jiaqi Deng, Xiaole Guo, Yiheng Xu, Chen Henry Wu, and 1 others. 2025a. Opencua: Open foundations for computer-use agents. arXiv preprint arXiv:2508.09123.
Zhenhailong Wang, Haiyang Xu, Junyang Wang, Xi Zhang, Ming Yan, Ji Zhang, Fei Huang, and Heng Ji. 2025b. Mobile-agent-e: Self-evolving mobile assistant for complex tasks. arXiv preprint arXiv:2501.11733. Ziwei Wang, Leyang Yang, Xiaoxuan Tang, Sheng Zhou, Dajun Chen, Wei Jiang, and Yong Li. 2025c. History-aware reasoning for gui agents. arXiv preprint arXiv:2511.09127. Jinyang Wu, Shuo Yang, Changpeng Yang, Yuhao Shen, Shuai Zhang, Zhengqi Wen, and Jianhua Tao. 2026. Spark: Strategic policy-aware exploration via dynamic branching for long-horizon agentic learning. arXiv preprint arXiv:2601.20209. Junde Wu, Jiayuan Zhu, and Yuyuan Liu. 2025. Agentic reasoning: Reasoning llms with tools for the deep research. arXiv preprint arXiv:2502.04644. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and 1 others. 2024. Os-atlas: A foundation action model for generalist gui agents. arXiv preprint arXiv:2410.23218. Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C Ho, Carl Yang, and Dong Yu. 2025. Retrieval-augmented gui agents with generative guidelines. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17877–17886. Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. 2024. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint arXiv:2412.04454. Jiabo Ye, Xi Zhang, Haiyang Xu, Haowei Liu, Junyang Wang, Zhaoqing Zhu, Ziwei Zheng, Feiyu Gao, Junjie Cao, Zhengxi Lu, Jitong Liao, Qi Zheng, Fei Huang, Jingren Zhou, and Ming Yan. 2025. Mobileagent-v3: Fundamental agents for gui automation. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yanda Li, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. 2025a. Appagent: Multimodal agents as smartphone users. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20. Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, and 1 others. 2025b. Agentcpm-gui: Building mobile-use agents with reinforcement finetuning. arXiv preprint arXiv:2506.01391. Pengxiang Zhao, Guangyi Liu, Yaozhen Liang, Weiqing He, Zhengxi Lu, Yuehao Huang, Yaxuan Guo, Kexin Zhang, Hao Wang, Liang Liu, and 1 others. 2025. Mas-bench: A unified benchmark for shortcutaugmented hybrid mobile gui agents. arXiv preprint arXiv:2509.06477.
Xurui Zhou, Gongwei Chen, Yuquan Xie, Zaijing Li, Kaiwen Zhou, Shuai Wang, Shuo Yang, Zhuotao Tian, and Rui Shao. 2025. Hiconagent: History context-aware policy optimization for gui agents. arXiv preprint arXiv:2512.01763.
A
Action Space
Action Type
Description
click
Tap a specific coordinate (x, y) on the screen. Press and hold at (x, y) for a specified duration. Perform a swipe gesture from (x1 , y1 ) to (x2 , y2 ). Output a textual answer to the task. Enter text into the currently focused input field. Trigger a system-level button (e.g., Home, Back). Launch an APP on the device. Pause execution for a given number of seconds to allow UI changes. Stop execution and report task success or failure.
long_press swipe answer type system_button open wait terminate
Table 3: Action space in AndroidWorld automation.
System Message You are a GUI agent. You are given a task and your action history, with screenshots. You need to ...
Reward Definition
B.1
Type Reward (rtype )
rtype ∈ {0, 1} indicates whether the predicted action type matches the ground-truth action type. Let apred and agt denote the predicted and ground-truth action types, respectively. The type reward is defined as rtype = I[apred = agt ]. B.2
Accuracy Reward (racc )
racc ∈ {0, 1} evaluates whether the predicted action is accurate given the ground-truth action, conditioned on the action type being correct and after coordinate normalization. Let ppred and pgt denote the predicted and ground-truth coordinates, respectively. • Wait / Terminate The prediction is accurate if the action type exactly matches: racc = I[apred = agt ]. • System Button
Action Space
User Message
Instruction Open the �ile task.html in Downloads in the �ile manager; when prompted open it with Chrome. Then click the button 5 times, remember the numbers displayed, and enter their product in the form.
Screen
Let buttonpred and buttongt denote the predicted and ground-truth system button names. The prediction is accurate if the button names match in a case-insensitive manner: racc = I[buttonpred = buttongt ]. • Type / Answer / Key / Open
History Summary Step 1. I have located the File Manager app and opened ... Step 2. I have clicked the 'Click Me' button once ...
Response Message
Reasoning <tool> Calculator </tool> <result>20250</result><think> The product of the numbers , which equals 20250. Now, I need to enter ... </think> <action>action: type, text: 20250 </action> <summary>I have got the product '20250' into... <summary>
B
Action
Figure 10: Interaction example for UI-Copilot-7B.
Let textpred and textgt denote the predicted and ground-truth input strings. The prediction is accurate if the texts match under relaxed string matching: racc = I[textpred ∼ textgt ]. • Swipe pred
pred
Let p1 , p2 denote the start and end points of the predicted swipe, and dir(·, ·) be the function that infers swipe direction. The prediction is accurate if the inferred swipe direction matches the ground truth: racc = pred pred I[dir(p1 , p2 ) = dirgt ]. • Click / Long Press Let B gt denote the enlarged ground-truth bounding box and ϵ be a fixed distance threshold. The prediction is accurate if the predicted point falls inside the bounding box or is sufficiently close to the ground-truth point: racc = I[ppred ∈ B gt ∨ ∥ppred − pgt ∥2 ≤ ϵ].
C
Theoretical Analysis
C.1
Preliminaries: Agentic RL
Agentic RL incorporates tool-call feedback during the reasoning process (Gou et al., 2024; Li et al., 2025c; Wu et al., 2025; Dong et al., 2025). The rollout sampling can be decomposed as: tR Y Pθ (R, a | I; T) = Pθ (Rt | R<t , I; T) · |t=1 ta Y
{z
Agentic Reasoning
}
(7)
Pθ (at | a<t , R, I; T)),
t=1
|
{z
Action Generation
}
where T denotes the set of available tools, R is the reasoning trajectory of length tR , interleaved with tool-call feedback, and a is the performed action with length ta . C.2
Multi-turn Action Prediction Learning
Multi-turn Training Alignment. Multi-turn (MT) training conditions on self-generated histories Htπ at each step, producing full trajectories π ) ∼ µ(· | I) and the gradient estimator (aπ1:T , T1:T [ ∇ θ J MT = EI∼pI
T hX
∇θ log µ(aπt | I, St , Htπ )
t=1
i
π R(I, aπ1:T , T1:T ) .
By aligning training histories with rollout histories, MT training better approximates the deployment objective: π π )∼µ [R(I, a1:T )] ≈ arg max J (θ) arg max E(aπ1:T ,T1:T θ
θ
reducing the train–inference mismatch. Conclusion. Single-turn training suffers from biased gradients due to off-policy histories Ht∗ , whereas multi-turn training uses self-generated histories Htπ , leading to more consistent and stable optimization toward deployment-time performance.
We prove in this section that why we use on-policy multi-turn rollout for action prediction learning.
C.3
Setup. Let π(· | θ) denote the training policy and µ(· | θ) the rollout (inference) policy. Given a high-level instruction I, the deployment objective is
We consider the policy gradient h ∇θ J (θ) = E A(I, T , a) ∇θ log Pθ (T | I) i + ∇θ log Pθ (a | T , I) .
h i J (θ) = EI∼pI E(a1:T ,T1:T )∼µ(·|I) R(I, a1:T , T1:T ) .
Single-turn Training Mismatch. In single-turn (ST) training, each step conditions on off-policy histories Ht∗ , yielding [ ∇ θ J ST = EI∼pI ,at ∼π(·|I,St ,Ht∗ ) h i ∇θ log π(at | I, St , Ht∗ ) R(I, a1:T , T1:T ) . Evaluation, however, uses self-generated histories Htπ : aπt ∼ µ(· | I, St , HtMT ), so that π [ π ∇ [R(I, aπ1:T , T1:T )]. θ J ST ̸= ∇θ E(aπ 1:T ,T1:T )∼µ
Decoupled Sampling in RL Training
where A(·) denotes the advantage function. RL , tool Tool-learning instructions. For I ∼ Dtool calls are supervised and consistent across trajectories. Conditioned on a fixed tool T , the action distribution Pθ (a | T , I) becomes highly concentrated, yielding a small action-level advantage: Ea∼Pθ (·|T ,I) |A(I, T , a)| ≈ 0.
As a result, the action-related gradient term contributes negligibly and can be ignored when estimating the policy gradient.
arg max Ea1:T ∼π [R(I, a1:T )] ̸=
Progress-learning instructions. For I ∼ RL , no tool is required and we explicitly fix Daction the tool set to T = None in the prompt. In this case, the policy reduces to action generation only, and the gradient simplifies to
arg max Eaπ1:T ∼µ [R(I, aπ1:T )]
∇θ Jprog (θ) = E[A(I, None, a) ∇θ log Pθ (a | I)]
Equivalently,
θ
θ
which illustrates the biased gradient and deployment gap.
where rewards and advantages are computed solely based on task progress.
Figure 11: Action Type Distribution Figure 12: Trajectory Length Distri- Figure 13: Tool Type Distribution of RL RL RL of Daction . bution of Daction . Dtool .
D
Data Description
D.1
Training Dataset
RL Daction Composition.
Figures 11 and 12 illustrate the distributions of action types and trajectory lengths across 2000 training trajectories in RL , respectively. Among action types, CLICK is Daction the most prevalent, followed by TERMINATE, which consistently serves as the final action in all successfully completed trajectories, and SWIPE. In terms of trajectory length, most trajectories comprise between 5 and 9 interaction steps. We provide some cases below, Open the Zoho Meet app, view the ,→ scheduled meetings. Go to the mobile category after closing ,→ the pop up and browse the products. Check out all of the suggested products ,→ and compare pricing because I'm ,→ looking for a budget friendly sofa. I want to view the recipe for Welsh ,→ Cakes in the kitchen stories app. Open the Yahoo Mail App, Select the ,→ Starva Mail and Unmark an Email as ,→ read. If I lose connection to the internet, I ,→ want to make the agents.txt file ,→ accessible offline in Google Drive ,→ so that I can readily access it. I want to share "Oscar and the wolf ,→ -somebody wants" music to my friend ,→ [email protected] via gmail. In the HealthifyMe app, view your today ,→ activity Add paintings to cart on the Rtistiq ,→ app. Set 10 minutes before notification for ,→ the birthday event in gmail calendar
Im going to Deutsches Museum from Ulm ,→ city, so get the traffic update on ,→ the route of the Deutsches Museum ,→ from my location. I want to search for some interesting ,→ activities in Hawaii. Open the TickTick app and mark microsoft ,→ training update classes as complete. I want to read the reviews of the Nike ,→ Fly.By Mid 3 shoe in the Nike app. Search for Kayak mail in gmail app. Use the gmail address ,→ [email protected] to send ,→ Karin information about the bus that ,→ leaves at 1:55 a.m. In the Simple Habit app, In order to ,→ improve my meditation, I would like ,→ to listen to the sound of ocean. RL Composition. The dataset D RL is conDtool tool structed using GPT-4o and includes 170 memoryintensive queries, 80 calculation-required queries, and 350 tasks requiring no tool usage, as illustrated in Figure 13. We also provide some cases below,
How much would it cost to buy two pairs ,→ of Nike Fly.By Mid 3 shoes, and ,→ which option is more budget-friendly ,→ compared to similar models? Among the suggested sofas, which one is ,→ the most budget-friendly based on ,→ their prices and features? What is the total price after adding the ,→ Rockrider City Cycle Btwin bicycle ,→ to my cart, including any applicable ,→ discounts? What is the total cost of all selected ,→ paintings added to the cart in the ,→ Rtistiq app?
Figure 14: Tool Type Distribution of Tool-call-Test.
Figure 15: Difficulty Level Distribution of AndroidWorld.
Figure 16: Difficulty Level Distribution of MemGUI-Bench.
What is the cheapest available flight ,→ from Knoxville to Hawaii, and how do ,→ prices vary across different dates? What is the estimated travel time and ,→ cost when traveling from Los Angeles ,→ to Oakland by train? What are the available bus options from ,→ Amsterdam Centraal to Rotterdam ,→ Centraal on October 26, and how long does each trip take? ,→ How long will it take to travel from Ulm ,→ to the Deutsches Museum based on ,→ current traffic conditions? How much earlier will I be notified if I ,→ set the calendar reminder to 10 ,→ minutes before the birthday event? Which file was made available offline in ,→ Google Drive when preparing for loss ,→ of internet connection? Which music track was shared via Gmail, ,→ and to which recipient was it sent? Which bus departure information was sent ,→ to [email protected]? What route and traffic conditions were ,→ shown for traveling from Ulm to the ,→ Deutsches Museum? Which activities were shown after ,→ searching for things to do in ,→ Hawaii? Which Microsoft training classes were ,→ marked as complete in the TickTick ,→ app? What account or session status was shown ,→ after signing out of the Babbel app?
We define the maximum response length as 12288 tokens and learning rate as 1 × 10−6 .
Training Details. We train Qwen2.5VL-7B with TIPO for 50 steps on 8 A100 GPUs. Each batch samples 16 prompts, with 8 rollouts per prompt.
Figure 17: Golden steps comparison between AndroidWorld and MemGUI-Bench.
D.2
Evaluation Dataset
Tool-call-Test. As shown in Figure 14, the Toolcall-Test subset consists of 1000 tasks generated RL in using GPT-4o and is carefully aligned with Dtool terms of task-type distribution. AndroidWorld vs MemGUI-Bench. As illustrated in Figures 15 and 16, more than half of the tasks in AndroidWorld are categorized as Easy. In contrast, MemGUI-Bench contains a substantially larger proportion of challenging tasks, with the distribution of Easy, Medium, and Hard tasks being nearly uniform (approximately 1:1:1). Furthermore, MemGUI-Bench exhibits a much longer average trajectory length, requiring 36.2 optimal steps on average, which significantly exceeds that of AndroidWorld (8.4 steps), as shown in Figure 17. Despite the increased task difficulty and longer interaction horizons, our UI-Copilot-7B achieves an accuracy of 20.3% on MemGUI-Bench and 39.1% on AndroidWorld, demonstrating robust and relatively balanced performance across benchmarks of varying complexity.
Models Closed-source Models GPT-4o (Hurst et al., 2024) Claude-CU (Anthropic, 2024) Open-source Models OS-Atlas-4B (Wu et al., 2024) OS-Atlas-7B (Wu et al., 2024) Qwen2.5VL-3B (Bai et al., 2025) Qwen2.5VL-7B (Bai et al., 2025) SeeClick (Cheng et al., 2024) UI-R1-3B (Lu et al., 2025b) UI-R1-7B (Lu et al., 2025b) GUI-R1-3B (Luo et al., 2025a) GUI-R1-7B (Luo et al., 2025a) OS-Genesis-7B (Sun et al., 2024) Aguvis-7B (Xu et al., 2024) NaviMaster-7B (Luo et al., 2025b) PAL-UI-3B (Liu et al., 2025d) PAL-UI-7B (Liu et al., 2025d) UI-AGILE-3B (Lian et al., 2025) UI-AGILE-7B (Lian et al., 2025) UI-S1-7B (Lu et al., 2025c) AgentCPM-GUI-8B (Zhang et al., 2025b) UI-TARS-7B (Qin et al., 2025) Ours 7B Models UI-Copilot-7B
ScreenSpot V2 Pro Avg
TM
AC-Low GR SR
18.3 83.0
0.8 17.1
9.6 50.1
74.3 74.3
0.0 0.0
19.4 19.4
66.3 63.7
0.0 0.0
20.8 12.5
34.3 60.9
0.0 0.0
3.3 3.1
71.9 84.1 80.9 89.0 55.1 85.4 90.0 85.0 88.2 – 81.8 – – – 88.6 92.1 90.1 – 91.6
3.7 18.9 28.7 28.7 1.1 17.8 33.5 28.6 31.3 – 22.9 – – – 37.9 44.0 30.6 – 35.7
37.8 51.5 54.8 58.9 28.1 51.6 61.8 56.8 59.8 – 52.4 – – – 63.3 68.1 60.4 – 63.7
91.9 93.6 62.0 83.4 93.0 79.2 86.6 83.7 85.2 90.7 93.8 – – – 85.4 87.7 92.2 94.4 95.2
83.8 88.0 74.1 87.0 73.4 82.4 83.7 81.6 85.4 – – 93.9 – – 87.6 88.1 89.3 – 89.3
80.6 85.2 59.3 62.5 75.0 66.4 69.7 64.4 66.5 74.2 89.4 69.5 – – 74.3 77.6 89.2 90.2 91.8
49.0 57.4 47.8 62.2 82.9 57.9 72.4 58.0 71.6 65.9 65.6 72.9 60.4 71.3 78.6 80.1 79.9 77.7 83.7
49.5 54.9 46.5 72.5 62.9 55.7 62.8 56.2 65.6 – – – 58.7 70.5 60.7 61.9 73.4 – 80.5
22.8 29.8 38.9 52.7 59.1 45.4 54.2 46.6 51.7 44.4 54.2 54.0 49.3 57.8 56.8 60.6 68.2 69.2 72.5
49.6 60.4 37.4 67.4 71.0 52.2 67.1 54.8 65.5 11.7 26.7 64.4 56.7 65.1 – – 76.3 90.8 94.6
34.6 39.7 26.5 56.3 52.4 34.5 41.3 41.5 43.6 – – – 36.9 46.8 – – 61.7 – 90.1
20.3 27.0 26.7 52.4 53.9 32.5 43.5 41.3 38.8 3.6 13.5 36.9 34.6 41.7 – 37.0 59.5 75.0 87.0
90.0
31.6
60.8
93.6
88.2
89.2
82.9
72.2
71.8
74.5
63.8
57.2
TM
AC-High GR SR
GUI Odyssey TM GR SR
Table 4: Model Comparison on Single-turn Benchmarks.
Single-turn Benchmarks Single-turn tasks evaluate the grounding capability and GUI Understanding capability of the end-to-end GUI model in conversations without historical context. We use ScreenSpot-V2 (Cheng et al., 2024) and ScreenSpot-Pro (Li et al., 2025a) to evaluate the grounding ability. We also adopt AndroidControlLow, AndroidControl-High (Li et al., 2024) and GUI Odyssey (Lu et al., 2024), for comprehensive GUI understanding evaluation. The action type match accuracy (TM), grounding accuracy rate (GR) and step success rate (SR) are reported. Single-turn Performance. Table 4 demonstrates that UI-Copilot-7B maintains competitive singleturn performance. Compared to the base model, UI-Copilot-7B achieves consistent improvements, with gains of +19.1% on AC-High SR and +4.8% on GUI Odyssey SR. Notably, although models trained with single-turn RL (e.g., AgentCPM-GUI8B) excel on single-turn tasks, they struggle with multi-turn execution (only 16.4% on AndroidWorld). This performance gap can be attributed to two primary factors: (1) a mismatch between the training and the evaluation dynamics, particularly regarding whether the the historical context is
on-policy or not (see Appendix C); and (2) overfitting to local reward signals, leading to ignorance of global training objectives. Pass@k Validation. We evaluate UI-Copilot-7B ’s pass@k accuracy (k=1,2,3,4) in Figure 18, under the setting without cross-session long-term memory. The results indicate that increasing k consistently improves performance, as larger k provides more opportunities to succeed in the presence of instability in dynamic online environments. Notably, UI-Copilot-7B exhibits stronger pass@k scaling behavior than Qwen2.5VL-7B on MemGUI-Bench, highlighting its superior potential for long-horizon tasks.