Reinforcement Learning for Computer-Use Agents with Autonomous Evaluation∗ Marta Sumyk1 , Oleksandr Kosovan1 1 Ukrainian Catholic University, Lviv, Ukraine [email protected], [email protected]
arXiv:2606.24515v1 [cs.AI] 23 Jun 2026
Abstract Computer-Use Agents (CUAs) execute high-level user goals by perceiving and acting directly within graphical user interfaces. However, reinforcement learning for CUAs remains difficult because openended desktop environments rarely provide scalable, machine-readable reward signals: task success is often visually grounded and hard to specify with handcrafted reward functions or dense manual labels. We propose an RL fine-tuning framework that uses autonomous vision-language evaluation as a scalable supervision signal for GUI agents. Given a final screenshot and the original instruction, a Vision-Language Model judges task completion and provides terminal feedback without taskspecific heuristics or manual labels during policy optimization. Because autonomous evaluators are imperfect, we model their feedback as a noisy binary reward channel and derive a noise-corrected reward estimator for Proximal Policy Optimization. Experiments across macOSWorld, Windows Agent Arena, and OSWorld show that corrected evaluator rewards outperform both zero-shot baselines and raw evaluator rewards, improving success rates by an average of 12.6 percentage points over zeroshot performance and 5.1 points over raw evaluator fine-tuning. These results suggest that autonomous evaluation can serve as a practical reward signal for RL in GUI environments when evaluator noise is explicitly modeled and corrected.
1
Introduction
Computer-Use Agents (CUAs) aim to autonomously operate graphical user interfaces (GUIs) from natural-language instructions using visual observations and executing actions [Liu et al., 2025; Sun et al., 2025]. Recent progress ∗
Accepted to the 4th International Workshop on Generalizing from Limited Resources in the Open World (GLOW @ IJCAI 2026): https://glow-ijcai-2026.github.io/glow-ijcai-2026/.
Figure 1: Bias correction framework for autonomous evaluation rewards.
shows promising generalization across applications and operating systems, positioning CUAs as a foundation for serviceagnostic desktop automation [Wang et al., 2025; Sager et al., 2025]. Despite these advances, current CUAs remain unreliable in unconstrained, real-world settings. Specifically, on the OSWorld benchmark [Xie et al., 2024], state-of-the-art desktop agents achieve success rates of only around 60%, highlighting a significant gap between current capabilities and practical deployment. This gap motivates the need for more effective learning mechanisms to improve robustness and task success in real-world desktop environments. Reinforcement learning (RL) provides a principled framework for improving agent behavior by enabling exploration, recovery from errors, and learning through interaction rather than reliance on static demonstrations [Sutton and Barto, 2018]. However, applying RL to GUI-based environments remains fundamentally constrained by the absence of reliable reward signals [Sumyk and Kosovan, 2025]. Unlike robotics simulators [Tang et al., 2024] or game environments [Koyamada et al., 2024], where success criteria are explicitly defined and machine-readable, desktop applications rarely expose clear indicators of task completion. Instead, task objectives are typically implicit, visually grounded, and highly unstructured [Xie et al., 2024]. As a result, prior work often relies on brittle, task-specific heuristics, such as DOM element checks in web-based environments [Qi et al., 2025], or on manually annotated suc-
cess labels [Xie et al., 2024]. These approaches fail to scale across different applications, operating systems, and task distributions, and they substantially limit generalization beyond narrowly defined settings. An alternative strategy is to define task success internally within the computer-use agent itself [Wang et al., 2025]. However, this introduces additional reliability concerns, as the agent’s own perception and reasoning are inherently imperfect and are precisely the components that RL aims to improve. Consequently, the lack of a scalable and reliable reward mechanism remains a central bottleneck for effective RL fine-tuning of computer-use agents. A promising alternative is to employ autonomous evaluators that assess whether a task has been successfully completed. Recent works [Sumyk and Kosovan, 2025; Lin et al., 2025; Sumyk and Kosovan, 2026; Rosset et al., 2026] demonstrate that evaluators based on Vision-Language Models (VLMs) can achieve high accuracy and generalize across diverse applications. However, these evaluators inevitably introduce noise: they may incorrectly label unsuccessful executions as completed (false positives) or fail to detect success (false negatives). Naively using evaluator outputs as reward signals can therefore introduce bias and lead to unstable reinforcement learning [Wang et al., 2020a; Li et al., 2024a]. In this paper, we address this challenge by introducing a principled framework for integrating autonomous evaluators into reinforcement learning for CUAs. Rather than treating evaluator outputs as ground truth, we explicitly model evaluator noise and derive a statistically grounded, asymptotically unbiased reward estimator. This formulation enables effective RL fine-tuning under imperfect feedback, allowing CUAs to learn from large-scale interaction data without manual labeling. This work investigates the following research questions: • RQ1: Can autonomous VLM-based evaluation serve as a scalable reward signal for RL fine-tuning of ComputerUse Agents? • RQ2: Does modeling and correcting evaluator noise improve policy learning compared to using raw evaluator rewards? To address these questions, we propose a principled framework that integrates autonomous evaluators into the reinforcement-learning loop by explicitly modeling their error characteristics. Our approach derives a statistically grounded, asymptotically unbiased reward estimator that corrects evaluator-induced noise and enables stable policy optimization. Empirically, we demonstrate that this framework yields improvements in robustness and task success across three operating systems, Windows, macOS, and Linux, without relying on manual annotations or task-specific heuristics. The implementation is available at https://github.com/ martasumyk/rl with autonomous feedback.
2
Related Work
2.1
Computer-Use Agents
Computer-Use Agents (CUAs) are end-to-end autonomous systems that complete natural-language tasks by perceiving
the rendered desktop GUI, typically via screenshots, and executing actions such as clicking, typing, scrolling, and dragging [Sager et al., 2025]. Recent CUA architectures combine vision-language reasoning with explicit action grounding to plan and carry out long-horizon workflows across diverse applications and operating systems [Liu et al., 2025; Sun et al., 2025; Wang et al., 2025; Qin et al., 2025]. Unlike API-based or function-calling agents that require explicit service integrations, CUAs adopt a service-agnostic interaction model: they perceive and manipulate software exclusively through its rendered interface. This design enables interaction with arbitrary applications without bespoke engineering and allows a single agent to generalize across different software applications and operating systems [Sun et al., 2025; Sager et al., 2025]. Consequently, CUAs are increasingly regarded as a promising foundation for general-purpose computer automation. However, this generality introduces significant challenges for reasoning and verification. Because CUAs rely solely on visual feedback, they are susceptible to silent or partial failures caused by unexpected interface states, asynchronous rendering, visual occlusions, or subtle distribution shifts in UI layouts [Gur et al., 2023; Humphreys et al., 2024; Li et al., 2024b]. Furthermore, many real-world tasks lack explicit, machine-readable success criteria, making it difficult to reliably determine whether an agent has truly completed the intended objective [Sumyk and Kosovan, 2025]. Moreover, state-of-the-art desktop agents achieve success rates of only around 60% on OSWorld [Xie et al., 2024] and approximately 40% on macOSWorld [Yang et al., 2025]. This performance gap highlights the need for more robust training and fine-tuning methods for CUAs.
2.2
Training and Fine-Tuning of Computer-Use Agents
Training CUAs typically combines supervised learning with reinforcement learning [Lai et al., 2025; Wang et al., 2025]. Many systems first use behavioral cloning to map visual observations and instructions to low-level GUI actions from human demonstrations or scripted trajectories [Gur et al., 2023; Li et al., 2024b; Humphreys et al., 2024]. This approach is effective for short-horizon action prediction and instruction following, but it often struggles with compounding errors and long-horizon tasks that require recovery, exploration, and verification [Humphreys et al., 2024]. To address these limitations, recent work explores RLbased fine-tuning to improve robustness and task success beyond demonstrations [Liu et al., 2025; Sun et al., 2025; Qin et al., 2025]. In practice, RL pipelines either operate in environments where rewards can be programmatically defined, such as synthetic web interfaces [Gur et al., 2023; Humphreys et al., 2024], or rely on task- and platformspecific heuristics, such as DOM parsing or string matching, that do not transfer to general desktop settings [Xie et al., 2024]. More broadly, existing approaches remain limited by the dependence on a reliable task-completion signal, which is rarely available in real-world desktop GUIs and motivates research on autonomous evaluation.
2.3
Autonomous Evaluation
Autonomous evaluation seeks to determine whether an agent has successfully completed a user’s instruction based solely on the observed GUI state, producing feedback suitable for both benchmarking and learning [Pan et al., 2024]. In realworld desktop environments, task success is rarely accompanied by explicit, machine-readable signals, as objectives are often implicit and visually grounded [Sumyk and Kosovan, 2025]. Consequently, many existing benchmarks and training pipelines rely on human verification of final states, which is costly, time-consuming, and difficult to scale across diverse applications and operating systems [Xie et al., 2024]. Recent work proposes autonomous evaluators, typically based on Vision-Language Models (VLMs), that assess task completion by jointly analyzing the final GUI state and the natural-language instruction, and output a binary success judgment [Sumyk and Kosovan, 2025; Lin et al., 2025]. These evaluators enable scalable, automated success labeling and can serve as reward or feedback providers for agent improvement. However, their predictions are imperfect and can include false positives and false negatives. Therefore, naively treating evaluator outputs as ground-truth rewards can bias learning and destabilize policy optimization [Wang et al., 2020a]. This motivates methods that explicitly model evaluator error and incorporate noise-aware reward correction, which is the focus of our approach.
2.4
Reinforcement Learning with Noisy Feedback
A broad line of research studies reinforcement learning when the agent does not observe the true reward, but instead receives a noisy proxy produced by measurement artifacts, imperfect annotators, or automated verification systems [Wang et al., 2020a; Cai et al., 2025a; Wang et al., 2020b]. Wang et al. [2020a] formalize reward corruption using a confusion matrix over discrete reward values. They show that, given an estimate of the corruption process, one can construct an unbiased reward estimator. This allows standard RL algorithms to recover optimal policies despite observing only corrupted feedback. Complementary work investigates learning from noisy evaluative signals provided by humans or teachers. For example, CANDERE-COACH [Li et al., 2024a] considers unreliable binary approve/disapprove feedback and proposes online denoising mechanisms that filter feedback before policy updates, demonstrating robustness under substantial noise. Most closely related to our setting, Cai et al. [2025b] study reinforcement learning with verifiable rewards, where policies are trained from noisy binary signals generated by automated verifiers. They model asymmetric false-positive and false-negative errors and derive correction strategies that debias policy-gradient updates by appropriately transforming the observed feedback. Our method instantiates these ideas for computer-use agents by treating a vision-based task-completion judge as a noisy binary reward channel. Following prior work [Wang et al., 2020a; Cai et al., 2025b], we derive a simple correction that yields an asymptotically unbiased reward estimator under a mild separability condition and integrate it directly into PPO. Unlike approaches that denoise feedback at the data
level [Li et al., 2024a], our correction operates at the reward level, making it straightforward to plug into standard policygradient fine-tuning.
3
Methodology
We formulate a CUA as a Markov Decision Process M = (S, A, P, r, γ) [van Otterlo and Wiering, 2012]. We consider a collection of m tasks indexed by i ∈ {1, . . . , m}, where each task defines an episode with a fixed natural-language instruction di and horizon T . At timestep t of task i, the agent observes the current GUI and executes a single atomic interaction. The state is defined as si,t = (xi,t , di ), where xi,t denotes the rendered screen image and di the task description, fixed throughout the episode. The action space A consists of low-level GUI operations: click, type, scroll, and drag. A trajectory for task i is: τi = (si,0 , ai,0 , . . . , si,T ).
3.1
(1)
Dataset
We construct a synthetic dataset of computer-use tasks spanning 42 applications for each OS, including built-in macOS applications and functionally similar applications on Linux and Windows. For each application, we define 60 naturallanguage task descriptions, resulting in a total of 7,560 tasks that cover a broad range of GUI interaction patterns across operating systems. For each application, tasks are randomly partitioned into three disjoint splits of 20 tasks each. Two splits, a total of 40 tasks per application and two-thirds of the dataset, are used for reinforcement-learning fine-tuning, where agent rollouts are collected and PPO updates are performed using evaluatorbased rewards. The remaining split, one-third of the dataset, is reserved for evaluator calibration and is used to estimate the evaluator’s true-positive, false-positive, true-negative, and false-negative rates. This separation ensures that evaluator calibration and policy optimization are conducted on disjoint task sets, preventing information leakage. In addition to our dataset, we incorporate task descriptions from three existing GUI interaction datasets, OmniAct [Kapoor et al., 2024], GUI-World [Chen et al., 2025], and GUIDE [Chawla et al., 2024], to further diversify the task distribution used for RL fine-tuning. From these datasets, we use only the natural-language task specifications as input to the agent. A summary of all datasets and their task characteristics is provided in Table 1. In addition to task definitions, the dataset includes execution logs of a base computer-use agent, UI-TARS [Qin et al., 2025], and of agents fine-tuned with reinforcement learning. Each log records a complete agent trajectory, including screenshots, executed actions, and intermediate reasoning steps. For evaluation purposes, task outcomes in the evaluator calibration and RL test splits are annotated with ground-truth success labels, which are used exclusively for evaluator assessment and final benchmarking and are never exposed to the agent during training.
Dataset OmniAct [Kapoor et al., 2024] GUI-World [Chen et al., 2025] GUIDE [Chawla et al., 2024] Ours
Platforms
Task Types
macOS / Windows / Linux Web Web macOS / Windows / Linux
Browser, file managers, shell, system utilities Search, email, documents, SaaS dashboards Enterprise dashboards, productivity tools OS utilities, file I/O, app navigation, settings
Table 1: Summary of datasets whose task descriptions are used as input for RL fine-tuning. All datasets provide natural-language task specifications for GUI interaction; our dataset additionally includes native desktop tasks explicitly designed for autonomous evaluation and reward correction.
3.2
Autonomous Evaluation
For autonomous evaluation, we use the Qwen2-VL-7B model [Wang et al., 2024], motivated by its strong multimodal reasoning and empirically demonstrated superiority as a vision-based evaluator among open-source models [Sumyk and Kosovan, 2025; Lin et al., 2025]. Given a task description di and the final GUI state xi,T , the evaluator outputs a binary signal: r̃i ∈ {0, 1}, (2) where r̃i = 1 indicates that the task is judged completed. The evaluator operates in a zero-shot setting and is fully decoupled from the acting agent: it observes neither the agent’s action history nor its internal reasoning, basing its judgment solely on the final screenshot and instruction.
3.3
Reinforcement Learning Fine-Tuning
We now describe how we fine-tune a pre-trained CUA using reinforcement learning with world feedback from the autonomous evaluator. Figure 2 provides an overview of the full pipeline. Terminal Reward and Noise Model. In our setting, reward is terminal-only: the true task-completion signal ri⋆ ∈ {0, 1} is defined only at the end of the episode and is unobserved. Instead, the evaluator provides a noisy binary judgment r̃i based on the final GUI state xi,T . We characterize evaluator noise using conditional error rates: e+ := Pr(r̃i = 0 | ri⋆ = 1)
(false negative),
(3)
e− := Pr(r̃i = 1 | ri⋆ = 0)
(false positive).
(4)
D := 1 − e+ − e− ,
(5)
and assume D > 0, ensuring that the evaluator provides informative feedback beyond random guessing. Since r̃i is binary, its
E[r̃i | ri⋆ ] = e− + D · ri⋆ .
(6)
Solving for ri⋆ yields the corrected reward estimator: r̃i − e− , D
(7)
E[r̂i | τi ] = ri⋆ ,
(8)
r̂i := which satisfies:
r̂i♯ :=
r̃i − ê−
, (9) D̂ which is asymptotically unbiased as calibration data grows and D̂ remains bounded away from zero. Policy Optimization. Since reward is terminal-only, the discounted return for task i is: R̂i := γ T r̂i ,
(10)
h i J(θ) := Ei∼D, τi ∼πθ R̂i .
(11)
and the RL objective is:
We optimize J(θ) using Proximal Policy Optimization (PPO) [Schulman et al., 2017]. The actor parameterizes a stochastic policy πθ (a | s) over GUI actions; the critic estimates the value function Vϕ (s). Advantage estimates are computed by propagating the terminal corrected reward r̂i backward through the trajectory, enabling stable policygradient updates despite evaluator-induced noise. By explicitly correcting evaluator errors, our approach provides a reliable world feedback signal for RL without requiring groundtruth labels or task-specific heuristics.
4
We define the separability constant:
Noise-Corrected Reward Estimator. conditional expectation satisfies:
and is therefore an asymptotically unbiased estimator of true task completion. In practice, error rates e+ and e− are unknown and must be estimated from the held-out calibration split. Let ê+ and ê− denote empirical estimates and D̂ := 1 − ê+ − ê− . The resulting plug-in estimator is:
Results and Evaluation
We evaluate our approach along three dimensions: (i) the accuracy and error profile of the autonomous evaluator used to generate reward signals; (ii) the effect of evaluator-based rewards on PPO fine-tuning, comparing raw evaluator feedback with the proposed noise-corrected reward estimator; and (iii) an ablation comparing per-OS fine-tuning against a single unified model trained across operating systems.
4.1
Autonomous Evaluation
We first quantify the reliability of the vision-based evaluator, since its error rates directly determine the reward correction in Section 3. Table 2 reports normalized confusion matrices on the held-out evaluator split for macOS, Windows, and Linux. While the evaluator achieves high accuracy overall, its behavior varies by OS.
Figure 2: Overview of our RL fine-tuning pipeline with autonomous, noise-corrected rewards. For each task instance, the computer-use agent interacts with the desktop environment, observing the GUI state st and executing actions at . A vision-language evaluator predicts a noisy binary completion signal r̃ from the final observed state, a screenshot and instruction, which we transform into a corrected reward r̂ = f (r̃) using estimated false-positive and false-negative rates. PPO then updates the agent parameters θ using r̂, closing the interaction–evaluation– learning loop.
These asymmetries are important for reinforcement learning: false positives are particularly harmful because they can assign high reward to unsuccessful trajectories, reinforcing incorrect behaviors; false negatives, in contrast, primarily reduce the effective reward frequency and can slow learning. We therefore estimated (e+ , e− ) per OS from the evaluator split and used these values to compute the corrected reward estimator r̂t during PPO fine-tuning.
4.2
Reinforcement Learning Fine-Tuning
We evaluate reinforcement learning fine-tuning across three desktop environments: macOS, Windows, and Linux, using macOSWorld [Yang et al., 2025], Windows Agent Arena [Bonatti et al., 2024], and OSWorld [Xie et al., 2024], respectively. Results are summarized in Table 3. We compare five training configurations: (1) a zero-shot baseline, (2) PPO fine-tuning with raw binary evaluator rewards r̃t using a unified cross-OS model, (3) PPO fine-tuning with the corrected reward estimator r̂t using a unified model, (4) PPO fine-tuning with raw evaluator rewards using per-OS models, and (5) PPO fine-tuning with the corrected reward estimator using per-OS models. Across all benchmarks, reinforcement learning with evaluator feedback improves performance over the zero-shot baseline. However, the choice of reward signal and model parameterization substantially affects both final performance and generalization. Using raw evaluator rewards yields moderate gains in most settings, but these gains are consistently smaller than those obtained with the corrected reward estimator. The proposed noise-corrected reward r̂t achieves the strongest performance across all three benchmarks when combined with per-OS fine-tuning, improving success rates from 0.084 → 0.203 on macOSWorld, 0.331 → 0.442 on Windows Agent Arena, and 0.283 → 0.432 on OSWorld.
These improvements demonstrate that explicitly accounting for evaluator false positives and false negatives yields a substantially more reliable learning signal for PPO. In contrast, fine-tuning with raw evaluator rewards exhibits inconsistent behavior. While raw rewards can improve indomain performance, they underperform the corrected estimator in all settings and, in some cases, reduce generalization. This effect is most evident on OSWorld, where the corrected reward consistently outperforms raw rewards under both unified and per-OS training. These results indicate that ignoring evaluator error can introduce bias that negatively impacts reinforcement learning, particularly under distribution shift.
5
Discussion and Limitations
Firstly, our approach assumes that the evaluator’s falsenegative and false-positive rates (e+ , e− ) are approximately fixed for a given model and evaluation protocol and can be reliably estimated on a held-out calibration split. In practice, evaluator behavior may be non-stationary: error rates can vary with task type, UI complexity, language phrasing, visual themes, and operating-system-specific interface conventions. Moreover, as the agent improves during training, the distribution of visited GUI states may shift, potentially changing the evaluator’s error profile and introducing residual bias when using plug-in estimates (ê+ , ê− ). A natural extension is to estimate error rates conditionally, for example, per task family or application type. Another limitation is that our formulation relies on a binary success signal, which is inherently sparse and often effectively terminal. While PPO can learn under sparse rewards, exploration remains challenging for long-horizon GUI tasks with delayed success. The proposed correction improves reward accuracy but does not increase reward density,
Predicted Windows
True
macOS Pos Neg
Linux
Pos
Neg
Pos
Neg
Pos
Neg
0.5024 0.0488
0.1060 0.3429
0.6929 0.0583
0.1226 0.2488
0.2488 0.1226
0.2690 0.3595
Table 2: Normalized confusion matrices of the autonomous evaluator across operating systems. Green cells indicate correct predictions, red cells indicate misclassifications.
Training Setup
macOSWorld
WindowsAgentArena
OSWorld
0.084 0.129 0.144 0.149 0.203
0.331 0.383 0.403 0.357 0.442
0.283 0.385 0.399 0.418 0.432
Zero-shot baseline RL + raw evaluator reward (r̃t ), unified model RL + corrected reward estimator (r̂t ), unified model RL + raw evaluator reward (r̃t ), per-OS models RL + corrected reward estimator (r̂t ), per-OS models
Table 3: Task success rates across desktop benchmarks. We compare the zero-shot baseline with PPO fine-tuning using (i) raw binary evaluator rewards and (ii) the proposed noise-corrected reward estimator, under both per-OS and unified cross-OS training. Best results are shown in bold, second-best are underlined.
and therefore cannot by itself resolve exploration difficulties. Incorporating shaping signals, such as intermediate evaluator judgments, progress estimators, or subgoal completion signals, may substantially improve sample efficiency, but would require additional calibration and may introduce new sources of bias. The effectiveness of the correction also depends on the quality of the calibration set used to estimate (e+ , e− ). With limited calibration data, statistical uncertainty in these estimates can propagate into the corrected reward and increase the variance of policy-gradient updates, particularly when the separability term D = 1−e+ −e− is small and the correction amplifies noise. In such regimes, conservative strategies such as clipping the corrected reward, shrinking estimates toward priors, or abstaining when evaluator confidence is low may be necessary to maintain training stability. −) In addition, the corrected reward r̂t = (r̃t −ê is realD̂ valued and can fall outside the [0, 1] range, which alters the scale of advantages and may affect PPO optimization in finite-sample settings. Although policy-gradient methods are invariant to affine reward transformations in expectation, practical stability can still depend on careful normalization and hyperparameter choices. Finally, vision-based evaluators have inherent blind spots: they may miss semantically correct outcomes that are visually subtle or overestimate success based on superficial visual cues. Binary success labels also ignore partial progress and alternative valid solutions, which are common in realworld GUI tasks. Extending evaluators to produce calibrated confidence scores, multi-level outcomes, or preference-based feedback could provide richer learning signals, but would require revisiting both the noise model and the correction mechanism. While our results demonstrate improved generalization to OSWorld, robustness under broader distribution shifts, such as new application versions, different screen resolutions, or accessibility settings, remains an open direction for future
work.
6
Conclusion
We presented a principled approach for reinforcementlearning fine-tuning of computer-use agents using reward signals produced by an autonomous vision-based evaluator. Our key idea is to treat the evaluator as a binary reward channel and to correct its false-positive and false-negative errors via a simple reward transformation that is asymptotically unbiased under a mild separability condition. This yields a drop-in replacement for raw evaluator rewards that can be used with standard policy-gradient methods such as PPO. Empirically, PPO fine-tuning with evaluator feedback improves over the zero-shot UI-TARS baseline across operating systems. While fine-tuning with the raw evaluator signal can help on in-domain tasks, we find that explicitly correcting evaluator noise produces more consistent gains, especially on macOS and Windows, and improves transfer to the OSWorld benchmark, where naive use of raw rewards slightly degrades performance. Finally, our ablation comparing perOS and unified fine-tuning suggests that OS-specific policies provide more reliable generalization overall, supporting the use of per-OS models in our main experiments. Overall, these results indicate that autonomous evaluation can serve as a scalable supervision signal for RL in GUI environments, provided that evaluator noise is explicitly modeled and corrected. We view this as a step toward practical RL fine-tuning pipelines for general desktop agents that learn from interaction without requiring manual success labels.
References [Bonatti et al., 2024] Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale, 2024.
[Cai et al., 2025a] Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers, 2025. [Cai et al., 2025b] Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers. arXiv preprint arXiv:2510.00915, 2025. [Chawla et al., 2024] Rajat Chawla, Adarsh Jha, Muskaan Kumar, Mukunda NS, and Ishaan Bhola. Guide: Graphical user interface data for execution, 2024. [Chen et al., 2025] Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, and Lichao Sun. Gui-world: A video benchmark and dataset for multimodal gui-oriented understanding, 2025. [Gur et al., 2023] Izzeddin Gur, Arjun Pal, Tianyu Li, Marc Brockschmidt, Swarat Chaudhuri, Mark Riedl, and Jacob Andreas. Browsergym: A benchmark for browser agents. arXiv preprint arXiv:2307.04492, 2023. [Humphreys et al., 2024] Peter Humphreys, Ansong Ni, Han Pan, Izzeddin Gur, Victor Zhong, and Jacob Andreas. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2024. [Kapoor et al., 2024] Raghav Kapoor, Yash Parag Butala, Melisa Russak, Jing Yu Koh, Kiran Kamble, Waseem Alshikh, and Ruslan Salakhutdinov. Omniact: A dataset and benchmark for enabling multimodal generalist autonomous agents for desktop and web, 2024. [Koyamada et al., 2024] Sotetsu Koyamada, Shinri Okano, Soichiro Nishimori, Yu Murata, Keigo Habara, Haruka Kita, and Shin Ishii. Pgx: Hardware-accelerated parallel game simulators for reinforcement learning, 2024. [Lai et al., 2025] Hanyu Lai, Xiao Liu, Yanxiao Zhao, Han Xu, Hanchen Zhang, Bohao Jing, Yanyu Ren, Shuntian Yao, Yuxiao Dong, and Jie Tang. Computerrl: Scaling end-to-end online reinforcement learning for computer use agents, 2025. [Li et al., 2024a] Yuxuan Li, Srijita Das, and Matthew E. Taylor. Candere-coach: Reinforcement learning from noisy feedback. arXiv preprint arXiv:2409.15521, 2024. [Li et al., 2024b] Zhiyuan Li, Yuxuan Zhao, Qingyang Chen, Yujia Zhao, Hao Zhang, Liang Yuan, Bill Yuchen Lin, Yizhou Wang, and Wenhao Zhang. Seeact: A multimodal agent for web interaction via visual grounding and action generation. arXiv preprint arXiv:2404.05719, 2024. [Lin et al., 2025] Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, Ke Li, and Xing Sun. Cuarewardbench: A benchmark for evaluating reward models on computerusing agent, 2025.
[Liu et al., 2025] Yuhang Liu, Pengxiang Li, Zishu Wei, Congkai Xie, Xueyu Hu, Xinchen Xu, Shengyu Zhang, Xiaotian Han, Hongxia Yang, and Fei Wu. Infiguiagent: A multimodal generalist gui agent with native reasoning and reflection, 2025. [Pan et al., 2024] Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents, 2024. [Qi et al., 2025] Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025. [Qin et al., 2025] Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents, 2025. [Rosset et al., 2026] Corby Rosset, Pratyusha Sharma, Andrew Zhao, Miguel Gonzalez-Fernandez, and Ahmed Awadallah. The art of building verifiers for computer use agents, 2026. [Sager et al., 2025] Pascal J. Sager, Benjamin Meyer, Peng Yan, Rebekka von Wartburg-Kottler, Layan Etaiwi, Aref Enayati, Gabriel Nobel, Ahmed Abdulkadir, Benjamin F. Grewe, and Thilo Stadelmann. A comprehensive survey of agents for computer use: Foundations, challenges, and future directions, 2025. [Schulman et al., 2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. [Sumyk and Kosovan, 2025] Marta Sumyk and Oleksandr Kosovan. “are we done yet?”: A vision-based judge for autonomous task completion of computer use agents, 2025. [Sumyk and Kosovan, 2026] Marta Sumyk and Oleksandr Kosovan. Cuaaudit: Meta-evaluation of vision-language models as auditors of autonomous computer-use agents, 2026. [Sun et al., 2025] Zeyi Sun, Ziyu Liu, Yuhang Zang, Yuhang Cao, Xiaoyi Dong, Tong Wu, Dahua Lin, and Jiaqi Wang. Seagent: Self-evolving computer use agent with autonomous learning from experience, 2025. [Sutton and Barto, 2018] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, Cambridge, MA, 2 edition, 2018. [Tang et al., 2024] Chen Tang, Ben Abbatematteo, Jiaheng Hu, Rohan Chandra, Roberto Martı́n-Martı́n, and Peter Stone. Deep reinforcement learning for robotics: A survey of real-world successes, 2024.
[van Otterlo and Wiering, 2012] Martijn van Otterlo and Marco Wiering. Reinforcement learning and markov decision processes. Reinforcement Learning: State of the Art, pages 3–42, 2012. [Wang et al., 2020a] Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. arXiv preprint arXiv:1810.01032, 2020. [Wang et al., 2020b] Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards, 2020. [Wang et al., 2024] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution, 2024. [Wang et al., 2025] Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, Wanjun Zhong, Yining Ye, Yujia Qin, Yuwen Xiong, Yuxin Song, Zhiyong Wu, Aoyan Li, Bo Li, Chen Dun, Chong Liu, Daoguang Zan, Fuxing Leng, Hanbin Wang, Hao Yu, Haobin Chen, Hongyi Guo, Jing Su, Jingjia Huang, Kai Shen, Kaiyu Shi, Lin Yan, Peiyao Zhao, Pengfei Liu, Qinghao Ye, Renjie Zheng, Shulin Xin, Wayne Xin Zhao, Wen Heng, Wenhao Huang, Wenqian Wang, Xiaobo Qin, Yi Lin, Youbin Wu, Zehui Chen, Zihao Wang, Baoquan Zhong, Xinchun Zhang, Xujing Li, Yuanfan Li, Zhongkai Zhao, Chengquan Jiang, Faming Wu, Haotian Zhou, Jinlin Pang, Li Han, Qi Liu, Qianli Ma, Siyao Liu, Songhua Cai, Wenqi Fu, Xin Liu, Yaohui Wang, Zhi Zhang, Bo Zhou, Guoliang Li, Jiajun Shi, Jiale Yang, Jie Tang, Li Li, Qihua Han, Taoran Lu, Woyu Lin, Xiaokang Tong, Xinyao Li, Yichi Zhang, Yu Miao, Zhengxuan Jiang, Zili Li, Ziyuan Zhao, Chenxin Li, Dehua Ma, Feng Lin, Ge Zhang, Haihua Yang, Hangyu Guo, Hongda Zhu, Jiaheng Liu, Junda Du, Kai Cai, Kuanye Li, Lichen Yuan, Meilan Han, Minchao Wang, Shuyue Guo, Tianhao Cheng, Xiaobo Ma, Xiaojun Xiao, Xiaolong Huang, Xinjie Chen, Yidi Du, Yilin Chen, Yiwen Wang, Zhaojian Li, Zhenzhu Yang, Zhiyuan Zeng, Chaolin Jin, Chen Li, Hao Chen, Haoli Chen, Jian Chen, Qinghao Zhao, and Guang Shi. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning, 2025. [Xie et al., 2024] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. [Yang et al., 2025] Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents, 2025.