How Benchmarks Mis-Score Computer-Use Agents Zihan Dong1 , Zhiyuan Ma2 , Zekun Wang1 , Yunqing Li3 , Zirou Liu4 , Ruixuan Deng1 , Qishi Zhan5 , Rui Qian6 1
Georgia Institute of Technology North Carolina State University 3 Lenovo AI Technology Center 4 University of North Carolina at Chapel Hill 5 Marquette University 6 Fudan University 2
arXiv:2607.28367v1 [cs.AI] 30 Jul 2026
Abstract Computer-use agents (CUA) are being deployed to browse the web and operate desktop software, yet their benchmark scores are still commonly produced by brittle scripted oracles. A score is the output of a pipeline in which tasks can be stale, trajectories can omit decisive visual evidence, evaluators can reject valid alternatives, and aggregate reports can hide the cause of failure. We organize these problems into a reliability framework spanning task construction, trajectory observation, scoring, and reporting. We then audit 150 public failure-scored trajectories from five web, enterprise-workflow, and desktop-control benchmarks, find that 15.3% of FAIL verdicts are wrong: 10.7% are evaluator false negatives and 4.7% are broken tasks. For genuine failures, a three-tier diagnostic taxonomy shows that verification/feedback and planning failures dominate execution/grounding errors, while a single scalar success rate can not explain. We connect these findings to newer long-horizon CUA benchmarks and derive stage-specific design rules for CUA evaluation.
1
Introduction
Computer-use agents (CUA) are already being placed between users and consequential web and desktop interfaces: they search, fill forms, manipulate files, and complete multi-application workflows (Field, 2025; Stein, 2025; Alibaba Group, 2026; Xu, 2025). Their evaluation, however, still depends heavily on scripted oracles that inspect a URL, a DOM field, or a final machine state. The deployment–measurement gap becomes sharper as tasks grow from the roughly 30-action episodes of OSWorld to hour-scale, cross-application workflows: OSWorld 2.0 reports a median human completion time of about 1.6 hours, while Odysseys and
WeaveBench explicitly target long-horizon, multisite or hybrid-interface work (Xie et al., 2024; Yuan et al., 2026; Jang et al., 2026; Li et al., 2026). These settings cannot be measured faithfully by treating execution as self-validating. A useful solution may follow an unforeseen path; a final state may look correct while violating an intermediate constraint; and an agent may fail because the live site, virtual machine, or checker is wrong rather than because its CUA capability is weak. Newer benchmarks have therefore begun to add graded rubrics, trajectory-aware judges, and separate partial-completion signals (Jang et al., 2026; Li et al., 2026; Lin et al., 2026). These developments make evaluator reliability more current, not less: each additional judge is another measurement instrument that must be validated. The central problem is that an agent score is the output of a pipeline, not a direct observation of capability. Stale or leaked tasks, broken environments, incomplete trajectories, scoring scripts that reject valid alternatives, and aggregate success rates that obscure failure causes all distort evaluation. These are not interchangeable “agent failures.” They arise at distinct stages of task construction, observation, scoring, and reporting. Collapsing them into a single category produces leaderboards that mis-rank systems and directs engineering effort toward the wrong component (El hattami et al., 2025; Lù et al., 2025; Han et al., 2025). We therefore study the reliability of the CUA evaluation pipeline. Figure 1 provides the paper’s organizing schema. We first locate three threats—contamination and task invalidity, evaluator fragility, and diagnostic opacity—at the stages where they enter (§2). We then examine their current manifestations across web, enterpriseworkflow, and desktop evaluation (§3), and test
A
LIVE CUA EXECUTION
revise and retry
User goal
Plan
constraints
strategy
Act
World state
Observe
Verify
tool or GUI
environment
result + feedback
revise or stop
BENCHMARK CAPTURES A LOSSY TRACE B
FOUR-STAGE MEASUREMENT PIPELINE 01
C
TASK CONSTRUCTION
02
EXECUTION ENVIRONMENT
03
SCORING
04
REPORTING
Domain-valid + specified
Reproducible + observable
Recognizes valid outcomes
Supports diagnosis
goal · constraints · references
runtime · tools · state · trace
scripts · rubrics · judges
verdict · cause · recovery · cost
WHERE RELIABILITY BREAKS STAGES 1 + 2
STAGE 3
STAGES 2 + 4
TASK / ENVIRONMENT VALIDITY & CONTAMINATION
EVALUATOR FRAGILITY
OBSERVABILITY & DIAGNOSTIC OPACITY
drift · leakage · broken setup
valid result rejected · shortcut rewarded
missing evidence · hidden failure cause
Figure 1: A CUA benchmark score is a pipeline output, not a direct observation of capability. The benchmark projects the execution loop into four measurement stages. Task validity and contamination enter during construction, evaluator fragility enters during scoring, and observability and diagnostic opacity span trajectory observation and reporting.
them directly by re-auditing 150 public CUA trajectories (§4). We expect readers can trace a CUA verdict through the full evaluation pipeline, distinguish evaluator false negatives and broken tasks from genuine agent failures, and diagnose those failures with a three-tier codebook adapted from MAST (Cemri et al., 2025). Our audit of 150 failure-scored trajectories across five benchmarks shows why these distinctions matter: 15.3% of FAIL verdicts are wrong (10.7% evaluator false negatives and 4.7% broken tasks), while genuine failures are dominated by feedback-blindness and planning errors. We translate these findings into stage-specific guidance for task construction, environments, scoring, and reporting. More broadly, we hope this reliability perspective inspires further CUA innovation, evaluation reform, and evidence-based governance and regulation. Our scope focused on the evaluation of CUA across current web, desktop/OS, enterprise-workflow, and mobile tasks. Prior surveys organize evaluation by capability, application, or methodology (Yehudai et al., 2025; Mohammadi et al., 2025); they ask what evaluation resources exist, whereas this paper asks when a CUA verdict should be distrusted and supports that diagnosis with trajectory-level evidence.
2
Evaluation Reliability Framework
We framed CUA evaluation as a four-stage measurement pipeline: task construction, trajectory observation, scoring, and reporting. Reliability depends on solvable tasks that are independent of
system development, evidence-complete trajectories, evaluators that recognize acceptable outcomes, and reports that support diagnosis. Figure 1 locates three recurring threats across these stages. 2.1
Task Validity and Contamination
A CUA benchmark produces credible rankings when its tasks, environments, and hidden references remain valid and independent of system development. Live sites drift, credentials expire, VM images lack assumed hardware, and professional software changes its menus or file formats. SearchTime Contamination (STC) could happen when public instructions and reference artifacts enter training or benchmark-specific tuning, while webenabled agents can retrieve evaluation-specific instructions, answers, or near-duplicate artifacts during a run, so a score can reflect access to benchmark material rather than the intended interaction capability (Han et al., 2025). Living task pools such as Agents’ Last Exam and state-rich suites such as OSWorld 2.0 reduce staleness and saturation pressure; explicit versioning of tasks, environments, and graders makes these changes traceable (Sun et al., 2026b; Yuan et al., 2026). Mitigation follows the CUA threat model. Closed self-hosted sites can disallow external retrieval and whitelist network access. On the live web, lookup is part of the capability, so timestamped tasks and ground truth track freshness directly. In closed enterprise settings, the risks shift from external STC to internal leakage (answers in the retrieval index) and process leakage (evaluation
logs reused in training) (Han et al., 2025). 2.2
Evaluator Fragility
Open-ended CUA tasks admit multiple valid outcomes and paths. Hard-coded checks of a string, URL, DOM node, or machine state can reject acceptable alternatives; outcome-only checks can instead reward a superficially correct artifact produced through invalid shortcuts (El hattami et al., 2025; Li et al., 2026). “Verified” subsets improve task and evaluator quality while exposing noise in unaudited automation. Semantic, rubric-based, and judge-based matching broaden coverage but add measurement instruments whose reliability depends on independent calibration (Zhuge et al., 2024; Lù et al., 2025; Lin et al., 2026). 2.3
Observability and Diagnostic Opacity
A scalar success rate cannot distinguish a wrong plan from a wrong click, a stale environment, or a checker error. Logs that omit screenshots, tool responses, timestamps, or state transitions can make post-hoc diagnosis impossible, creating an observability failure before taxonomy is applied. Replayable, screenshot-inclusive trajectories preserve the evidence, and a shared vocabulary localizes failure within the CUA loop (Mohammadi et al., 2025; Cemri et al., 2025; Xie et al., 2024). §4 operationalizes this sequence by auditing verdict validity before classifying genuine failures by stage.
3
CUA Evaluation Landscape
We organize the discussion around the three task settings summarized in Table 1, focusing on the cross-cutting demands that distinguish them. 3.1
Specialized Professional GUI
Professional interfaces make domain state part of the task. A visually plausible action can violate a workflow dependency, change the wrong structured object, or produce an artifact that cannot support later work. The generalization problem is thus not transfer to an unfamiliar layout but transfer across domain conventions, latent dependencies, and representations with no visible equivalent on screen (Yu et al., 2026; Dong et al., 2026; Zou et al., 2026; Wang et al., 2026). This setting also separates interface fluency from professional competence. Static target localization can establish whether an agent can operate a dense interface, but end-to-end use additionally requires maintaining domain intent while the artifact and
application state evolve over many actions (Li et al., 2025b; Ai et al., 2025). Thus, progress in generic GUI grounding does not by itself imply reliable transfer to professional work. 3.2
Web Navigation
Web tasks are distinguished by remote, mutable state rather than by the browser alone. The same action can have different consequences depending on authentication, session history, asynchronous updates, and changes made by other services. Moreover, many goals admit several legitimate routes, so success depends on preserving intent across navigation choices instead of reproducing a single interaction sequence (Zhou et al., 2023; Le Sellier de Chezelles et al., 2024; Drouin et al., 2024). The interface representation is likewise incomplete from any single view. DOM structure exposes semantics that pixels may obscure, whereas screenshots retain visual relationships absent from accessibility trees. Agents must reconcile these views while pages reflow and content changes, making web navigation a problem of state interpretation as much as element selection (Koh et al., 2024; Lù et al., 2024). Longer, multi-site tasks amplify this coupling because early retrieval and navigation choices become inputs to much later actions (Jang et al., 2026). 3.3
OS and Mobile
OS-level tasks compose applications through persistent machine state. Information may move from a page to a clipboard, file, application object, or system setting, and later steps may depend on transformations that are no longer visible. Crossapplication competence therefore requires continuity of state and constraints across tool boundaries, not just competence within each interface (Xie et al., 2024; Yuan et al., 2026; Li et al., 2026). Desktop and mobile control share this persistence but impose different interaction bottlenecks. Desktop workflows expose more simultaneous state and richer tool combinations; mobile workflows constrain the viewport and route more transitions through OS-managed surfaces. In both cases, small local mistakes can propagate because subsequent applications consume the state produced earlier, turning recovery and state reconstruction into central long-horizon capabilities (Rawles et al., 2024; Kong et al., 2025).
Task setting
Specialized professional GUI
Application Healthcare administration and clinical software CAD and electronic design automation Scientific instruments and data workflows Creative production
Dynamic navigation Web navigation Enterprise workflows
Desktop control Device control Mobile interaction
Metric / evaluation focus Workflow validity and safety: Deterministic task/subtask completion, correct cross-system state, and clinically consequential error checks. Professional GUI execution: Screenshot-to-action accuracy and successful operation of dense, highresolution industrial CAD interfaces. State- and result-based completion: Subtask and end-to-end success for instrument control, scientific software, and data-pipeline workflows. Artifact and interaction quality: Completion of longhorizon image, design, video, audio, and 3D workflows, including clarification and revision.
Representative benchmarks HealthAdminBench (Bedi et al., 2026), MedCUA-Bench (Yu et al., 2026), ProSoftArena (Ai et al., 2025) GUI-EDA (Li et al., 2025a), CADWorld (Dong et al., 2026), ProSoftArena (Ai et al., 2025), GUI-vs-CLI (Zhou et al., 2026), ScreenSpot-Pro (Li et al., 2025b) LabOSBench (Zou et al., 2026), ScienceBoard (Sun et al., 2026a), Spider2-V (Cao et al., 2024), ProSoftArena (Ai et al., 2025) DeskCraft (Wang et al., 2026), PSBench (Zhang et al., 2026), ProSoftArena (Ai et al., 2025), ScreenSpot-Pro (Li et al., 2025b)
WebArena (Zhou et al., 2023; El hattami et al., 2025), Mind2Web Semantic success: Type-aware exact matching with (Deng et al., 2023), MiniWoB++ (Liu et al., 2018), Assistantsemantic normalization (handling DOM mutations). Bench (Lù et al., 2025), Odysseys (Jang et al., 2026) VisualWebArena (visualwebarenaRepo; Koh et al., 2024), WebVisual and workflow completion: Visual understandVoyager (He et al., 2024), WebLINX (Lù et al., 2024), WorkArena ing of elements; completion of administrative forms. (Drouin et al., 2024; Boisvert et al., 2024) OSWorld (Xie et al., 2024), OSWorld 2.0 (Yuan et al., 2026), Execution-based snapshot: Final file-system state Agents’ Last Exam (Sun et al., 2026b), WeaveBench (Li et al., verification across Ubuntu/Windows apps. 2026) Cross-app workflow: Success rate on multi-app tasks; MobileWorld (Kong et al., 2025), AndroidWorld (Rawles et al., pixel-coordinate precision (GUI grounding). 2024), Mobile-Env (Zhang et al., 2023)
Table 1: CUA benchmark landscape by task setting, application, evaluation focus, and representative benchmarks.
4
Empirical Audit of Benchmark Verdicts
To determine when a benchmark verdict should be distrusted and what a binary failure score conceals, we audit 150 failure-scored trajectories from five CUA benchmarks. For each trajectory, we first assess whether the recorded failure reflects an agent failure, an evaluator false negative, a broken task, or insufficient evidence; only genuine agent failures are then assigned a process-level diagnosis. Two vision-enabled LLM judges independently review the complete trajectories, including reasoning, actions, and screenshots. We anchor their judgments with two human groups that label the same review set independently and blind to the LLM outputs and to each other. The remainder of this section presents the audit setting, labeling procedure, diagnostic taxonomy, and results in that order. 4.1
Audit Setting
We sampled public trajectories with step-level reasoning, actions, and screenshots using a deterministic, stratified procedure (random seed 20260717). From AgentRewardBench (Lù et al., 2025), we formed unique (benchmark, task, agent, experiment) tuples, shuffled within each benchmark– agent stratum, round-robin interleaved agent strata, and kept only trajectories with released cumulative reward zero until reaching quotas of 25 WebArena, 25 WorkArena, 24 VisualWebArena, and 24 AssistantBench trajectories. From OSWorld-Verified, we took 20 zero-reward trajectories from each of three released agent runs, shuffling within application domains and round-robin interleaving the domains. This produced 158 trajectories; eight
(one per OSWorld run and five spread across the web benchmarks and agent models) were selected for codebook calibration and excluded from analysis. The remaining 150 comprise 57 OSWorld trajectories and 93 AgentRewardBench trajectories: 23 each from WebArena, VisualWebArena, and AssistantBench, and 24 from WorkArena. We perform no new agent runs. Because every sampled unit was released as FAIL, the audit estimates false negatives among recorded failures but not false positives among successes. The five benchmarks also expose different evidence to their released oracles. OSWorld evaluates final virtual-machine and application state with task-specific getters and metrics; WebArena and VisualWebArena combine answer, URL, and programmatic state checks, with the latter also requiring visual evidence; WorkArena checks taskspecific ServiceNow records and fields; and AssistantBench matches an open-web answer against a reference answer. These mechanisms define the benchmark-side evidence against which we compare the trajectory-level judgments. 4.2
Labeling and Combination
We operationalize the Introduction’s question— when should a CUA verdict be distrusted?—as a two-stage labeling decision. In Stage 1, each recorded FAIL is labeled genuine agent failure when the trajectory does not satisfy the task and a human grader would also reject it; evaluator false negative when the observed outcome satisfies the task but the checker rejects it; broken task when the specification is infeasible, the reference state is
stale, or the environment or harness blocks completion; or unclear when the released evidence cannot establish the correct judgment. In Stage 2, only genuine failures receive a process diagnosis from the three-tier taxonomy below. This sequence keeps benchmark faults from counting as agent faults. Two vision-enabled LLM annotators from different vendors—GPT-5.5 through OpenAI Codex CLI v0.144.5 and Anthropic Claude Sonnet—label all 150 trajectories independently and blind to each other’s outputs. The experiment artifacts retain the Claude model family but not its exact Sonnet revision. Both receive the same written codebook and the same step-level reasoning, actions, and screenshots. Human review includes all 74 rows on which the LLMs disagree in verdict or category and a seeded, benchmark-by-verdict stratified sample of 30 LLM-agreement rows. Two human groups independently and blindly label the same 104 rows through a screenshot-replay interface; thus, each reviewed trajectory has four independent judgments. After annotation is complete, human agreement determines the final label for reviewed rows, with disagreements resolved by majority vote. The remaining 46 rows use the two-LLM consensus because every LLM disagreement is human-reviewed. 4.3
Three-Tier Diagnostic Taxonomy
Once Stage 1 establishes that the agent genuinely failed, Stage 2 localizes the earliest decisive failure in the agent’s perception–action loop. We adapt MAST (Cemri et al., 2025) from multi-agent coordination to single-agent computer use, informed by OSWorld’s perception–action loop (Xie et al., 2024). We remove intrinsically multi-agent categories, reinterpret communication breakdowns as failures of state propagation across steps, and elevate GUI grounding and long-horizon verification. This separates failures in goal interpretation and strategy from failures that prevent a plausible intent from being executed or maintained (Shinn et al., 2023). Appendix B records the full mapping. Tier 1—planning and specification. The three categories separate specification violations (T1S PEC), repeated planning without a viable alternative (T1-L OOP), and plans built around nonexistent features or capabilities (T1-H ALL). These are intent or strategy failures before successful execution. Tier 2—execution and grounding. Given a plausible intent, grounding failures (T2-G ROUND) select the wrong coordinates or GUI control, state-
propagation failures (T2-S TATE) lose context established earlier, and tool failures (T2-T OOL) use invalid arguments or encounter an action-space limitation (OpenAI, 2025; Xie et al., 2024). Tier 3—verification and feedback. The categories distinguish premature termination (T3T ERM), missed or incorrect verification (T3V ERIF), and feedback-blind repetition of actions that do not change state (T3-N OOP). These failures occur after an action can be checked and corrected. Boundary rule. We assign the earliest failure that makes the trajectory unsuccessful, rather than the most visually salient later symptom. Repetition is Tier 1 when the agent cannot develop an alternative strategy; it is Tier 3 when the agent has a viable plan but fails to notice that an action had no effect. The distinction is therefore causal rather than visual. 4.4
Results and Discussion
Verdict reliability. Across the five benchmarks, 15.3% of audited FAIL verdicts are wrong (95% Wilson CI [10.4, 22.0]): 10.7% [6.7, 16.6] are evaluator false negatives and 4.7% [2.3, 9.3] are broken tasks; another 3.3% [1.4, 7.6] remain unclear from the released evidence (Table 2). The benchmark mechanisms fail in different ways. WebArena’s answer matching rejects acceptable behavior (21.7% false negatives), while AssistantBench’s open-web tasks are vulnerable to environment drift, including dead search engines and CAPTCHA walls (21.7% broken tasks). OSWorld’s state checkers miss alternative valid solutions (14.0%) and include a task that is unsatisfiable on the released VM image. WorkArena’s programmatic state checkers produce no detected wrong verdicts in this sample (0/24). Illustrative cases. WebArena scored FAIL after an agent correctly listed the five single-book recommendations among the top-ten books posts. An AssistantBench eatery query encountered a DuckDuckGo error and then a Google CAPTCHA. OSWorld’s battery-percentage task was impossible because /sys/class/power_supply was empty. These cases show why a recorded failure should be distrusted when the trajectory demonstrates task completion that the oracle does not recognize or when the task cannot be completed in the released environment. Failure diagnoses. Among the 122 genuine failures, the scalar score conceals a strongly skewed
Benchmark
Released oracle and inspected evidence
n GF EFN Broken Unclear Wrong (%)
OSWorld WebArena VisualWebArena WorkArena AssistantBench
Execution-based getters over final VM/application state String, URL, and program checks on answer and DOM state Visual and program checks on answer, URL, and DOM state Programmatic checks over ServiceNow records and fields Open-web answer matching against a reference answer
57 23 23 24 23
All Tier
Subcategory diagnoses among genuine failures (n = 122)
Tier 1 Tier 2 Tier 3 Other Ambiguous
Specification violation 18 Plan loop Grounding 9 Tool arguments Feedback-blind no-op 36 Premature stop Genuine failure outside the three tiers Released evidence does not distinguish tiers after label combination
Subcategory 1
47 18 18 23 16
8 5 3 0 0
2 0 0 0 5
0 0 2 1 2
17.5 21.7 13.0 0.0 21.7
150 122
16
7
5
15.3 Total Share
n Subcategory 2
n Subcategory 3
n
122 100%
20 Hallucinated feature 6 State loss 8 Missed verification
5 2 4
43 35.2% 17 13.9% 48 39.3% 6 4.9% 8 6.6%
Table 2: Unified audit setting and results. The upper panel combines each benchmark’s released oracle with its verdict audit; “Wrong” is the percentage of audited FAILs that are evaluator false negatives or broken tasks. The lower panel reports tier totals and the non-overlapping Stage 2 subcategory diagnoses of genuine failures. GF denotes genuine failure and EFN evaluator false negative.
diagnosis. Tier 3 verification and feedback failures account for 39.3%. Feedback-blind no-op repetition alone accounts for 29.5%, the largest single category. Tier 1 planning failures account for 35.2%, including specification violations (14.8%) and planning loops (16.4%). Tier 2 execution and grounding errors account for only 13.9%; 6.6% remain ambiguous across tiers after label combination, and 4.9% fall outside the three tiers. For example, in a genuine WorkArena failure, the agent clicked “ALL RESULTS” 27 times while “No Results” and the screenshot remained unchanged, exemplifying the dominant Tier 3 no-op pattern. Annotation reliability and observability. The two LLM judges agree substantially on Stage 1 verdicts (κ = 0.71, raw agreement 92.7%) but only moderately on Stage 2 diagnoses (κ = 0.41). On the 104-row common blind-review set, the two human groups reach κ = 0.59 (85.6% raw agreement); pairwise human–LLM κ ranges from 0.19 to 0.32 (76.0–81.7% raw), and four-rater Fleiss’ κ is 0.36. Because this set deliberately contains every LLM disagreement, these are stress-set reliability statistics rather than population agreement estimates. We also quantify variability in the binary wrongverdict score (evaluator false negative or broken task versus all other Stage 1 outcomes) on these 104 commonly reviewed rows. Across GPT-5.5, Claude Sonnet-5, human group 1, and human group 2, the annotator-specific rates are 13.5%, 7.7%, 20.2%, and 13.5%, respectively: mean 13.7%, sample standard deviation 5.1 percentage points, sample variance 0.0026, and range 7.7–20.2%. The final 15.3%
estimate instead uses the post-combination labels for all 150 rows and its Wilson interval above. The evidence available to the judge materially changes these decisions: removing screenshots flips 12 Codex verdicts and 13 Claude verdicts, reduces the evaluator false negatives detected by the two judges from 14 to 8 and from 10 to 6, respectively, and lowers their Stage 1 agreement to κ = 0.60. Thus, screenshot-inclusive trajectories are necessary both for auditing checker validity and for diagnosing genuine failures. Implications. The audit directly answers the paper’s motivating question: a CUA failure verdict warrants distrust when replay evidence contradicts the checker or reveals that the task or environment is invalid. Even when the verdict is correct, it is incomplete unless the released trajectory supports a diagnosis. The difference between substantial verdict agreement and moderate diagnosis agreement also shows that fine-grained labels require explicit operational rules and human calibration.
5
Designing Reliable Benchmarks
This section turns the audit results of §4 into the baseline controls a CUA benchmark needs before its verdicts deserve trust, following the four measurement stages of Figure 1B; Table 3 summarizes the controls for each stage. Construct tasks that practitioners would recognize. Seven recorded failures (4.7%) were broken tasks, not agent failures: AssistantBench’s openweb tasks hit dead search engines and CAPTCHA walls, and one OSWorld task was unsatisfiable on
Stage
Trustworthy baseline (§5)
Threat addressed
Toward better benchmarks (§6)
Task construction
Replay a stratified subsample pre-release to confirm Broken or invalid tasks are solvable as shipped; author tasks around tasks; drift; workflows practitioners use; version tasks, memorization environments, and oracles together, retiring stale ones
Living task pools and procedural refresh; expert-authored workflows and acceptance criteria
Environment
Pin and disclose the execution stack—OS image, application versions, and scaffold packages (e.g., Playwright vs. PyAutoGUI); declare the retrieval boundary (§2); support deterministic replay
Irreproducible or Seeded perturbations and scaffoldself-evolving environments under confounded scores; deterministic replay contamination
Scoring
Check exactly what the task statement declares: accept every solution in that space and verify each constraint; audit checkers against valid alternatives and human labels; report false-negative rates
Evaluator false negatives; reward hacking
Calibrated judge- and rubric-based scoring of process and outcome
Reporting
Report success rate plus a per-failure locus and process-stage diagnosis; release evidence-complete trajectories (screenshots, tool I/O, timestamps); disclose hardware, prices, and evaluation date
Diagnostic opacity; economically misleading comparisons
Automatic diagnostic probes; critical-window slack and deadline-miss curves (latency); dollar cost per verified success
Table 3: Benchmark-design guidelines by measurement stage (Figure 1B). The second column lists the baseline controls a benchmark needs for its verdicts to deserve trust (§5); the last column previews the extensions of §6.
the released VM image. Pre-release replay and joint versioning of tasks, environments, and oracles address this. Validity, however, demands more than solvability: a task must exercise the workflow the profession actually uses (Dong et al., 2026). Benchmark design should thus involve domain experts in defining tasks and the artifacts that count as professionally acceptable. Disclose the execution stack and retrieval boundary. An agent’s score is conditioned on its scaffold: the same model can look competent under one scaffold and helpless under another because the action and observation spaces differ, not the capability. Scores are therefore comparable only when the OS image, application versions, and scaffold packages are pinned and disclosed, and when released environments replay deterministically. The retrieval boundary needs the same rigor. A blanket network ban is inappropriate when live lookup is part of the task; instead, benchmarks should declare a policy matched to the STC threat model of §2: whitelist network access in closed worlds, timestamp tasks and log retrieved sources on the live web, and audit retrieval indexes and evaluation-log reuse in enterprise settings (Han et al., 2025). Score the task that was stated. A checker is valid when it tests exactly what the task statement declares (no less or more). Testing less rejects legitimate work: 16 of 150 audited failures (10.7%) were evaluator false negatives, including five WebArena answers rejected by string matching and
eight OSWorld solutions whose alternative valid paths the state checkers missed. Well-designed oracles therefore accept any solution inside the declared space, using type-aware or semantic normalization whenever the answer space is open (El hattami et al., 2025; Zhuge et al., 2024). Testing more loosely than the statement invites hacking in the other direction: when a task constrains the method, an outcome-only check would credit an equivalent artifact built with a different workbench, rewarding a capability the task never claimed to measure. Task statements and checkers must be co-designed so this “exactly” is tested with both positive and negative cases. Report evidence, not just a verdict. Success rate remains the right top-line metric, but a scalar cannot direct engineering effort. Each failure should carry a two-level diagnosis: first its locus (task/environment, trajectory/evidence, evaluator, or agent) and then, for genuine agent failures, the process stage and reason category. In our audit, verification and feedback failures (39.3%) and planning failures (35.2%) dominated execution and grounding errors (13.9%)—a distribution the scalar conceals entirely. Diagnosis also depends on the released evidence: removing screenshots flipped 12 Codex and 13 Claude verdicts and roughly halved the evaluator false negatives either judge detected. Benchmarks should therefore release actions, observations, screenshots, tool I/O, timestamps, and state transitions alongside the scores (Cemri et al.,
2025; Guo et al., 2025), and disclose serving hardware, price schedules, and the evaluation date so that latency and cost can be reconstructed; §6 develops the corresponding first-class metrics.
6
Design Better Benchmarks
This section develops, for each measurement stage of §5, methods that are more diagnostic, scalable, and harder to saturate, as previewed in Table 3. Task construction: living pools and professional standards. Static task sets saturate and leak, so tasks must evolve faster than the models they test: Agents’ Last Exam maintains a living task pool on a declared schedule, and self-evolving or procedural task generation varies file names, layouts, and data values to defeat memorization (Sun et al., 2026b; Xi et al., 2025; Guo et al., 2025; Wang et al., 2019). The second frontier is who defines the tasks. Existing suites measure whether an agent can operate professional software; almost none ask whether the profession would accept the result. Extending the domain-validity guideline of §5, domain experts should author not only the workflows but also the acceptance criteria for finished, correct work (Ai et al., 2025; Dong et al., 2026). Environment: realistic variation under deterministic replay. Beyond a pinned, disclosed stack, better environments inject controlled variation: seeded randomization reproduces real-world mess like network lag, transient UI drift, and asynchronous updates, testing robustness without giving up deterministic replay, while self-evolving environments such as AgentGym grow alongside the agent so the evaluation never hits a ceiling (Wang et al., 2019; Xi et al., 2025; Guo et al., 2025). Scoring: calibrated judges over process, not just outcome. As open-ended tasks outgrow brittle scripts, scoring is shifting from outcome-only checks to rubric- and process-aware instruments: WeaveBench inspects deliverables together with screenshots, files, and action traces; Odysseys grades rubric completion per step; and CUARewardBench separately tests outcome and process reward models (Li et al., 2026; Jang et al., 2026; Lin et al., 2026). The instrument that makes this scale is Agent-as-a-Judge: a judge more capable than the system under test, performing verification against logged trajectories. This judge is trusted only after meta-evaluation against human labels, as §4.2 practices (Zhuge et al., 2024; Lù et al., 2025).
With practitioner-authored rubrics, the same machinery can grade work by professional standards rather than generic step checklists. Reporting: automatic diagnostics before expensive judges. The diagnostic reporting of §5 does not require annotating every failure with humans or judges. The most common failure in our audit that involves an action repeated while the screen observably does not change (29.5% of genuine failures), can be flagged by scanning each trajectory for repeated actions with unchanged consecutive observations, at zero model cost; judge or human effort is then spent only on the planning and grounding failures that need semantic judgment. Such a probe must first be validated against human labels, since legitimate no-change intervals, such as page loads and asynchronous rendering, would otherwise be misflagged. Reporting: latency and cost as first-class metrics. Standard reports say whether an agent succeeded, not whether it acted in time or at what price. Short-lived controls such as BIOS prompts and countdown links can disappear before an agent acts. For such a task, let R be the number of benchmark-controlled observation–action opportunities during which the target remains actionable and r ∈ {1, . . . , R} the opportunity on which the correct action is issued; critical-window slack, Swin = (R − r + 1)/R (zero on a miss), is hardware-independent because it counts environment opportunities, not seconds. Physical latency cannot be normalized away, so wall-clock deadlinemiss rate and success-versus-deadline curves at declared compute limits and hardware complete the picture. Token counts are not comparable across models or providers; with C covering agent inference, tool calls, environment runtime, and verification, reports should give dollar cost per attempt (C/Nattempt ) and per verified success (C/Nsuccess ) with the price schedule, exposing a cost–success frontier that trajectory pruning makes cheaper to occupy: AgentDiet cuts input tokens by up to 59.7% without sacrificing success (Xiao et al., 2025).
7
Conclusion
We showed how contamination, evaluator fragility, and diagnostic opacity distort CUA benchmark verdicts. We hope readers see scores as measurementpipeline outputs, judge when a verdict deserves trust, and build benchmarks that earn it.
Limitations Audit scope. Our claims are restricted to GUI computer-use evaluation. The audit covers 150 trajectories from five benchmarks, all drawn from public artifacts in web, enterprise-workflow, and desktop-control settings. The specialized healthcare, CAD/EDA, scientific-instrument, and creative-production resources listed in Table 1 are contextual and receive no empirical claim. Our 2026 benchmark discussion establishes continuity with newer long-horizon evaluation designs but does not retrospectively add them to the audit sample. Per-benchmark samples are small (23–57), so the benchmark-level rates in Table 2 carry wide intervals and should be read as evidence that evaluator error occurs at a material rate, not as calibrated per-benchmark estimates. We sampled only failurescored trajectories, so we estimate evaluator false negatives but say nothing about false positives; the complementary audit of PASS verdicts remains open. Survey scoping. Screening was iterative and interleaved with drafting rather than executed as a single logged pass, so we report the included corpus rather than PRISMA-style identification and exclusion counts (Appendix D). Unvalidated design inference. The detectioncost argument of §6 is conceptual; we do not validate a programmatic probe against our human labels, and we make no claim about what accuracy such a probe would achieve.
References Jiaxin Ai, Yukang Feng, Fanrui Zhang, Jianwen Sun, Zizhen Li, Chuanhao Li, Yifan Chang, Wenxiao Wu, Ruoxi Wang, Mingliang Zhai, and Kaipeng Zhang. 2025. ProSoftArena: Benchmarking hierarchical capabilities of multimodal agents in professional software environments. arXiv preprint arXiv:2601.02399. Alibaba Group. 2026. Alibaba’s qwen app advances agentic ai strategy by turning core ecosystem services into executable ai capabilities. Alibaba Group announcement. Accessed 2026-01-31. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, and 1 others. 2025. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. In Proceedings of the 63rd Annual Meeting of the Association for
Computational Linguistics (Volume 2: Short Papers), pages 238–255. Suhana Bedi, Ryan Welch, Ethan Steinberg, Michael Wornow, Taeil Matthew Kim, Haroun Ahmed, Peter Sterling, Bravim Purohit, Qurat Akram, Angelic Acosta, Esther Nubla, Pritika Sharma, Michael A. Pfeffer, Sanmi Koyejo, and Nigam H. Shah. 2026. HealthAdminBench: Evaluating computer-use agents on healthcare administration tasks. arXiv preprint arXiv:2604.09937. Léo Boisvert, Megh Thakkar, Maxime Gasse, Massimo Caccia, Thibault Le Sellier De Chezelles, Quentin Cappart, Nicolas Chapados, Alexandre Lacoste, and Alexandre Drouin. 2024. Workarena++: Towards compositional planning and reasoningbased common knowledge work tasks. Preprint, arXiv:2407.05291. Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, and 4 others. 2024. Spider2-V: How far are multimodal agents from automating data science and engineering workflows? In Advances in Neural Information Processing Systems, volume 37, pages 107703–107744. Mert Cemri, Melissa Z Pan, Shuyi Yang, Lakshya A Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Dan Klein, Kannan Ramchandran, and 1 others. 2025. Why do multi-agent llm systems fail? arXiv preprint arXiv:2503.13657. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Xuehai Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track. Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Kaixin Li, and Qishi Zhan. 2026. CADWorld: A CAD-centric benchmark for spatial, precise, and long-horizon computer-use agents. Manuscript. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, and 1 others. 2024. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718. Amine El hattami, Megh Thakkar, Nicolas Chapados, and Christopher Pal. 2025. Webarena verified: Reliable evaluation for web agents. In Workshop on Scaling Environments for Agents. Hayden Field. 2025. Openai operator agent can automate tasks such as vacation planning. CNBC. Published 2025-01-23; accessed 2026-01-31.
Dadi Guo, Tianyi Zhou, Dongrui Liu, Chen Qian, Qihan Ren, Shuai Shao, Zhiyuan Fan, Yi R Fung, Kun Wang, Linfeng Zhang, and 1 others. 2025. Towards self-evolving benchmarks: Synthesizing agent trajectories via test-time exploration under validate-by-reproduce paradigm. arXiv preprint arXiv:2510.00415. Ziwen Han, Meher Mankikar, Julian Michael, and Zifan Wang. 2025. Search-time data contamination. arXiv preprint arXiv:2508.13180. Shijie He and 1 others. 2024. Webvoyager: Building an end-to-end web agent with large multimodal models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). Lawrence Keunho Jang, Jing Yu Koh, Daniel Fried, and Ruslan Salakhutdinov. 2026. Odysseys: Benchmarking web agents on realistic long horizon tasks. arXiv preprint arXiv:2604.24964. Jing Yu Koh and 1 others. 2024. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). Quyu Kong and 1 others. 2025. Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive, and MCP-augmented environments. arXiv. ArXiv:2512.19432. Thibault Le Sellier de Chezelles, Maxime Gasse, Alexandre Lacoste, Alexandre Drouin, Massimo Caccia, Léo Boisvert, and 1 others. 2024. The BrowserGym ecosystem for web agent research. Transactions on Machine Learning Research (TMLR). Chunyi Li, Longfei Li, Zicheng Zhang, Xiaohong Liu, Min Tang, Weisi Lin, and Guangtao Zhai. 2025a. Using GUI agent for electronic design automation. arXiv preprint arXiv:2512.11611. Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma, Zhiyong Huang, and Tat-Seng Chua. 2025b. ScreenSpot-Pro: GUI grounding for professional high-resolution computer use. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 8778–8786. Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li, and Caihua Shan. 2026. Weavebench: A long-horizon, real-world benchmark for computer-use agents with hybrid interfaces. arXiv preprint arXiv:2606.09426. Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, and 1 others. 2026. CUARewardBench: A benchmark for evaluating reward models for computer-using agents. In Proceedings of the 43rd International Conference on Machine Learning. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. In International Conference on Learning Representations (ICLR). ArXiv:1802.08802.
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J Pal, and Siva Reddy. 2025. Agentrewardbench: Evaluating automatic evaluations of web agent trajectories. arXiv preprint arXiv:2504.08942. Xing Han Lù, Zdeněk Kasner, and Siva Reddy. 2024. WebLINX: Real-world website navigation with multiturn dialogue. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Mahmoud Mohammadi, Yipeng Li, Jane Lo, and Wendy Yip. 2025. Evaluation and benchmarking of llm agents: A survey. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6129–6139. OpenAI. 2025. Computer-using agent. OpenAI blog. Published 2025-01-23; accessed 2026-01-27. Christopher Rawles and 1 others. 2024. Androidworld: A dynamic benchmarking environment for autonomous agents. In Advances in Neural Information Processing Systems (NeurIPS). Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems (NeurIPS). Robby Stein. 2025. AI mode in search gets new agentic features and expands globally. Google blog (Products / Search). Published 2025-08-21; accessed 2026-0131. Qiushi Sun, Zhoumianze Liu, Chang Ma, Zichen Ding, Fangzhi Xu, Zhangyue Yin, Haiteng Zhao, Zhenyu Wu, Kanzhi Cheng, Zhaoyang Liu, Jianing Wang, Qintong Li, Xiangru Tang, Tianbao Xie, Xiachong Feng, Xiang Li, Ben Kao, Wenhai Wang, Biqing Qi, and 2 others. 2026a. ScienceBoard: Evaluating multimodal autonomous agents in realistic scientific workflows. In The Fourteenth International Conference on Learning Representations. Yiyou Sun, Xinyang Han, Weichen Zhang, Yuanbo Pang, Tianyu Wang, Yuhan Cao, Yixiao Huang, and 1 others. 2026b. Agents’ last exam. arXiv preprint arXiv:2606.05405. visualwebarenaRepo. 2024. VisualWebArena: A benchmark for multimodal agents. GitHub repository. Accessed 2026-01-27. Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. 2024. Mobile-agent-v2: Mobile device operation assistant with effective navigation via multiagent collaboration. Advances in Neural Information Processing Systems, 37:2686–2710.
Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. 2019. Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv:1901.01753. Wenkai Wang, Tao Xiong, Jingchen Ni, Yunpeng Bao, Xiyun Li, Tianqi Liu, Hongcan Guo, Zilong Huang, and Shengyu Zhang. 2026. DeskCraft: Benchmarking desktop agents on professional workflows and human-in-the-loop collaboration. arXiv preprint arXiv:2606.03103. Zhiheng Xi, Yiwen Ding, and 1 others. 2025. AgentGym: Evaluating and training large language modelbased agents across diverse environments. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 27914–27961.
Yinuo Zhang, Zian Cheng, Ziya Zhao, Zongyu Li, Bingshuo Liu, Qingbin Liu, Junxian Cai, Xi Chen, Zhiying Tu, Dianhui Chu, Xiaoyan Yu, and Dianbo Sui. 2026. PSBench: Editing image via GUI agents in photoshop. In Proceedings of the 43rd International Conference on Machine Learning, volume 306 of Proceedings of Machine Learning Research. Shuyan Zhou and 1 others. 2023. Webarena: A realistic web environment for building autonomous agents. In Advances in Neural Information Processing Systems (NeurIPS). Xiao Zhou, Siyue Zhang, Yilun Zhao, Jinbiao Wei, Tingyu Song, Arman Cohan, and Chen Zhao. 2026. GUI vs. CLI: Execution bottlenecks in screen-only and skill-mediated computer-use agents. arXiv preprint arXiv:2606.24551.
Yuan-An Xiao, Pengfei Gao, Chao Peng, and Yingfei Xiong. 2025. Improving the efficiency of llm agent systems through trajectory reduction. arXiv preprint arXiv:2509.23586.
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. 2024. Agent-as-ajudge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934.
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, and 1 others. 2024. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37:52040–52094.
Anqi Zou, Han Deng, Chengyu Zhang, Junquan Hu, Yu Wang, Yuxiang Xing, Aokai Zhang, Hanling Zhang, Zhaoyang Liu, Ben Fei, Zhihui Wang, and Wanli Ouyang. 2026. LabOSBench: Benchmarking computer use agents for scientific instrument control. arXiv preprint arXiv:2606.16802.
Eunice Xu. 2025. Bytedance’s agentic AI smartphone dials up a digital backlash from china’s top apps. South China Morning Post. Published 2025-12-07; accessed 2026-01-31. Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of LLM-based agents. arXiv preprint arXiv:2503.16416. Jia Yu, Zilong Wang, Xinyang Jiang, Dongsheng Li, and Shuo Wang. 2026. MedCUA-Bench: A screenshotonly benchmark for clinical computer-use agents. arXiv preprint arXiv:2606.03203. Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, and 1 others. 2026. OSWorld 2.0: Benchmarking computer use agents on long-horizon real-world tasks. arXiv preprint arXiv:2606.29537. Chi Zhang and 1 others. 2024. Appagent: Multimodal agents as smartphone users. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Danyang Zhang and 1 others. 2023. Mobile-env: Building qualified evaluation benchmarks for LLM-GUI interaction. arXiv preprint arXiv:2305.08144.
A
Benchmark Details
This appendix gives supporting detail for the GUICUA benchmarks summarized in Table 1. A.1
Specialized Professional GUI Use
Healthcare. HealthAdminBench evaluates endto-end prior-authorization, appeals, and equipmentorder workflows across an EHR, payer portals, and a fax system, using fine-grained deterministic checkpoints in addition to task success (Bedi et al., 2026). MedCUA-Bench instead reconstructs clinical software from real manuals and open-source systems, separating intent- and step-level goals and checking both completion and clinical safety dimensions (Yu et al., 2026). ProSoftArena adds a broader Health & Medicine split covering GUI operation of ImageJ, ChemDraw, and RGui for diagnostic, pharmaceutical, and public-health work (Ai et al., 2025). CAD and electronic design. GUI-EDA targets screenshot-grounded interaction with five industrial electronic-design CAD tools and reports comprehension and execution accuracy (Li et al., 2025a). CADWorld targets spatially precise, longhorizon GUI interaction in CAD software (Dong et al., 2026). ProSoftArena contributes executionchecked AutoCAD and SolidWorks tasks (Ai et al., 2025); the matched GUI-vs-CLI benchmark contributes a screen-only GUI arm that includes FreeCAD artifact-editing tasks (Zhou et al., 2026). Scientific workflows. LabOSBench evaluates GUI agents on eight simulated scientific instruments, with execution-based checks at subtask and end-to-end levels (Zou et al., 2026). ScienceBoard covers real scientific software and 169 workflows across six disciplines, including GUI-only and GUI–CLI tasks (Sun et al., 2026a). Spider2-V evaluates GUI management of professional datascience and engineering systems across the data pipeline (Cao et al., 2024). ProSoftArena supplies further ImageJ, ChemDraw, ArcGIS, ANSYS, MultiSim, and RGui tasks (Ai et al., 2025). Creative production. DeskCraft covers longhorizon design, video, audio, and 3D-creation workflows and additionally evaluates mid-task clarification and post-task revision (Wang et al., 2026). PSBench contributes 600 human-annotated Photoshop tasks and category-specific checks for nondestructive image editing (Zhang et al., 2026); ProSoftArena adds execution-checked Photoshop
and Illustrator tasks (Ai et al., 2025). ScreenSpotPro supplies a complementary, static test of target localization in high-resolution professional interfaces; because it measures grounding rather than completed workflows, we treat it as an additional capability evaluation rather than an end-to-end application benchmark (Li et al., 2025b). A.2
Web Navigation
Semantic web-navigation benchmarks. • WebArena: A foundational, self-hosted benchmark involving functional e-commerce, forums, and GitLab instances. It shifted the focus from static snapshots to functional correctness in stateful environments (Zhou et al., 2023; El hattami et al., 2025). • Mind2Web: A large-scale dataset of realworld web trajectories across > 130 domains, highlighting the difficulty of generalizing across diverse UI patterns (Deng et al., 2023). • MiniWoB++: A reinforcement learning-style environment with controlled, programmatic tasks that serve as a baseline for low-level browser interactions (Liu et al., 2018). • AgentRewardBench: A meta-benchmark that evaluates the reliability of reward models (evaluators) themselves, revealing that heuristic evaluators in web tasks often have high false negative rates (Lù et al., 2025). Multimodal and enterprise web benchmarks. • VisualWebArena: Requires agents to process visual cues (e.g., “Find the red dress in the image”), exposing the limitations of agents that cannot “see” beyond DOM structure (visualwebarenaRepo; Koh et al., 2024). • WebVoyager: An end-to-end framework for evaluating agents on the live web, highlighting divergence between snapshot-based evaluation and real-world performance drift under dynamic content updates (He et al., 2024). • WorkArena: Focuses on specialized enterprise workflows (ServiceNow), testing the limits of agents in administrative environments where state mutation is complex and requires strict adherence to business logic (Drouin et al., 2024; Boisvert et al., 2024).
• WebLINX: A multimodal benchmark for midhorizon web navigation that emphasizes alignment between visual frames and HTML structures (Lù et al., 2024). A.3
OS and Mobile
Long-horizon professional computer use. • OSWorld 2.0 extends desktop evaluation to 108 end-to-end workflows whose median human completion time is about 1.6 hours. It reports both binary and partial completion and analyzes failures involving constraint tracking, intermediate information, hidden state, and verification (Yuan et al., 2026). • Agents’ Last Exam (ALE) contains 1K+ expert-sourced tasks spanning 55 professional subfields and 13 industry clusters. Agents operate real software in reproducible Windows and Linux sandboxes through GUI and terminal interfaces; deterministic and judge-based graders compare output artifacts against hidden references (Sun et al., 2026b). • WeaveBench contains 114 hybrid-interface tasks that combine GUI, CLI, code, and browser operations. Its trajectory-aware judge inspects files, screenshots, logs, and action traces, providing a contemporary example of evidence-rich evaluation (Li et al., 2026). Visual grounding systems. OSWorld: Evaluates multimodal agents in real computer environments spanning Ubuntu, Windows, and macOS, with 369 tasks requiring cross-application workflows (Xie et al., 2024). AppAgent: An architecture for multimodal agents as smartphone users that illustrates how even conceptually correct identification of a UI element can fail due to small spatial grounding errors (Zhang et al., 2024). OpenAI Computer-Using Agent: A frontier system that operates directly on the computer interface, highlighting the shift from text-only evaluation to pixel-level motor control (OpenAI, 2025). Exploration and mobile-control benchmarks. MobileWorld: Restores headroom for mobile evaluation with 201 tasks emphasizing longer horizons (average 27.8 steps) and cross-application interactions (Kong et al., 2025). AndroidWorld: Established a reproducible, emulated environment for mobile agents, though it has
reached a “saturation point” for specialized agents on simpler, single-app tasks (Rawles et al., 2024). Mobile-Agent-v2: Mitigates exploration cost by utilizing improved memory and multi-agent coordination to navigate effectively (Wang et al., 2024).
B
Failure-Taxonomy Details
This appendix records the mapping from MAST’s multi-agent categories to the single-agent computer-use taxonomy. MAST category / mode
Disposition
CUA audit category
Disobey task specification Disobey role specification Step repetition
Kept Removed Reinterpreted
T1-SPEC Not applicable to one agent T1-LOOP if replanning fails; T3-NOOP if feedback is ignored T2-STATE T3-TERM
Loss of conversation history Reinterpreted Unaware of termination con- Reinterpreted ditions Reasoning–action mismatch Reinterpreted Communication breakdowns Reinterpreted (reset, ignored input, withholding) Other intrinsically inter-agent Removed misalignment Premature termination Kept Incorrect or incomplete verifi- Kept/elevated cation GUI grounding (not explicit Reinterpreted / in MAST) elevated
T2-GROUND or T2-TOOL Within-agent state propagation: T2-STATE Not applicable to one agent T3-TERM T3-VERIF; observable nochange becomes T3-NOOP T2-GROUND
Table 4: Provenance mapping from MAST (Cemri et al., 2025) to the CUA audit codebook.
Methodology. We adapt MAST to the singleagent setting by removing intrinsically multi-agent categories, reinterpreting communication breakdowns as failures of state propagation across steps, and adding GUI grounding and long-horizon verification as first-class computer-use categories.
C
Future-Direction Details
This appendix expands the future-directions discussion of §6: Agent-as-a-Judge methodology, trajectory and efficiency details, and dynamicbenchmark mechanisms. C.1
Agent-as-a-Judge
Methodology: A strong model grades from the user request, the agent’s trajectory, and the final output, scoring both workflow quality and outcome correctness (Zhuge et al., 2024; Bavaresco et al., 2025). Performance & Cost: Agent-as-a-Judge can reach ∼90% agreement with human experts (vs. ∼70% for simple LLM-as-a-Judge prompts) while cutting evaluation cost by ∼97% (e.g., 86 hours to 2 hours) (Zhuge et al., 2024).
Meta-Evaluation: Judge trust must be earned via audits against human-labeled trajectories; benchmarks like AgentRewardBench stress-test reliability (e.g., verbosity bias, self-preference) before judge scores are treated as evidence (Lù et al., 2025). C.2
Trajectory and Efficiency
The field is moving from “outcome-based” evaluation (did it pass?) to “process-based” evaluation (how did it pass?). The TRACE framework makes the trajectory a first-class artifact and validates the step-by-step logic under a validate-by-reproduce requirement (Guo et al., 2025). Efficiency Metrics (AgentDiet): As benchmarks shift to reward “lean” agents instead of capable ones, AgentDiet prunes redundant or expired context from trajectories. Empirical studies show this reduces input tokens by up to 59.7% and costs by up to 35.9% without sacrificing success rates (Xiao et al., 2025). This matters most in token-heavy judge-based evaluations (Tin + Tout ): pruning lowers the marginal cost of verification and transitions complex judge-based audits from “research-only” into budgeted, periodic runs for smaller teams. C.3
Dynamic Benchmarks
To combat benchmark saturation and contamination, evaluation environments must evolve faster than the models they test. (i) AgentGym & AgentEvol: These frameworks propose environments that “grow.” AgentGym provides a suite of diverse, interactive environments, and its AgentEvol method allows agents to evolve by exploring them rather than imitating static datasets (Xi et al., 2025). Combined with task-evolution mechanisms such as TRACE’s evolutionary proposer (Guo et al., 2025), this creates an “arms race” between the agent and the benchmark, ensuring the evaluation never hits a ceiling. (ii) Procedural Generation: Future benchmarks will likely use procedural generation to create infinite variations of tasks (e.g., varying file names, UI layouts, and data values in OSWorld) to prevent overfitting and memorization (Wang et al., 2019). (iii) Controlled stochasticity (seeded randomization): To reconcile realism with reproducibility, benchmarks can inject “seeded randomization.” By introducing stochastic elements like network lag or transient UI drift via an explicit random seed, labs can test agents against real-world “mess” while maintaining deterministic reproducibility. In this
framework, system snapshots provide the foundation, while seeded perturbations provide the realism layer.
D
Scoping-Review Protocol
We characterize our literature process as a scoping review: it aims for representative coverage of agentevaluation practice, not the exhaustive, protocolregistered coverage of a PRISMA-style systematic review. Sources. ACL Anthology (ACL, EMNLP, NAACL); OpenReview and official proceedings for NeurIPS, ICLR, and ICML; arXiv (cs.CL, cs.AI, cs.HC); official benchmark repositories on GitHub; and industry technical reports and documentation. Query strategy. Four query families, each run across the sources above and complemented by forward and backward citation chasing from seed papers: (i) (“computer-use agent” OR “GUI agent”) AND (benchmark OR evaluation OR leaderboard); (ii) (“web agent” OR “desktop agent” OR “mobile agent”) AND evaluation; (iii) seed-benchmark citation chasing from WebArena, WorkArena, OSWorld, and ScreenSpot-Pro; (iv) (contamination OR “evaluator bias” OR “reward hacking”) AND (“computer use” OR “GUI agent”). Inclusion criteria. An artifact is included if it (i) evaluates LLM-based agents that perceive and act through graphical web, desktop, or mobile interfaces, or audits such an evaluation; (ii) was released between 2022 and mid-2026; (iii) is available in English; and (iv) provides enough methodological detail to classify its evaluator type. Exclusion criteria. We exclude (i) pure text generation, static QA, API-only tool use, and code- or command-line-only benchmarks with no GUI action loop; (ii) multi-agent-only coordination frameworks, which fall outside our single-agent scope; (iii) position papers without evaluable methodology; and (iv) duplicate versions of the same work (arXiv and venue versions counted once, citing the venue version where available). Corpus. Screening was performed iteratively alongside drafting rather than in a single logged pass, so we do not report PRISMA-style identification and exclusion counts; per-stage tallies were not recorded, which is a limitation of the protocol (and an instance of the logging discipline that §5 asks of
benchmarks). The verifiable outcome of screening is the cited corpus itself, whose benchmark-bearing subset is coded in Table 1. Borderline candidates were excluded chiefly under criteria (i) (no action loop) and (iii) (no evaluable methodology). Coding. Included artifacts were coded along four dimensions: domain, evaluator type (string/state matching, programmatic state checks, LLM or agent judge, human), failure modes discussed, and mitigations proposed. Table 1 preserves the screened GUI-CUA landscape, while the empirical audit centers the benchmarks with released trajectories.