iOSWorld: A Benchmark for Personally Intelligent Phone Agents

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

iOSWorld: A Benchmark for Personally Intelligent Phone Agents

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

iOSWorld: A Benchmark for Personally Intelligent Phone Agents Lawrence Keunho Jang, Mareks Woodside,∗ Geronimo Carom,∗ Andrew Keunwoo Jang,∗ Jing Yu Koh, Ruslan Salakhutdinov Carnegie Mellon University {ljang, rsalakhu}@cs.cmu.edu

arXiv:2606.09764v1 [cs.LG] 8 Jun 2026

Abstract A useful phone agent needs to be personally intelligent. It should reason over a user’s identity, history, and preferences as they exist on the device, not just follow isolated instructions in an impersonal sandbox. Existing mobile agent benchmarks lack this kind of personalization. We introduce iOSWorld, the first interactive native iOS simulator benchmark built around a persistent user identity spanning 26 newly built iOS apps. These apps contain connected data such as transactions, messages, travel records, social relationships, and financial activity. iOSWorld includes 133 tasks across three increasingly difficult categories. Single-app tasks (27) test one app, multi-app tasks (60) span 2 to 8 apps, and memory and personalization tasks (46) require agents to infer patterns from personal data. We evaluate frontier and open-source computer-use models in both vision-only and privileged vision+XML settings. The best configuration reaches 52% overall but only 37% on multi-app tasks. Privileged vision+XML access improves frontier models by up to 26 percentage points, while smaller models do not benefit from added accessibility-tree input. We release iOSWorld as an open-source benchmark with all apps, seeded data, tasks, rubrics, and evaluation code at https://iosworld.io.

Introduction

A person’s phone is not a blank slate. Transactions, messages, social connections, and financial records accumulate across many applications, forming a record that any useful assistant has to understand and navigate. We call the corresponding agent capability personally intelligent: reasoning over a user’s identity, history, and preferences as they exist on the device, rather than executing sandboxed, isolated tasks. Current phone-agent benchmarks ignore this dimension. Tasks are issued against app states with no persistent user data, no cross-app continuity, and no notion of a real user. An agent that taps the right button on a settings screen but cannot find its owner’s most common commute route has not shown useful capability. Existing benchmarks evaluate digital agents on Android (Rawles et al., 2025; Kong et al., 2025), web (Zhou et al., 2024; Koh et al., 2024), and desktop (Xie et al., 2024; Yang et al., 2025; Bonatti et al., 2025). iOS serves over 2.5 billion active devices1 and 58–60% of U.S. mobile OS usage2 , yet interactive phone-agent benchmarks target Android, not native iOS. None populate apps with persistent user identity. We exclude tasks centered on web browsing, since web agents are already well-served by existing work (Zhou et al., 2024; Koh et al., 2024; He et al., 2024) and a phone agent with browser access can be evaluated there directly. Our focus is on native iOS apps and the personal data they hold. ∗ Equal contribution. 1Apple

installed-base figure, 2026: https://finance.yahoo.com/news/ apple-installed-tops-2-5-170414353.html 2 StatCounter Global Stats (accessed March 2026): https://gs.statcounter.com/os-market-share/ mobile/united-states

26 Custom iOS Apps

CityRide

QuickBite

TeamChat

LockedIn

Jordan Avery

iOSWorld 133 Tasks 1 Persistent User Identity

San Francisco

DineSpot

410 Brannan St

MegaMart

QuickChat

SkyTrip

Northstar Studio

Interconnected data across all applications: contacts, transactions, travel, and preferences

Figure 1: Overview of iOSWorld. 26 purpose-built iOS applications share a single user identity (Jordan Avery) and connected data across apps. The benchmark includes 133 tasks across single-app, multi-app, and memory/personalization categories. iOSWorld is the first dynamic native iOS simulator benchmark built around a user’s personal identity. We built 26 native iOS applications and populated them with connected data for a single persona, Jordan Avery. The same contacts appear across messaging, payment, and email. A food order on one app produces a bank charge and a receipt email in others. An upcoming flight matches a hotel booking and confirmation emails across separate apps. We release 133 tasks in three categories. Single-app tasks (27) test basic interaction within one app. Multi-app tasks (60) carry information across 2 to 8 applications. Memory and personalization tasks (46) require agents to discover implicit patterns from in-app data without being told where to look. The release includes a schema for adding new tasks and seeding personalized data. Our contributions: • The first interactive native iOS simulator benchmark with one user identity spanning 26 purpose-built applications containing connected personal data. • 133 tasks across three categories, evaluated with an LLM-as-a-judge pipeline validated against human annotators (κ=0.77). • A comparison of five frontier models and one open-source baseline (Qwen3.5 35B-A3B) under vision-only and privileged vision+XML settings. The best overall configuration achieves 52% overall (82% single-app, 54% memory, 37% multi-app). Privileged vision+XML access improves the stronger frontier models by up to 26 percentage points, while smaller models do not show the same gain. • We open-source all apps, seed data, tasks, rubrics, and evaluation code, plus an AWSrunner (EC2-managed Mac instances) so non-Mac researchers can run the benchmark. Code at https://github.com/ljang0/iOSWorld and site at https://iosworld.io.

Related Work

2.1

GUI Agent Benchmarks

GUI Agents have largely been centered on the web and desktop. Foundational benchmarks such as MiniWoB, MiniWoB++, WebShop, WebArena, and VisualWebArena (Shi et al., 2017; Liu et al., 2018; Yao et al., 2022; Zhou et al., 2024; Koh et al., 2024) emulated tasks on the web. Mind2Web (Deng et al., 2023) and WebVoyager (He et al., 2024) extended to real websites, and Xue et al. (2025) studied how evaluations transfer to live conditions. At the OS and Desktop level, OSWorld (Xie et al., 2024) covers Linux, Windows Agent Arena (Bonatti et al., 2025) targets Windows, MacOSWorld (Yang et al., 2025) covers macOS, and WorkArena (Drouin et al., 2024) benchmarks enterprise knowledge work. GAIA (Mialon et al., 2024), TheAgentCompany (Xu et al., 2025a), and τ-bench (Yao et al., 2025) probe multistep and multi-tool reasoning. All of these benchmarks present agents on desktops with predominantly impersonal or single user environments and explicit instructions. 2

2.2

Mobile Device Agents

Existing interactive mobile-agent benchmarks target Android. AndroidEnv (Toyama et al., 2021) provides an RL interface for phone agents, Android-in-the-Wild (Rawles et al., 2023) provides evaluation using human demonstrations on Android, and AndroidWorld (Rawles et al., 2025) offers dynamic tasks with programmatic verification on a live Android simulator. MobileWorld (Kong et al., 2025) extends AndroidWorld with long-horizon and tool use. Additional benchmarks have expanded coverage across different digital domains, such as AndroidLab (Xu et al., 2025b), SPA-Bench (Chen et al., 2025), B-MoCA (Lee et al., 2025), and GUI Odyssey (Lu et al., 2025). On the modeling side, CogAgent (Hong et al., 2024), AppAgent (Zhang et al., 2025), Mobile-Agent (Wang et al., 2024), UI-TARS (Qin et al., 2025), and AutoDroid (Wen et al., 2024) explore architectures ranging from fine-tuned VLMs to RL-trained agents (Bai et al., 2024; 2025) for mobile agents. Earlier work on inducing mobile skills from user demonstrations (Shen et al., 2019) pre-dates the LLM-agent era. Ferret-UI (You et al., 2024) develops a multimodal model for understanding mobile UI on both Android and iOS. There remains a gap for dynamic iOS evaluations. iOS differs from Android in its UI framework, navigation patterns, and accessibility infrastructure. No mobile benchmark on any platform seeds applications with a user identity or evaluates reasoning over extensive personal data distributed across apps.

Jordan’s Digital Life

26 apps across 10 life domains · one persistent identity Clock

Mail

Apple Clock

ITIE UTIL

Weather

MES

Apple Mail

SAG

ING

QuickChat WhatsApp

Apple Weather

TeamChat Slack

Strava

FI TN

TrailBlaze

MyBank

CalTrack

Chase

ScoreZone

SplitPay

TicketBox Cinephile

SkyTrip TRAVEL

SeatGeek

Venmo

ENTERT AINMEN

ESPN

MyFitnessPal

Jordan Avery Staff Automation Engineer

Letterboxd

StayFinder Airbnb

Northstar Studio

San Francisco, CA · Age 31

Notes

Delta

CityRide

[email protected]

Apple Notes

Uber

CloudSlides

QuickBite

Google Slides

DoorDash

CloudSheets D FO O

CT IV Y

CloudDrive Google Drive

LockedIn

TasteRank Beli

Google Docs

Instacart

DU O

CloudDocs

FreshCart PR

Google Sheets

DineSpot G

PIN NAL SHOP

PROFESSIO

OpenTable

MegaMart Amazon

Figure 2: Jordan Avery’s digital life. 26 iOS apps across 10 domains sharing one identity. We display app names in bold and real-life analogues in italics. Edge thickness represents the number of shared data points; Mail is the primary hub. See Table 4 for details on the applications.

iOSWorld

3.1

Environment

We model the iOSWorld environment as a partially observable Markov decision process (POMDP) (Kaelbling et al., 1998): E = (S , A, Ω, T ), where S is the set of simulator states, A is the action space, Ω is the observation space, and T : S × A → S defines deterministic 3

transitions. At each step t, the agent receives a partial observation ot ∈ Ω of state st and produces an action at ∈ A, which transitions the simulator to st+1 . Observation space. We evaluate agents under two observation modalities. Unlike Android, where tools like UIAutomator expose accessibility data, iOS is closed-source. The richest structured UI data available to third-party tools comes through Apple’s XCUITest framework, which requires a Mac running Xcode. A deployed agent without privileged information would have access only to what is visible on the screen. We evaluate both settings to separate visual grounding, reasoning, and privileged access. In the vision-only setting, the agent receives a screenshot at each step. The raw simulation captures are 1206×2622; we resize to 706×1536. This 1536-pixel cap on the longest edge stays within Anthropic’s 1568-pixel API limit for Claude Computer Use, and we apply it uniformly across all providers for a fair comparison. The agent must visually identify UI elements, estimate their coordinates, and infer the application state from pixels alone. We do not evaluate in text-only mode since all frontier computer-use models require image input. In the vision+XML setting, the agent additionally receives a cleaned accessibility tree in XML extracted via XCUITest. For each interactive element, the tree reports the element type (e.g., Button, TextField, Cell), display name, label, current value, center coordinates in a normalized 0–1000 space, and an accessibility identifier when available. The tree is filtered to interactive and visible elements, capped at 200 elements and 15 levels of depth. Action space. The available actions differ by modality and provider adapter (Table 1). In vision-only mode, agents are limited to six actions and must estimate all tap coordinates from screenshots. In vision+XML mode, action adapters expose additional tools. The most useful are tap, which targets elements by accessibility identifier, and launch app, which opens apps through their bundle identifier instead of visual home-screen navigation. Table 9 gives the exact mapping of each provider adapter. Action

Parameters

Mode

tap xy type swipe home wait stop

x, y ∈ [0, 1000] text string direction, optional origin — duration (seconds) answer string

Vision, Vision+XML Vision, Vision+XML Vision, Vision+XML Vision, Vision+XML Vision, Vision+XML Vision, Vision+XML

tap launch app terminate app open url

accessibility identifier bundle identifier bundle identifier URL string

Vision+XML only Vision+XML only Vision+XML only Vision+XML only

Table 1: Action space. The top block is available in both modalities, while the bottom block requires the accessibility tree and is exposed when supported by the model provider-specific action adapter (Table 9). The tap action targets elements by identifier, enabling pixel-perfect interaction without coordinate estimation.

Translating computer-use agents (CUAs) to iOS. Frontier computer-use models each define their own desktop action space. We adapt them to iOS with one translation layer. Click becomes tap xy, scroll becomes swipe with inverted direction, and coordinates are normalized to 0–1000. All models also receive iOS-specific system prompts for touchscreenonly interaction. For the open-source Qwen3.5 baseline, we follow the official Qwen3-VL mobile-agent cookbook. This exposes the cookbook’s mobile use tool (click, long press, swipe, type, system button, wait, terminate) on a 999×999 grid that we rescale to our 0–1000 schema. The full action mapping is in Table 9. Infrastructure. The agent loop runs on macOS using Appium with the XCUITest driver, controlling an Xcode-managed iPhone simulator. Each task starts from a deterministic 4

Task: Order dinner from Nobu on QuickBite for delivery to Home, post a deployment update in TeamChat #launch-war-room, then check #general for team announcements.

Vision-only (50 steps, fail)

Vision+XML (22 steps, pass)

Step 8

Step 15

Step 22

Step 50

Step 8

Step 12

Step 17

Step 22

QuickBite: search Nobu

Nobu menu: small tap target

Cart: $36 item added

Checkout: stuck on card toggle

QuickBite: cart ready

QuickBite: order enabled

TeamChat: posting update

TeamChat: #general checked

Score: 0.20 / 1.0 | 50 steps Never opens TeamChat stuck on payment toggle

Score: 1.0 / 1.0 | 22 steps Order placed, deployment update posted, \#general read

Figure 3: We visualize a multi-app QuickBite → TeamChat task across two modalities with the same model (Opus 4.6). Vision-only finds Nobu, adds an item, and reaches checkout, but spends the remaining budget trying to toggle payment confirmation and never opens TeamChat (50 steps, score 0.20). Vision+XML places the order, posts a deployment update in #launch-war-room, and checks #general announcements in 22 steps (score 1.0).

home state with all 26 apps pre-installed and seeded. The loop captures observations, sends them to the model, executes the returned actions, and repeats until the agent issues stop or reaches the step limit (Fig. 3 shows two same-task runs under two different modalities). Each cloned simulator instance uses 2–4 GB of RAM. Accounting for OS overhead (∼8 GB), authors found that a 36 GB Mac Studio (M4 Max) safely supports 8 parallel workers and a 24 GB MacBook Pro (M4) supports 4. Workers are managed via xcrun simctl clone, with each clone receiving dedicated Appium and WebDriverAgent ports. Evaluation. Each task is scored with an LLM-as-a-Judge (Zheng et al., 2023) framework using GPT-5.4-Mini. The judge reviews the full trajectory, including screenshots, actions, and the final answer, then returns a binary pass/fail judgment. Human validation on 128 Opus 4.6 trajectories confirms substantial agreement (κ=0.77 at task level, 89% accuracy; see §4.3). We also tested a per-step variant that evaluates screenshots independently, but it was more lenient without improving discrimination. We use the trajectory-level judge throughout and report binary pass rate as the primary metric. Details on per-step evaluation and rubric scoring are in Appendix C. 3.2

App Ecosystem and User Identity

All 26 applications share a single user identity: Jordan Avery, a San Francisco-based professional living at 410 Brannan Street who works at Northstar Studio and trains for a half marathon (Fig. 2). Jordan’s contacts, Maya Patel, Leo Chen, Kai Santos, appear as QuickChat correspondents, SplitPay payees, Mail senders, LockedIn connections, and TeamChat colleagues. A Chipotle order in QuickBite produces a charge in MyBank and a receipt in Mail. An upcoming SFO→JFK flight in SkyTrip aligns with a StayFinder booking and a Notes reminder. These cross-references make multi-app and memory tasks require evidence from more than one application. Apps were developed or adapted in SwiftUI using Claude Code as a coding assistant, then manually verified by human developers for correct navigation, data rendering, and seed data consistency. The applications implement tab-based navigation, searchable lists, detail views, and editing flows that follow their real-world counterparts. Two apps build 5

Category Single-App

Multi-App

Memory

Task ID

Task Instruction

Apps

dinespot-001

Search for restaurants in San Francisco with “Outdoor Seating” on DineSpot and make a reservation at Harborline Seafood for 2 tonight at 7 PM.

DineSpot

quickchat-003

Search my QuickChat for “Brooklyn Half” and find which conversation mentioned it. Reply to that thread confirming I’m registered.

QuickChat

multi-009

Check my most recent Chipotle order on QuickBite. Then check my MyBank credit card for the corresponding charge. Find the receipt email in Mail and note any price differences in Notes.

QuickBite, MyBank, Mail, Notes

multi-011

Check my StayFinder trip for Catalina Island (Apr 18–21). Look up the weather for those dates. Check my TasteRank “Want to Try” list for nearby restaurants and compile everything in Notes.

StayFinder, Weather, TasteRank, Notes

mem-002

Look at my CityRide app and figure out my most common route based on my saved locations. Then request a ride along that route.

CityRide

mem-005

Review my TrailBlaze activities to figure out my regular running schedule and favorite routes. Check the Weather for conditions during my typical run time and message my running group.

TrailBlaze, Weather, QuickChat

Table 2: Example tasks from each category. on open-source foundations: Notes is based on snowNotes3 and Cinephile draws from MovieSwiftUI4 . User data is encoded in Swift seed fixtures and JSON snapshots loaded at build time. The 26 apps span finance, messaging, travel, food, shopping, productivity, entertainment, fitness, utilities, and professional networking. Full details are in Table 4. 3.3

Task Design

iOSWorld includes 133 tasks in three increasingly difficult categories (Table 2). Single-app tasks (27) test basic navigation and interaction within one app, such as logging a meal in CalTrack or finding an upcoming flight in SkyTrip. Multi-app tasks (60) span two to eight applications and require transferring information between them. For example, one task asks the agent to check a Chipotle order on QuickBite, find the matching charge in MyBank, locate the receipt email in Mail, and note any price differences in Notes. Memory and personalization tasks (46) require discovering latent patterns that are never stated. The agent is asked questions like “What is my most common commute route?” or “Find my most frequently ordered restaurant and place a reorder.” Correct answers require exploration, pattern finding, and synthesis across apps. Task creation. We generated tasks using Claude Code (Anthropic, 2026a) with full access to each app’s source code and seed data. The coding agent examined seeded JSON files, view controllers, and navigation flows, then produced tasks grounded in the actual app state. Each task is written in first-person voice and accompanied by rubric criteria that decompose the objective into verifiable steps (Appendix C). Human annotators reviewed and refined every task that required changes. Quality assurance. Grounding tasks in seed data required careful verification. Human annotators manually executed every task end-to-end on the iOS simulator and verified feasibility. Forty-four of the 175 candidate tasks required corrections, including nonexistent flight routes, mismatched food names, and rubric criteria that referenced unreachable app states. All 26 apps were independently tested for UI elements, seed data, and navigation flows. The initial pool contained 175 tasks. We trimmed the single-app set for broad app coverage with minimal duplication and kept all multi-app and memory tasks, leaving a final set of 133 tasks. Memory tasks involve 4.4 apps per task on average, since answering them requires exploring several data sources. QuickChat appears in 44 of 133 tasks and Notes in 41, with CloudDocs at 35 and Mail at 29, making them the most frequently referenced apps. 3 https://github.com/probablyhades/snowNotes 4 https://github.com/Dimillian/MovieSwiftUI

Single (27)

Multi (60)

Memory (46)

Overall (133)

Model

+XML

Pass Steps

Pass

Steps

Opus 4.6

✗ ✓

70.4% 23.1 81.5% 16.4

20.0% 45.5 36.7% 38.2

8.7% 54.3%

49.4 39.3

26.3% 51.9%

42.3 34.1

Sonnet 4.6

✗ ✓

77.8% 26.3 92.6% 15.4

18.3% 46.9 35.0% 42.0

13.0% 34.8%

49.8 44.8

28.6% 46.6%

43.7 37.5

GPT-5.4

✗ ✓

63.0% 32.5 81.5% 12.1

11.7% 48.5 26.7% 37.1

6.5% 32.6%

48.3 37.2

20.3% 39.8%

45.2 32.1

GPT-5.4 Mini

✗ ✓

70.4% 24.0 66.7% 14.9

18.3% 46.3 1.7% 43.8

10.9% 4.3%

48.9 41.5

26.3% 15.8%

42.7 37.2

Gemini 3 Flash

✗ ✓

70.4% 11.0 70.4% 11.5

13.3% 23.9 18.3% 38.0

21.7% 17.4%

23.5 33.3

27.8% 28.6%

21.2 31.0

Qwen3.5 35B-A3B

✗ ✓

40.7% 23.0 48.1% 31.4

6.7% 36.2 0.0% 43.9

4.3% 2.2%

33.4 37.9

12.8% 10.5%

32.6 39.3

Table 3: Pass rates (%) and average steps by task category. Rows with ✗ are vision-only. ✓ denotes vision+XML. Rubric-based scoring in Appendix C.

Single-App (27) 100%

Pass Rate

80%

93 82 70

70 63

70 70 54

48 41

40%

37 20

20% 0%

Memory (46)

60%

Multi-App (60) Vision-only Vision+XML

13 2

Opus Sonnet GPT GPT-5.4 Gemini Qwen3.5 4.6 4.6 5.4 Mini 3 Flash 35B-A3B

18 7

9 0

Opus Sonnet GPT GPT-5.4 Gemini Qwen3.5 4.6 4.6 5.4 Mini 3 Flash 35B-A3B

17 4 2

Opus Sonnet GPT GPT-5.4 Gemini Qwen3.5 4.6 4.6 5.4 Mini 3 Flash 35B-A3B

Figure 4: Pass rates by task category and observation modality across all six models. Vision+XML (blue) outperforms vision-only (gray) for the stronger frontier models. GPT-5.4 Mini and the opensource Qwen3.5 baseline do not show the same benefit from the additional modality.

Experiments

4.1

Setup

We evaluate five frontier computer-use models: Claude Opus 4.6 and Claude Sonnet 4.6 (Anthropic, 2026b), GPT-5.4 and GPT-5.4 Mini (OpenAI, 2026), and Gemini 3 Flash (Google, 2026). Each provider offers a dedicated computer-use API with native screenshot understanding and action generation. We also include Qwen3.5 35B-A3B (Qwen Team, 2026), an open-weights mixture-of-experts model with 35B total and 3B active parameters, served via vLLM and prompted with the official Qwen3-VL mobile-agent cookbook. We test each model under both Vision-only and Vision+XML, yielding twelve configurations. All runs use a 50-step limit and screenshots capped at 1536 pixels on the longest edge. Within each modality, all models receive equivalent system prompts adapted to their action vocabularies. We use GPT-5.4 Mini as the trajectory judge. Human agreement analysis on 128 Opus 4.6 trajectories confirms substantial agreement (κ=0.77 task-level; Appendix J). 4.2

Results

Privileged vision+XML access helps the stronger frontier models (Fig. 3 and Appendix Fig. 8). Opus rises from 26% to 52% overall (+25.6 pp), Sonnet from 29% to 47% (+18.0 pp), and GPT-5.4 from 20% to 40% (+19.5 pp). Fig. 3 shows the gap on a multi-app QuickBite 7

Task: "Give me a full picture of my finances. Check MyBank balances, SplitPay pending requests, MegaMart subscriptions, FreshCart upcoming deliveries, and CloudDocs budget. Project next month s spend."

Step 3

Step 7

Step 22

Step 28

MyBank: balances pulled

SplitPay: pending requests

CloudDocs: Budget Tracker

CloudDocs: synthesis written

Result: Pass (score 1.0) | 29 steps | Memory (5 apps)

Figure 5: Successful memory trajectory (Opus 4.6, vision+XML, 29 steps, score 1.0). For “Give me a full picture of my finances,” Opus pulls balances from MyBank, checks pending requests in SplitPay, opens the Budget Tracker in CloudDocs, and writes a synthesis spanning five apps in 29 steps.

→ TeamChat task. Vision-only Opus reaches checkout but gets stuck on a small paymentconfirmation control and never opens TeamChat. Vision+XML Opus places the order, posts the deployment update, and checks #general announcements in 22 steps. With vision+XML, Sonnet reaches 93% on single-app tasks, while Opus leads on memory at 54% and multi-app at 37%. Multi-app tasks remain the hardest category. Fig. 5 traces a successful memory trajectory, where Opus pulls balances from MyBank, checks SplitPay pending requests, then synthesizes a budget projection in CloudDocs. In vision-only mode, frontier models cluster between 20% and 29%. Sonnet (29%) and Opus (26%) lead through strong single-app numbers, while Gemini at 28% is the most step-efficient at 21 steps per task versus 42–45 for Anthropic and OpenAI models. GPT-5.4 Mini and Qwen3.5 do not show the same gain from the extra accessibility-tree context, suggesting a capacity limit rather than a problem with the modality itself (see §4.3). 4.3

Analysis

Why does XML help so much? Much of the vision-to-XML gap comes from ordinary iOS friction. Dense screens make coordinates hard to estimate, app switching can derail from the home screen, the accessibility tree can expose labels that are visually small or off-screen, and iOS has no universal back button. We therefore treat XML as privileged access, not just better text input. Across the 26 Opus tasks where vision-only fails (score <0.5) and vision+XML passes, ∼70% include a home-screen or app-switching failure that launch app removes. The lift is largest on memory (Opus: 9% → 54%), where labels and values matter most. It also helps multi-app tasks once agents can launch and target apps reliably (Opus: 20% → 37%; Sonnet: 22% → 35%). Appendix K quantifies two of these iOS-specific factors directly. Smaller models struggle with the extra context. More interface information is not always useful. GPT-5.4 Mini drops from 26% vision-only to 16% vision+XML, and 22 of the 35 tasks it solves vision-only become failures under XML. This is consistent with the added ∼3,100 tokens per step exceeding its effective context budget (Fig. 10). Qwen3.5 35B-A3B degrades more sharply. XML takes it from 13% to 11% overall and from 7% to 0% on multi-app, with ∼50% of its 119 XML failures dominated by action loops (Fig. 11). With structured per-app MCP tools, pass rate rises from 12.8% to 24.8% and mean rubric score from 0.33 to 0.683 (Appendix E). 8

Task: Analyze my commuting patterns and costs. Check CityRide saved routes and Mail inbox for ride receipts. Cross-reference with MyBank transactions for total spending. Enter findings in the Budget Tracker in CloudSheets.

Step 3

Step 17

Step 24

Step 50

CityRide: recent destinations

Mail: receipt search

MyBank: transaction history

CloudSheets: data entry incomplete

Result: Score 0.45 / 1.0 | 50 steps (hit limit) | Ran out of budget before completing data entry

Figure 6: A representative budget-exhausted failure (Opus 4.6, vision+XML, 50 steps, score 0.45). Opus explores CityRide (step 3), finds Mail receipts (step 17), reaches MyBank transactions (step 24), but exhausts the 50-step budget before completing data entry in CloudSheets. Budget-exhausted runs account for 51% of frontier-model failures.

Failure taxonomy. We group the 422 frontier vision+XML failures into three modes. Budget exhausted covers full 50-step runs and accounts for 51% of failures. Gave up covers early stops with score < 0.67 and accounts for 26%. Premature stops covers early stops with score ≥ 0.67 and accounts for 23%. Budget exhaustion is most common on multi-app (55%) and memory (52%) tasks, while premature stopping is most common on single-app tasks (48%). GPT-5.4 Mini gives up on 47% of its failures. Qwen3.5 has a different profile, with ∼50% of its 119 XML failures flagged as stuck-action loops under our ≥3-identical-action heuristic (Fig. 11). Fig. 6 shows a representative budget-exhausted failure. Full breakdowns are in Appendix D. Scaling and judge validation. Step-budget curves (Fig. 9, appendix) show single-app tasks saturating by step 20, while multi-app tasks keep improving through step 40. Memory tasks vary more. Opus climbs from 17% at step 30 to 54% at step 50, whereas GPT-5.4 Mini plateaus at 16% and Qwen3.5 reaches only 11%. The trajectory judge agrees with human annotators on 128 Opus 4.6 trajectories at κ=0.77 task-level accuracy (89%, F1=0.86) and κ=0.69 on rubric criteria (Pearson r=0.85). We find that cross-judge checks do not change the conclusions. Human-agreement details are in Appendix J.

Conclusion

iOSWorld is an interactive native iOS benchmark with a persistent user identity across 26 apps. The best vision+XML configuration reaches 93% on single-app tasks, but only 37% on multi-app and 54% on memory tasks. Qwen3.5 35B-A3B trails at 11% overall. Even frontier models often run out of room, with 51% of their failures exhausting the 50-step budget. Closing this gap will require stronger loop recovery, better action and visual grounding, and planning that is aware of the user’s history. We release the code, environments, and an AWS runner at https://iosworld.io. Our environment and open-source code allow seamless addition of new tasks, personas, and apps. We believe that iOSWorld can provide a strong foundation for furthering the research on mobile agents and a shift towards emphasizing the personalization aspect of agents in deployment.

References Anthropic. Claude code. https://docs.anthropic.com/en/docs/claude-code/overview, 2026a. Anthropic. Claude opus 4.6. https://www.anthropic.com/news/claude-opus-4-6, 2026b. Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in-the-wild device-control agents with autonomous reinforcement learning. NeurIPS, 2024. Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, and Aviral Kumar. Digi-q: Learning vlm q-value functions for training device-control agents. ICLR, 2025. Rogerio Bonatti, Dan Zhao, Francesco Bonacci, Dillon Dupont, Sara Abdali, Yinheng Li, Yadong Lu, Justin Wagle, Kazuhito Koishida, Arthur Bucker, Lawrence Jang, and Zack Hui. Windows agent arena: Evaluating multi-modal os agents at scale. ICML, 2025. Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. Spa-bench: A comprehensive benchmark for smartphone agent evaluation. ICLR, 2025. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. NeurIPS, 2023. Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. Workarena: How capable are web agents at solving common knowledge work tasks? ICML, 2024. Google. Gemini 3 flash. https://blog.google/products/gemini/gemini-3-flash/, 2026. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. Webvoyager: Building an end-to-end web agent with large multimodal models. ACL, 2024. Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, and Jie Tang. Cogagent: A visual language model for gui agents. CVPR, 2024. Leslie Pack Kaelbling, Michael L. Littman, and Anthony R. Cassandra. Planning and acting in partially observable stochastic domains. Artificial Intelligence, 101:99–134, 1998. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. ACL, 2024. Quyu Kong, Xu Zhang, Zhenyu Yang, Nolan Gao, Chen Liu, Panrong Tong, Chenglin Cai, Hanzhang Zhou, Jianan Zhang, Liangyu Chen, Zhidan Liu, Steven Hoi, and Yue Wang. Mobileworld: Benchmarking autonomous mobile agents in agent-user interactive and mcp-augmented environments, 2025. Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, and Kimin Lee. B-moca: Benchmarking mobile device control agents across diverse configurations. CoLLAs, 2025. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. ICLR, 2018. Quanfeng Lu, Wenqi Shao, Zitao Liu, Lingxiao Du, Fanqing Meng, Boxuan Li, Botong Chen, Siyuan Huang, Kaipeng Zhang, and Ping Luo. Guiodyssey: A comprehensive dataset for cross-app gui navigation on mobile devices. ICCV, 2025. 10

Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: A benchmark for general ai assistants. ICLR, 2024. OpenAI. Gpt-5.4. https://developers.openai.com/api/docs/models/gpt-5.4, 2026. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326, 2025. Qwen Team. Qwen3.5-35b-a3b. https://huggingface.co/Qwen/Qwen3.5-35B-A3B, 2026. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the wild: A large-scale dataset for android device control. NeurIPS, 2023. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents. ICLR, 2025. Yilin Shen, Avik Ray, Hongxia Jin, and Sandeep Nama. Skillbot: Towards automatic skill development via user demonstration. ACL, 2019. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. ICML, 2017. Daniel Toyama, Philippe Hamel, Anita Gergely, Gheorghe Comanici, Amelia Glaese, Zafarali Ahmed, Tyler Jackson, Shibl Mourad, and Doina Precup. Androidenv: A reinforcement learning platform for android. arXiv preprint arXiv:2105.13231, 2021. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-agent: Autonomous multi-modal mobile device agent with visual perception. ICLR, 2024. Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. MobiCom, 2024. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. NeurIPS, 2024. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking llm agents on consequential real world tasks. NeurIPS, 2025a. Yifan Xu, Xiao Liu, Xueqiao Sun, Siyi Cheng, Hao Yu, Hanyu Lai, Shudan Zhang, Dan Zhang, Jie Tang, and Yuxiao Dong. Androidlab: Training and systematic benchmarking of android autonomous agents. ACL, 2025b. Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su. An illusion of progress? assessing the current state of web agents. COLM, 2025. Pei Yang, Hai Ci, and Mike Zheng Shou. macosworld: A multilingual interactive benchmark for gui agents. NeurIPS, 2025. 11

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. NeurIPS, 2022. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. ICLR, 2025. Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui: Grounded mobile ui understanding with multimodal llms. ECCV, 2024. Chi Zhang, Zhao Yang, Jiaxuan Liu, Yucheng Han, Xin Chen, Zebiao Huang, Bin Fu, and Gang Yu. Appagent: Multimodal agents as smartphone users. CHI, 2025. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. NeurIPS, 2023. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents. ICLR, 2024.

Ethics Statement Synthetic data. All data in iOSWorld is entirely synthetic. The Jordan Avery persona is fictional, and no real user data was collected, processed, or used at any stage. Benchmark runs use deterministic seeded data and do not depend on real user accounts, real services, or external databases. We chose this design specifically to enable research on personalization and memory tasks without the privacy risks inherent in real user data. Malicious Agents. Phone agents capable of operating autonomously on a user’s device carry significant dual-use risks. An agent with access to personal messaging, banking, ridehailing, and email apps could be misused for surveillance, unauthorized transactions, social engineering, or data exfiltration. Even well-intentioned agents can cause harm through errors, such as sending a message to the wrong contact, making an unintended purchase, or leaking personal information across apps. The personalization and memory tasks in iOSWorld are particularly sensitive because they require agents to reason about personal data, which is the most damaging if mishandled. We encourage researchers to develop agents with explicit user consent mechanisms and action confirmation for irreversible operations. iOS access and reproducibility. iOSWorld requires macOS with Xcode to run the iOS Simulator, which limits reproducibility to researchers with access to Apple hardware. We release all source code, seed data, and evaluation scripts. We also release an AWS-runner deployment (EC2-managed Mac instances) so non-Mac researchers can submit task batches and receive the same evaluation bundle. The closed-source nature of iOS means the vision+XML modality relies on XCUITest, a developer tool unavailable to a deployed consumer agent. Vision-only numbers reflect deployed capability, while vision+XML represents an upper bound with privileged access. Single-persona scope. The release uses one fictional user (Jordan Avery) to keep personalization tasks ground-truth-verifiable. The persona-seeding pipeline, schema, task generator, and rubric framework are released so contributors can generate a comparable task suite for a new persona. Multi-persona evaluation is left to future work. Accessibility and intended use. Capable phone agents could improve accessibility for users with visual, motor, or cognitive impairments who find complex multi-step workflows difficult. iOSWorld is a research benchmark for measuring progress in a controlled simulator. Results should not be read as readiness for deployment on real devices with real user data. 12

App

Analogue

Key Seed Data and Features

MyBank

Chase

SplitPay

Venmo

QuickChat TeamChat Mail SkyTrip

WhatsApp Slack Mail Delta

CityRide StayFinder

Uber Airbnb

QuickBite

DoorDash

FreshCart MegaMart TasteRank DineSpot CalTrack CloudDocs

Instacart Amazon Beli OpenTable MyFitnessPal Google Docs

CloudDrive CloudSheets CloudSlides Notes

Google Drive Google Sheets Google Slides Apple Notes

Cinephile TicketBox TrailBlaze

Letterboxd SeatGeek Strava

ScoreZone Weather Clock LockedIn

ESPN Weather Clock LinkedIn

Checking $15,621; Savings $4,920; Credit $4,589/$8,000; 611 ledger transactions 17 core users, 24 suggestions, 80 transactions, $650/mo rent from Arnav, 6 pending 54 contacts, 40 conversations (26 DMs + 14 groups) 25 members, 11 channels (#eng-mobile, #launch-war-room), 8 DMs 84 inbox, 10 sent, 1 draft, 80 archive; senders span all other apps Gold Medallion, 127K miles, 5 upcoming trips (SFO→JFK, SEA, ORD, HNL, LHR) Home/Office saved, 79 trips, 20 drivers, 5 ride types Upcoming: Catalina Island, Barcelona sail. Past: Big Sur, Lisbon, London 82 restaurants, 5 saved addresses, 30+ past orders (Chipotle, Sweetgreen) 13 stores, 112 products, 35 orders, scheduled/active/delivered Prime member, 10 depts, 17 cart items, 51 saved items, 42 orders 60 restaurants, 8 saved lists (Date Night, Seoul Guide, etc.) 74 restaurants, 7 cities, outdoor/tasting/vegetarian filters 77+ foods, 2,450 cal goal, 170g protein target, weight 182 lbs Mobile Launch Plan, Compute Access Runbook, Agent Systems Review Product Strategy, Shared Assets, Archive folders Budget Tracker, Experiment Results (multi-sheet) Conference Practice Deck, Agent Systems Talk 4 folders, 14 notes, 3 pinned (Shopping List, Wifi Passwords, Team Standup) Wishlists, Seenlist, custom lists, fan clubs 17 venues, 66 events, Taylor Swift Eras Tour, Hamilton Bundled snapshot with 4 athletes, 3 routes, 6 activities, PRs (5K: 19:42), 2 clubs Favorites: Lakers, Chiefs, Yankees; 40 seeded teams, scores SF, NYC, Dallas; hourly and 10-day forecasts Alarms, world clocks (Tokyo, London), stopwatch, timer Connections, job postings, professional profile

Table 4: The 26 iOS applications in iOSWorld with real-world analogues and seed data.

LLM Disclosure

We used large language models in several stages of this work. All drafting and structural decisions were made by human authors. Claude Code was used to polish prose, check grammar, and verify consistency with human-in-the-loop review. Figures and plots were generated programmatically via Claude Code from human-provided sketches, with the human author directing layout and content at every iteration. We used a multimodal LLM coding agent to perform high-level quantitative analysis, such as aggregating scores, computing pass rates, and to flag qualitative patterns in agent trajectories (e.g., identifying failure modes from screenshots). All flagged results were reviewed, verified, and synthesized into written analysis by human authors. The 26 iOS applications were built in SwiftUI using Claude Code as a coding assistant with human developers verifying correctness, and tasks and rubrics were generated by Claude Code then reviewed, refined, and manually executed end-to-end by human annotators. Finally, we use GPT-5.4 Mini as an LLM-as-a-judge evaluator, validated against human annotators (κ=0.77).

Application Details and Dataset Statistics

Table 4 lists all 26 applications. QuickChat (44 task references), Notes (41), CloudDocs (35), and Mail (29) are the most frequently involved apps. 13

App CalTrack Mail MyBank ScoreZone TeamChat QuickBite MegaMart StayFinder CloudDocs CityRide TrailBlaze TicketBox Notes LockedIn SplitPay Clock CloudSlides SkyTrip FreshCart TasteRank Weather CloudSheets QuickChat DineSpot CloudDrive Cinephile

Pass

Total

Rate

11 17 13 5 12 9 9 5 17 8 7 7 17 4 9 6 3 4 5 4 6 5 9 3 1 1

17 29 22 9 22 17 17 10 35 17 15 16 41 10 23 16 8 11 15 12 21 18 44 17 7 8

65% 59% 59% 56% 55% 53% 53% 50% 49% 47% 47% 44% 41% 40% 39% 38% 38% 36% 33% 33% 29% 28% 20% 18% 14% 12%

Table 5: Per-app pass rate for Opus 4.6 (vision+XML) across all 26 apps. Per-app difficulty. Table 5 shows pass rate for Opus 4.6 (vision+XML) across all 26 apps. Cinephile is hardest (12%, 8 references), followed by CloudDrive (14%, 7 references), while CalTrack is easiest (65%, 17 references). Mail (59%, 29 references) and MyBank (59%, 22 references) are also among the strongest. QuickChat remains challenging (20%, 44 references) due to precise thread navigation across many tasks.

Rubric-Based Evaluation Details

Each task is accompanied by a rubric, a list of independently verifiable criteria decomposing the objective into steps. The benchmark contains 1,123 rubric items across 133 tasks, ranging from 4 to 13 per task (mean 8.4). Multi-app tasks are the most rubric-dense at 9.4 items on average, reflecting the number of intermediate steps needed to coordinate across applications. Rubric scores reveal partial progress. Binary pass rates understate agent capability. Under vision+XML, the average rubric score (fraction of criteria satisfied) ranges from 29% (Qwen3.5) to 81% (Opus), meaning frontier agents satisfy a majority of criteria even on tasks they ultimately fail. The rubric perfect rate (all criteria satisfied) tracks the binary pass rate within about 0.8–2.3 percentage points, confirming internal consistency between the holistic judge and per-criterion evaluation. Per-step evaluation. We also evaluated a per-step variant in which the judge reviews each screenshot independently and we take the maximum across steps per criterion. It yields higher rubric scores (73–87% average) but proved more lenient than the trajectory judge when validated against human annotators (κ=0.51–0.61 vs. 0.77 for the trajectory judge). Its mean rubric score is 0.83 versus 0.70 for humans, producing more than twice as many false-positive criteria (188 vs. 79). Since iOSWorld tasks are relatively straightforward and compact (mean 8.4 criteria, max 50 steps), the trajectory judge provides sufficient 14

Category (%) Failure Mode Budget exhausted Gave up Premature stop

Single

Multi

Mem

All

14 38 48

55 21 24

52 31 18

51 26 23

Table 6: Failure mode distribution under vision+XML across the five frontier models. Qwen3.5 35B-A3B

Pass rate

Rubric score

Cookbook mobile use (7 actions) Structured per-app tools (MCP)

12.8% 24.8%

0.33 0.683

∆

+12.0 pp

+0.353

Table 7: Qwen3.5 with and without MCP tools over the same 133-task suite.

discrimination without the added complexity of per-step evaluation. We use the trajectorylevel judge throughout the main text.

Failure Analysis Details

Methodology. Each failed trajectory is assigned to exactly one of three mutually exclusive modes, derived from its step count and final rubric score. A run that stopped before the 50-step limit is a premature stop if its rubric score is ≥ 0.67 (it ended a largely correct trajectory too early) and gave up otherwise. A run that used all 50 steps is budget exhausted. We do not split budget-exhausted frontier runs into loop failures versus continued effort, because the two are hard to separate without human review. An automatic heuristic (≥3 consecutive near-identical actions) over-flags benign repeated scrolls and near-coincident taps. For the open-source Qwen3.5 baseline the same heuristic is reliable, because its loops are blatant (e.g., 38 consecutive identical swipes; Fig. 11). We report that loop share separately below. Examples. In a budget exhausted case, Opus hits the 50-step limit on a commuting-patterns memory task with partial progress across CityRide, Mail, MyBank, and CloudSheets (mem021, score 0.45; Fig. 6). In a premature stop, GPT-5.4 reports fare and ETA correctly but stops at the final “Request” button after 8 steps without confirming the booking (cityride-001, score 0.80). In a gave up case, Opus abandons a game-day planning task after 44 steps with only partial progress across ScoreZone, QuickChat, and SplitPay (mem-020, score 0.65). Open-source baseline (Qwen3.5 35B-A3B). 119/133 vision+XML failures (10.5% pass rate). The ≥3-consecutive-identical-actions loop heuristic is reliable here because Qwen’s loops are extreme: 50% of its XML failures (60/119) are flagged as stuck-action loops.

MCP Tools Ablation

The main results attribute much of the vision-only/vision+XML gap to interface rather than reasoning. Precise element targeting and launch app remove failure modes introduced by coordinate estimation (§4.3). We test that interpretation on the open-source baseline by holding the model, task set, judge, and 50-step budget fixed while varying only the action interface. Screenshots remain available, but the agent receives a structured per-app tool layer instead of the 7-action cookbook mobile use tool. The MCP server exposes typed, app-specific operations such as caltrack.log food and mybank.send zelle over the same 26 apps. We release the MCP server alongside the benchmark so others can run the same comparison. 15

Task: search DineSpot for outdoor-seating restaurants in San Francisco and make a reservation at Harborline Seafood. Model: Qwen3.5 35B-A3B, vision-only.

Structured MCP tools

17 steps, score 1.0 (pass)

Step 1

Step 3

Step 8

Step 17

dinespot.launch()

apply_feature_filter (outdoor)

make_reservation()

Reservation confirmed

Cookbook mobile_use

50 steps, score 0.25 (fail)

Step 5

Step 15

Step 30

Step 50

Filter menu opens

Same filter tap, no change

Still on filter sheet

Budget out; no reservation

Figure 7: dinespot-001, Qwen3.5 vision-only. Top, structured MCP tools (17 steps, score 1.0). Typed calls take the agent directly to a confirmed Harborline Seafood booking. Bottom, cookbook mobile use (50 steps, score 0.25). The same model gets stuck on the filter sheet and never makes a reservation. Action

Opus

Sonnet

GPT-5.4

Mini

Gemini

Qwen3.5

tap xy swipe type launch app wait home

62% 21% 9% – 1% 7%

62% 23% 8% – 1% 7%

55% 12% 18% – 9% 7%

37% 35% 21% – 3% 3%

54% 10% 17% 18% <1% <1%

66% 13% 4% 1% 1% 16%

Table 8: Action type distribution per model under vision+XML. Result. Structured tools raise Qwen3.5’s pass rate from 12.8% to 24.8% and its mean rubric score from 0.33 to 0.683, but the baseline still trails the frontier models.

Additional Results Figures

Capacity-sufficient frontier models

Smaller-capacity (context-overflow regime)

Overall Pass Rate

60%

+25.6pp

+18.0pp

50%

+19.5pp

40%

+0.8pp

30%

-10.5pp

20%

-2.3pp

10% 0%

Opus 4.6

Sonnet 4.6

GPT 5.4

Gemini 3 Flash

Vision-only (CUA)

GPT-5.4 Mini

Qwen3.5 35B-A3B

Vision+XML

Figure 8: Vision-only vs. vision+XML accuracy per model. Privileged vision+XML access improves the stronger frontier models (Opus +25.6 pp, Sonnet +18.0 pp, GPT-5.4 +19.5 pp, Gemini +0.8 pp). Smaller models (GPT-5.4 Mini, Qwen3.5 35B-A3B) do not benefit from the additional accessibility-tree input.

Overall (133) 52%

50%

45%

Cumulative Pass Rate

40%

Single-App (27)

100% 80% 60% 40% 20% 0%

40%

Multi-App (60)

40% 30%

30%

29%

20% 10% 0%

20%

Memory (46)

16% 11%

10%

60% 50% 40% 30% 20% 10% 0%

15 Opus 4.6

Step Budget Sonnet 4.6

35 GPT-5.4

GPT-5.4 Mini

Gemini 3 Flash

10 Qwen3.5 35B-A3B

Step Budget Vision+XML

Vision-only

Figure 9: Cumulative pass rate vs. step budget. Left: overall. Right: by task category. Solid: vision+XML; dashed: vision-only. Single-app saturates by step 20; multi-app scales through step 40; memory shows varied scaling with Opus climbing steeply past step 30.

Task: Search for restaurants in San Francisco with "Outdoor Seating" on DineSpot and make a reservation at Harborline Seafood.

Vision-only (24 steps, pass)

Vision+XML (37 steps, fail)

Step 8

Step 16

Step 24

Step 5

Step 30

Step 37

DineSpot home screen

Harborline Seafood time slots

Reservation confirmed

DineSpot home screen

Filters modal still not toggled

Harborline found but forgot the goal

Score: 1.0 / 1.0 | 24 steps Clean navigation reservation confirmed

Score: 0.4 / 1.0 | 37 steps Looped through filters, forgot original goal

Figure 10: GPT-5.4 Mini on the same DineSpot reservation task under both modalities. Left: visiononly navigates cleanly to a confirmed booking in 24 steps (score 1.0). Right: vision+XML loops through filter menus for 30 steps and ultimately forgets the original goal (score 0.4, 37 steps). The accessibility tree overwhelms the smaller model’s context capacity.

Task: "Set a new alarm for 6:45 AM labeled 'Gym' in the Clock app and confirm it's set." (Qwen3.5 35B-A3B, vision+XML)

Step 2

Step 5

Step 25

Step 50

Clock: launched

Add Alarm: time picker open

Step 25: still swiping

Step 50: never saved

Result: Fail (score 0.4) | 50 steps | 38 consecutive identical swipe-down actions on time picker Figure 11: Qwen3.5 35B-A3B (vision+XML) on a simple “Set a 6:45 AM alarm labeled Gym” singleapp task. The agent reaches the Add Alarm screen by step 5 but then issues the same swipe-down action on the time picker 38 consecutive times (steps 6–46), never adjusting to 6:45, never setting the label, and never tapping Save. The 50-step budget is exhausted on a task Opus and Sonnet both solve in 25 steps. Repeated-action loops account for ∼50% of Qwen3.5’s 119 XML failures.

App and Task Construction

App creation. Apps were created or adapted using Claude Code with a structured prompt specifying constraints (SwiftUI, deterministic seeded data, accessibility identifiers), workflows, data models, and seed quantities. Apps underwent iterative refinement and manual verification by human developers. Task pipeline. Stage 1: Claude Code generated tasks grounded in app source code and seed data. Stage 2: A Python pipeline normalized app names, rewrote tasks in first-person voice, and generated rubric criteria. Stage 3: Human annotators executed every task on the simulator.

Prompts

Agent system prompt. All models receive the following iOS-specific instructions. Action names are adapted per provider (e.g., left click for Claude, click for OpenAI, click at for Gemini; see Tab. 9). The version below is for Claude CU; the vision+XML variant appends the accessibility tree instructions at the end. You are controlling an iOS Simulator (iPhone). This is a touch-screen mobile phone with NO mouse cursor, NO physical keyboard shortcuts, and NO right-click. CURRENT STATE: You start on the iOS home screen. You must find and open apps yourself. HOW TO OPEN APPS: - Tap an app icon on the home screen if it is visible. - To search for an app: swipe DOWN from the MIDDLE of the home screen to open Spotlight search, then type the app name and tap the result. - Swipe left/right on the home screen to browse additional pages of apps. TOUCH INTERACTIONS: - Use ‘left click’ for all touch/tap interactions (there is no mouse cursor). - To type text, click a text field first, then use the ‘type’ action to enter text. - To scroll content, use the ‘scroll’ action with delta x/delta y. iOS-SPECIFIC BEHAVIOURS: - HOME: Use key ‘Home’, or swipe up from the very bottom of the screen. - APP SWITCHER: Swipe up from the bottom and pause mid-screen. - BACK NAVIGATION: Look for a back button (top-left) or swipe from the left edge. - KEYBOARD DISMISS: Tap any area outside the text field. COMPLETING THE TASK: - When the task is complete, stop calling the computer tool and respond with a text summary. - IMPORTANT: If the task asks you to find, check, look up, or report ANY information, you MUST include that exact information in your final text response. ACCESSIBILITY TREE (appended in vision+XML mode only): On each turn you will also receive a text accessibility tree of the current UI. The tree lists every element with its type, name/label, value, accessibility IDs (shown as id="..."), and centre coordinates. IMPORTANT: The coordinates in the tree are in the SAME coordinate space as your action coordinates. You can use tree coordinates DIRECTLY as click/tap targets without any conversion or mapping. How to use the two inputs together: - The SCREENSHOT is ground truth for what is displayed on screen. - The TREE provides precise element names and coordinates for targeting. - Use tree coordinates DIRECTLY to click more precisely than visual estimation. - If the screenshot and tree disagree (e.g. an element appears in the tree

but not on screen), trust the screenshot; the element may be off-screen or obscured. - Elements marked [hidden] are in the DOM but not rendered on screen. - If you need to find elements not currently visible, try scrolling.

Trajectory-level evaluation. The judge (GPT-5.4 Mini) receives the following prompt structure. Per-step screenshots are attached as images: Goal: [task instruction] You are evaluating whether an iOS agent successfully completed the above goal. The agent executed N steps. Full trajectory with per-step screenshots: Step 1: [Screenshot 1 - before actions] Actions: [JSON] Step 2: ... [Screenshot N+1 - final state after all actions] The agent’s final answer was: "[answer]" N screenshots are attached, one per step showing the screen state before each action, plus one final screenshot showing the end state. Evaluate each of the following rubric criteria individually: 1. [criterion] 2. ... Respond with ONLY a JSON object (no code fences): {"success": true/false, "reasoning": "overall assessment", "rubric results": [{"criterion": "...", "satisfied": true/false, "reasoning": "..."}]} success=true means the goal is fully and completely achieved. For each rubric criterion, set satisfied=true only if there is clear evidence in the trajectory and screenshots that the criterion was met. For tasks that ask a question, evaluate whether the agent’s final answer is correct.

Per-step evaluation. Each step is evaluated independently in a separate LLM call. The judge receives one screenshot, the agent’s action, and its reasoning. For the final step, the agent’s answer is also included: Goal: [task instruction] You are evaluating an iOS agent’s progress at step K. The attached screenshot shows the device screen after the agent’s action. Agent’s action: [JSON] Agent’s reasoning: [text] Agent’s final answer: [text] (last step only) Which of the following rubric criteria are NOW satisfied based on the screenshot, the agent’s action, and its reasoning? 1. [criterion] 2. ... Respond with ONLY a JSON object: {"satisfied": [list of criterion numbers that are satisfied]} Return an empty list if none are satisfied. Only mark a criterion satisfied if there is clear evidence from the screenshot AND the agent’s actions/reasoning. Do not infer satisfaction from ambiguous or partial evidence.

All steps are evaluated in parallel (up to 8 concurrent calls per task), and we take the max across steps per criterion. Once a criterion is satisfied at any step, it remains satisfied. CUA action translation. Each provider uses its own action vocabulary. We map all native actions to our unified iOS action schema (Table 9):

Example Trajectories

iOS Action

Claude CU

OpenAI CUA

Gemini CU

Qwen mobile use

tap xy tap xy ×2 tap xy ×3 type type (keys) swipe swipe swipe home wait launch app hover open url stop

left click double click triple click type key scroll – left click drag key Home wait – – – (text)

click double click triple click type keypress scroll – drag keypress Home wait – – – (text)

click at – – type text at key combination scroll at scroll document drag and drop go home wait 5 seconds open app long press at open url (text)

click – – type system button swipe – – system button Home wait – long press – terminate / answer

Table 9: Action translation from provider-native vocabularies to our iOS action schema. Scroll direction is inverted for all providers (scroll down = swipe up on touchscreen). Claude and OpenAI output pixel coordinates scaled to 0–1000. Gemini outputs 0–999 directly, and Qwen mobile use outputs 0–999 under the cookbook contract. Actions marked “–” are not available for that provider. Qwen mobile use follows the official Qwen3-VL mobile-agent cookbook tool schema.

Task: "Open the Work folder in Notes, find the 'Team Standup Notes' note, and add a new bullet point about a bug fix completed today. Confirm the note was updated."

Step 3

Step 6

Step 10

Step 13

Spotlight: search Notes

Notes: Work folder

Editing: add bullet

Note updated with bug fix

Result: Pass (score 1.0) | 13 steps | Single-App Figure 12: Successful single-app Notes task. Team Standup Notes, add bug-fix bullet (Sonnet 4.6 CUA, 13 steps, score 1.0).

Task: "Find a restaurant in SF with outdoor seating on DineSpot and make a reservation for 6 at 7 PM Friday. Post details in TeamChat #general."

Step 2

Step 8

Step 18

Step 26

Home Screen: page 2

Harborline: restaurant detail

DineSpot: reservation confirmed

TeamChat: posted details

Result: Pass (score 1.0) | 26 steps | Multi-App Figure 13: Successful multi-app DineSpot → TeamChat task (Opus 4.6, vision+XML, 26 steps).

Task: "Check my SkyTrip departure time, request a CityRide Black to SFO, set an alarm 30 min before departure, and message Maya Patel in QuickChat with details."

Step 3

Step 22

Step 46

Step 50

SkyTrip: flight details

CityRide: ride booked

Clock: alarm set

QuickChat: budget exhausted

Result: Fail (score 0.56) | 50 steps | Completed 3/4 subtasks but ran out of budget before messaging Figure 14: Failed multi-app SkyTrip → CityRide → Clock → QuickChat task. The agent completed 3/4 subtasks but ran out of budget before messaging (50 steps, score 0.56).

Task: "Figure out whose birthday is coming up by checking Notes, QuickChat, and Mail. Find a gift on MegaMart within budget and make a DineSpot reservation."

Step 5

Step 11

Step 26

Step 50

Notes: birthday info

QuickChat: family discussion

MegaMart: gift found

DineSpot: waitlisted, incomplete

Result: Fail (score 0.50) | 50 steps | Found info and bought gift, but ran out of budget for dinner reservation Figure 15: Failed memory Notes → QuickChat → MegaMart → DineSpot task. The agent found birthday info and bought a gift but ran out of budget before the dinner reservation (50 steps, score 0.50).

Comparison

Cohen’s κ

Acc.

Human vs. Trajectory Judge Binary task success Rubric criteria (n=1,094)

0.77 0.69

0.86 0.90

0.89 0.86

0.77 0.69

Human vs. Per-Step Judge Binary task success Rubric criteria (n=1,094)

0.61 0.51

0.79 0.87

0.80 0.81

0.63 0.56

Trajectory Judge vs. Per-Step Judge Binary task success 0.55 Rubric criteria (n=1,094) 0.55

0.75 0.88

0.77 0.83

0.57 0.59

Table 10: Extended agreement between human annotators and automatic judges on 128 trajectories (1,094 rubric criteria), including judge-vs-judge comparison. ρ denotes Spearman correlation.

Human Agreement

To validate the automated evaluation pipeline, we collect human annotations on 128 trajectories from the Opus 4.6 vision+XML configuration, spanning all three task categories. Four annotators each reviewed a subset of trajectories and graded every rubric criterion as pass or fail, along with an overall binary success judgment. We compare these human judgments against two automated judges. The trajectory-level judge sees the full action trace. The per-step judge evaluates each screenshot independently and takes the max across steps. Table 10 reports task-level and rubric-criterion agreement, plus the per-step judge comparison. The trajectory-level LLM judge achieves substantial agreement with human annotators across all metrics. At the task level, binary success judgments agree 89% of the time (κ=0.77, F1=0.86). At the rubric-criterion level, the judge correctly classifies 86% of individual criteria (κ=0.69, F1=0.90). The rubric-level κ is lower despite comparable accuracy because Cohen’s κ is sensitive to marginal distributions. Since 67% of rubric criteria are satisfied, expected chance agreement is inflated and κ is mechanically lower. Accuracy and F1 are more directly interpretable here. Continuous rubric scores are highly correlated, with Pearson r=0.85 and Spearman ρ=0.86 between human and trajectory-judge rubric fractions. Mean absolute error is 0.10. The per-step parallel judge shows lower agreement (κ=0.51–0.61), mainly because it is more lenient. Its mean rubric score is 0.83 versus 0.70 for humans, producing 188 false-positive criteria compared to 79 for the trajectory judge. This matches the design difference. Per-step evaluation marks a criterion satisfied if any single screenshot shows evidence, which can overcount partial progress. At the criterion level, the 148 disagreements between humans and the trajectory judge split into 79 false positives (LLM too lenient) and 69 false negatives (LLM too strict), indicating no strong bias in either direction. Per-annotator analysis. Table 11 breaks down agreement by annotator. Four annotators each graded 26–47 trajectories. Task-level κ ranges from 0.64 to 0.92, while rubric-level κ is more tightly clustered (0.67–0.72), suggesting that per-criterion judgments are more consistent across annotators than holistic task-level judgments. The annotator with the lowest task-level κ (0.64, 47 tasks) has the highest rubric-level κ (0.72). This suggests that task-level disagreements come from borderline cases where most but not all criteria are satisfied, rather than from fundamentally different rubric interpretations. Human pass rates are consistent with the LLM judge across all annotators (36–50% human vs. 36–46% LLM), with no annotator showing a clear leniency or strictness bias.

Annotator A B C D

Tasks

κtask

Acc.

κrubric

Criteria

47 28 27 26

0.64 0.84 0.92 0.77

0.83 0.93 0.96 0.88

0.72 0.67 0.67 0.69

397 248 232 217

Table 11: Per-annotator agreement with the trajectory judge. κtask : Cohen’s kappa on binary task success. κrubric : Cohen’s kappa on individual rubric criteria. All annotators show substantial agreement (κ ≥ 0.64). Judge model

Provider

κ vs. human

Pass rate

Gemini 3 Flash GPT-5.4 Mini (published) GPT-5 mini Claude Opus 4.6 Claude Sonnet 4.6 GPT-5.4 (full)

Google OpenAI OpenAI Anthropic Anthropic OpenAI

0.79 0.77 0.72 0.68 0.67 0.51

35.2% 39.1% 35.2% 30.5% 35.9% 20.3%

Table 12: Cross-judge agreement with human annotators on the 128 validated Opus 4.6 vision+XML trajectories. Larger or in-family judges do not improve κ; GPT-5.4 (full) is an outlier due to systematic over-rejection. Cross-judge robustness. We re-scored all 128 validated Opus 4.6 vision+XML trajectories with five alternate judges (Table 12). The larger GPT-5.4 is the worst judge at κ=0.51 and over-rejects (1 FP vs. 27 FN against human, opposite of every other judge). Pairwise judge agreement sits in [0.74, 0.90] except GPT-5.4, which sits in [0.53, 0.74] against every other judge. Where the judge makes mistakes. At the task level, disagreements are directiondependent by category (Table 13). The judge over-accepts on single-app (3 FP, 0 FN), over-rejects on multi-app (0 FP, 5 FN), and is balanced on memory. At the criterion level (Table 14), the judge is perfectly reliable on observable atomic actions (taps and swipes at 0% error) and least reliable on semantic and report criteria such as “did the agent correctly summarize X?” (13 to 16% error). These are also the criteria where humans have the most interpretive, subjective room. Generalization beyond Opus. We collected human annotations on 64 additional trajectories (32 Gemini 3 Flash and 32 GPT-5.4 Mini) through the same web tool (Table 15). Agreement stays moderate to substantial on both new families (κ=0.49 and 0.60). The lower Gemini κ partly reflects smaller N (32 vs. 128) and the lack of the multi-annotator calibration used on the original set. Same-family bias check. The published judge shares a provider with two evaluated agents (GPT-5.4 and GPT-5.4 Mini). We ran the out-of-family Gemini judge on the subset of GPT trajectories available for this bias audit (Table 16). This table is a cross-judge rescoring check, not the release aggregate in Table 3. Gemini passes GPT trajectories at a comparable or higher rate than the in-family published judge in every audited cell (mean delta +3.0 pp), which is opposite of what in-family inflation would predict. On the 32 human-validated GPT-5.4 Mini trajectories, Gemini also agrees with humans slightly better than the in-family judge (κ=0.66 vs. 0.60). Bootstrap stability. A task-level bootstrap (5,000 resamples) on the Opus 4.6 vision+XML cell gives 95% CIs of ±8.3 pp Overall (mean 51.9%), ±11.7 pp Multi-app, ±14.1 pp Memory, and ±14.8 pp Single-app. The wider per-category CIs on the smaller subsets make multi-app and memory the more discriminative axes. Aggregate model ordering (Opus best Overall, Sonnet best Single-app, Gemini most step-efficient) is stable across resamples. 25