arXiv CSopen access

RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents

· arxiv_cs

arXiv CS · Papers · License: Open Access

software-architecture software-engineering testing

software engineering, software architecture, testing

2026-09-21

R ECREATION W ORLD: Scalable and Verifiable Environments for Hybrid Computer-Use Agents Alibaba Token Hub, Alibaba Group Website

GitHub

Hugging Face

ModelScope

Computer-use agents (CUAs) have advanced along two largely separate lines—operating applications through graphical interfaces, and building software through code and the command line—each blind to what the other does best: a GUI agent cannot construct the software behind an interface, while a terminal agent cannot see the interface its own actions produce. Real digital work demands both at once, interleaved rather than stacked end to end. We study hybrid CUAs that fuse the two: agents that autonomously decide when to explore an interface, when to implement, and when to run and visually verify their own artifacts. Building this capability requires environments that both evaluate it under control and generate verified experience at scale. We introduce R ECREATION W ORLD, a five-platform framework built around recreation: given a running reference, an agent must discover its behavior and build a faithful implementation with no prescribed workflow. Within this framework, recreation instantiates the hybrid loop in its purest form and supplies an objective, execution-grounded reward, because the running reference acts as an oracle from which hidden behavioral tests can be derived. To execute this loop across platforms, R ECREATION W ORLD provides reproducible environments on Ubuntu, macOS, Windows, Android, and Web, plus a unified harness with native GUI control and coding tools. We further scale up with high-quality open-source applications, using them to generate long-horizon recreation trajectories for training. Models trained on these trajectories improve across five out-of-distribution benchmarks spanning coding and hybrid computer use and more frequently verify their own rendered outputs—evidence that recreation builds hybrid-agent capabilities that transfer beyond recreation and support self-improvement. For held-out evaluation, we introduce R ECRE ATION B ENCH, which comprises 250 diverse tasks across different domains and platforms. At the evaluation layer, reference-grounded test generation converts executable behavior into implementation-agnostic supervision; programmatic and visual assertions cover action-conditioned outcomes at multiple interaction depths, and each is validated on the reference and by human reviewers before the suite is frozen for automatic scoring. On R ECREATION B ENCH, GPT-6 Astra leads at 58.1% overall, but passes all programmatic tests on just 2.8% of tasks. Our analysis shows that agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications remain substantially smaller and more monolithic than their references. We release the benchmark, environments, and test suites. Diverse executable tasks

Scalable verified experience Qwen3.7-Plus Qwen-Flash-CPT

Ubuntu

macOS

RecreationBench Programmatic

Visual

GPT-6 Astra Codex / max

Claude Opus 5

Claude Code / max

+12.1

Windows

GPT-5.6 Sol Codex / max

Grok 4.6

Gain (pp)

arXiv:2609.22000v1 [cs.CL] 18 Sep 2026

Abstract

Claude Code / xhigh

Qwen3.8-Max-0902 +5.8

Claude Code / xhigh

Kimi K3 Claude Code / max

Claude Opus 4.8 Claude Code / max

GLM-5.3 0

Android

Claude Code / max

Gemini 3.7 Flash Claude Code / high

Web

Qwen3.7-Plus Claude Code / enabled

−5

running reference → runnable recreation

Training progress (%)

100

Five-platform mean score (%)

Figure 1: Overview of R ECREATION W ORLD. Left: five-platform recreation tasks. Center: transfer to five out-of-distribution benchmarks, shown individually (pale) and on average (dark) relative to each model’s first checkpoint. Right: five-platform programmatic and visual benchmark scores.

Running Reference DESKTOP ANDROID

WEB

Hybrid Agent explore

implement

Candidate Delivery

verify workspace

iterate

src/ build script $ build

five platforms

reference app isolated sandbox live specification

source code / build scripts CLI

GUI

Hidden Evaluation functional

visual

UI assertions

VLM assertions

test.py

recreated app

open_menu() assert state assert value

Menu is open …

XML view is shown

Figure 2: Application recreation in R ECREATION W ORLD. Across five platforms, a hybrid agent autonomously interleaves GUI exploration, implementation, and verification to reconstruct a running reference; hidden programmatic and visual assertions score the delivered candidate.

Introduction

General-purpose computer-use agents (CUAs) have advanced along two largely separate lines. One line operates real applications through their graphical interfaces, perceiving screens and issuing clicks and keyboard inputs (Xie et al., 2024; Yuan et al., 2026; Lu et al., 2026); the other works through code and the command line—writing and running programs, invoking build, test, and shell tools, and driving the system through a terminal (Yang et al., 2026a; Jimenez et al., 2024). Each line is blind to what the other does best. A GUI-only agent can observe how a running system behaves but rarely builds or reconfigures the software behind it; a terminal-only agent can author programs and orchestrate tools but cannot see the interface its actions produce, and so has no exploratory visual feedback with which to judge whether the running result matches what was intended. Yet many software-intensive tasks demand both at once: understanding a system requires operating it, while acting on that understanding requires constructing or reconfiguring software. Crucially, they cannot be treated as two stages completed once in sequence. Information must flow back and forth—observations shape the implementation, and running the implementation reshapes what to observe next—so the capability is a genuine fusion rather than a GUI agent and a terminal agent stacked end to end. Existing benchmarks still tend to emphasize one interaction mode, leaving an agent’s autonomous coordination of this loop over long horizons under-measured. We refer to agents that sustain this fusion as hybrid CUAs: agents that interleave interface operation with code and command-line tool use within a single long-horizon process, deciding for themselves when to explore, when to implement, and when to run and verify. Two properties make this capability attractive. First, combining the two modalities widens the agent’s action space to match the environments people actually work in, where progress routinely alternates between operating an application and editing the software that drives it. Second, and more consequentially, the hybrid loop is self-grounding: because the agent builds something it can then run and inspect, every cycle produces concrete execution and visual feedback about its own output—feedback the agent can act on to correct itself within a trajectory, and that, being objective and executable, can be scored and selected to improve the agent across training. This makes hybrid computer use not only a broader capability but a naturally verifiable one, and therefore a promising substrate for self-improvement (Singh et al., 2024). To both measure and cultivate this capability, we introduce R ECREATION W ORLD, a framework built around recreation (Figure 2): given a running reference, an agent must discover its behavior and construct a faithful implementation, with no prescribed workflow. Recreation instantiates the hybrid loop in its purest form—the specification is encountered only by operating the reference, while the deliverable is code that must build and run—so an agent cannot succeed by clicking alone or by coding alone. This process is long-horizon by construction: success requires the agent to carry information across repeated cycles of reference exploration, implementation, candidate execution, visual verification, and revision. Recreation offers two additional advantages as an anchor for hybrid CUAs. It yields an objective, executiongrounded signal: the running reference is an oracle from which hidden behavioral tests can be derived and replayed against any candidate, giving an implementation-agnostic reward that, once the suite is validated and frozen, can be applied without per-candidate human judgment (Wang et al., 2026)—an instance of the asymmetry of verification (Wei, 2025). Recreation is also scalable: the large, continually refreshed supply of open-source applications can be turned into an open-ended stream of tasks and verified trajectories rather than a fixed, hand-authored set (Aggarwal et al., 2026; Cao et al., 2026). Concretely, R ECREATION W ORLD provides reproducible application recreation task environments across Ubuntu, macOS, Windows, Android, and Web, standardizing reference execution, candidate delivery, and experiment logging while preserving each platform’s native interaction semantics. The task pool spans application domains, interface frameworks, implementation languages, and levels of behavioral

and implementation complexity, so scaling introduces genuinely different executable systems rather than variations of a narrow task template. A unified agent harness exposes platform-native GUI control alongside coding tools through a common execution contract, while reference-grounded test generation turns heterogeneous application behavior into comparable, automatically scored outcomes. Together, these components make the same environments useful both for controlled evaluation and for generating diverse, execution-grounded training experience. We run recreation rollouts in task-isolated workers scheduled across a horizontally scalable virtualmachine pool and retain high-scoring trajectories as a shared training set for two model initializations. Across five out-of-distribution benchmarks covering coding, visual coding, and hybrid computer use, both training runs finish above their first evaluated checkpoints, with gains of up to 17.9 percentage points. These results provide initial evidence that recreation supervision transfers beyond the training environment. For held-out evaluation within R ECREATION W ORLD, we introduce R ECREATION B ENCH: 250 tasks—50 each on Ubuntu, macOS, Windows, Android, and Web—spanning heterogeneous UI frameworks, build systems, and accessibility and automation stacks. Reference implementations are withheld where the platform permits. To turn a running reference into a reliable evaluator, we design hidden test cases along two complementary axes: what they observe and how deeply they interact. Programmatic assertions read exact text, widget state, and action outcomes through each platform’s structured accessibility or automation substrate; visual assertions complement them by judging layout, color, canvas content, and other rendered properties absent from that substrate. Beyond observation coverage, each test case couples a trigger with its induced response and spans interaction depths from single-step state changes to multihop navigation, persistence, and end-to-end computations, rather than reducing fidelity to an inventory of visible elements. Platform-specific generation and filtering preserve this mix. Every proposed assertion must first pass on the reference and undergo human review before entering the fixed hidden suite, which is then replayed unchanged against the candidate. Together, these choices produce a graded measure of semantic and visual fidelity without constraining the candidate’s language, framework, or architecture. We evaluate ten frontier models on R ECREATION B ENCH and find a substantial gap to reference-level behavioral fidelity. GPT-6 Astra achieves the highest overall score at 58.1% and is the only model with full programmatic passes on multiple platforms, although such passes cover just 2.8% of tasks. Trajectory analysis shows that GUI interaction continues after implementation begins and that GPT-6 Astra combines broad reference exploration with a GUI-centered workflow; these descriptive comparisons do not establish that these behaviors cause higher scores. Across the training sweeps, agents at later checkpoints also check their own rendered outputs more often. Outcome analysis shows that agents reproduce static interface structure more reliably than interactions and computed outputs, while generated applications are substantially smaller and more monolithic than their references. We release R ECREATION B ENCH, its task environments, and test suites to support the study and development of long-horizon hybrid CUAs that jointly reason over interfaces and code.

R ECREATION W ORLD: Task and Framework

2.1

Task Formulation

Application recreation asks an agent to construct a runnable application from an executable reference. Each task is defined by three parts: 1) Input. The agent receives a high-level task prompt, interactive access to a running reference application A⋆ , and a prepared environment containing both GUI control and software-development tools; 2) Interaction. The agent may move repeatedly among operating the reference, writing code, building and launching its implementation, and visually and behaviorally checking the running candidate against what it has observed. The task specifies the desired artifact, but not the order or modality of these actions; 3) Output. The agent returns a complete source submission S that satisfies the platform’s build and launch contract: source with build and launch entry points on desktop, a Gradle project producing an APK on Android, or scaffolded source producing a self-contained b index.html on the web. Building and launching S produces the runnable candidate application A. The submission need not reproduce the reference’s architecture or source structure; only its observable behavior is evaluated. Section 4.4 defines the fidelity measure, while Section 4.1 describes the platformspecific reference visibility boundary. 2.2

Why Recreation?

Hybrid by necessity. Recreation cannot be completed by operating a graphical interface alone, nor by generating code alone. The specification is encountered through the reference application’s windows, 3

Verifiable by construction. The same executable reference that makes recreation challenging also provides a clean basis for verification. A generator can exercise the reference, capture concrete programmatic and visual outcomes, and turn them into a fixed hidden suite that is replayed against every candidate. Verification depends on observable behavior rather than source-level similarity, leaving the agent free to choose its architecture and workflow while keeping success objective and automatic. A submission that fails to build or launch receives zero; otherwise, the fraction of assertions it passes provides a graded signal beyond a single pass/fail outcome. Because new references and expected outcomes can be obtained from a broad, continually refreshed pool of open-source applications, task collection and evaluation can scale without making human-authored specifications the bottleneck. Figure 4 summarizes how executable references are converted into reproducible, verifier-backed tasks across platforms.

GUI↔code-edit switches per 100 calls

widgets, state transitions, and responses, whereas the deliverable is source code that must compile and run. This creates a recurring explore–implement–verify loop: observations of the reference update a partial behavioral specification; code turns that specification into a runnable candidate; and launching, interacting with, and visually inspecting the candidate exposes mismatches that trigger either another edit or renewed reference exploration. The task prescribes neither the order nor the frequency of these transitions, and the stages are not separable. Recreation also reverses the usual direction of a coding task. Rather than receiving a complete specification, the agent must recover one by forming hypotheses about hidden behavior and testing them against the running reference. The resulting feedback loop exercises perception, reasoning, code generation, tool use, and self-correction as a coherent capability rather than as isolated skills. Here, visual verification is an agent-side action within the rollout: the agent inspects its own rendered output to decide what to edit or explore next. It is distinct from the fixed hidden evaluation applied after submission. Section 6.1.1 measures how consistently agents complete this loop after their final source edit.

12 RecreationBench

WeaveBench

0 ProgramBench no GUI calls

OSWorld 2.0 GUI tools only

200

400

600

Trajectory horizon (tool calls)

Figure 3: Trajectory length and GUI–edit switching across neighboring benchmarks. Points show medians and bars interquartile ranges. Horizon counts top-level tool calls; switch rate counts adjacent GUI–edit transitions per 100 calls after projecting actions onto GUI and explicit file mutation. Other calls remain in the denominator.

Long-horizon interaction. Long horizons arise as a consequence of this feedback loop rather than from an imposed step count. Recovering behavior across screens and states, implementing it, resolving build and runtime failures, and visually and behaviorally checking the candidate against the reference require repeated cycles of exploration and revision. Figure 3 places this structure alongside nearby benchmarks under a common action taxonomy. R ECREATION B ENCH trajectories have a median of 282.5 top-level calls and 9.08 GUI–code-edit transitions per 100 calls. ProgramBench (Yang et al., 2026a) is longer but has no GUI calls, whereas OSWorld 2.0 (Yuan et al., 2026) records only computer-tool calls and thus has no separately observable code-edit calls. WeaveBench (Li et al., 2026b) uses both but is shorter and switches less often (178 calls and 1.56 transitions per 100 calls). This is an interface-level, structural comparison using each benchmark’s native scaffold and Claude Opus 4.8, not a controlled comparison of model performance; code entered through a graphical terminal remains a GUI action. Supporting trajectories of this duration requires an execution substrate that can preserve interactive state reliably for many hours and scale across concurrent runs. 2.3

Scalable Execution Infrastructure

The execution substrate in R ECREATION W ORLD is part of the experimental contract: it determines whether a reference can be reproduced, whether an agent can operate it reliably, and whether the resulting score reflects the candidate rather than host variation. Each rollout therefore receives a versioned, taskisolated worker that pins its graphical runtime, automation interface, and common build toolchains; reference artifacts are protected by the controls in Appendix C.6. R ECREATION W ORLD schedules these workers across a horizontally scalable virtual-machine pool, allowing many trajectories to execute concurrently. Each worker preserves its workspace, build artifacts, running processes, and graphical state across repeated explore–implement–verify cycles, maintaining continuity within a long-horizon rollout. Execution budgets are assigned at the trajectory level rather than constrained by short-lived requests; for recreation, we set the per-rollout wall-clock timeout to 20 hours. This lets task complexity determine the effective interaction horizon rather than an infrastructure-imposed

Sources

Behavior Discovery

Test Synthesis

Open-source apps

Orchestrator

Specialized generators Desktop

GitHub + F-Droid

Websites

Prepared references

Inspect source

Exercise app

Shared inventory Surface Trigger Outcome

Desktop Android

Web

Pinned & reproducible Build / install / snapshot

Reference replay

Web

Probe reference + author cases

</>

Licensed + synthetic

Android

Validation

Open

Dialog

Editor

Save

Saved text

Tool

Run

Exact value

Valid behavior Stable cases

Initial state + optional fixture

Web-only filters

Start

Discriminativeness Deduplication + quotas

1 hop 2+ hops

Human review Behavior + fixtures

One or both:

Programmatic tests.py

assert state assert value

Visual Screenshot + assertion

Frozen suite Hidden tests + fixtures

Revise / fill gaps

Figure 4: Reference-grounded task construction, from reference preparation and behavior inventory through test generation, validation, and suite freezing. cutoff. Workers share this lifecycle while retaining platform-native execution. Desktop workers expose an interactive operating-system session with native screenshot, input, and accessibility-tree access, together with the toolchains needed to build and launch candidate applications. Android workers host a pinned, hardware-accelerated emulator and provision device-side input helpers, while Web workers serve the reference locally and drive bundled headless Chromium through Playwright. The main benchmark exposes these platform actions as direct MCP tool calls. To limit infrastructure noise in the direct-MCP runs, control operations are bounded below the outer transport timeout, tool failures are distinguished from transport failures, and transient session failures are recoverable. Test generation and evaluation start from fresh platform instances, with task-required files installed explicitly as fixtures, preventing state from leaking between runs. Appendices C.2 and C.7 provide platform-specific runtime, provisioning, and harness details.

Scaling Recreation Environments and Data

Recreation requires agents to close the behavioral gap between a running reference and an evolving implementation. Doing so brings active exploration, visual verification, and iterative development into the same long-horizon trajectory, making the resulting experience useful supervision for a broad set of agentic capabilities. Recreation also turns the large and continually refreshed supply of open-source GUI applications and pinned commits into a scalable source of such experience. The executable reference acts as an oracle for constructing behavioral tests, and applying those tests to a candidate provides a direct, implementation-agnostic reward grounded in observable execution rather than annotator preference or source-code similarity. The experiments below test whether training on verified recreation trajectories transfers to coding and hybrid computer-use environments. 3.1

Training Setup

We source the training tasks from high-quality open-source GUI applications hosted on GitHub and cover the same five platform families as R ECREATION B ENCH. The application pool spans different domains, interface frameworks, programming languages, and levels of behavioral and implementation complexity. This variety exposes agents to different navigation structures, state transitions, and input-dependent behaviors, together with diverse build and runtime requirements. We deduplicate training tasks against the evaluation sets. We use Qwen3.8-Max to generate recreation trajectories through the agentic framework in R ECREATION W ORLD (Section 2). Agents use GUI and coding tools to explore the running reference, implement a candidate, and iteratively build, run, inspect, and refine it using execution feedback. The framework runs these rollouts concurrently on isolated workers and records their interaction histories for training. We apply rejection sampling using the task-specific behavioral verifiers to select high-scoring trajectories. We take 7,000 selected trajectories from each platform, yielding a balanced 35,000-trajectory SFT mixture. The balance is by trajectory count; trajectory lengths and hence token contributions may differ across 5

ProgramBench Partial (% tests passed) 45

GameCraft-Bench

Vision2Web

OSWorld 2.0

Overall (%)

Overall score (%)

Partial score (%)

44.6

26.5

18.4

56.6

17.4

Qwen3.7-Plus

17.9

35.7 57.7

47.6

58.2

48.2

52.2

41.9

Overall (%)

24 42

WeaveBench

46.1 60.2

55 46.8

Qwen-Flash-CPT

50%

100%

30.6

50%

49.8

100%

28.2

50%

100%

50%

100%

54.2

50%

100%

Figure 5: Transfer of recreation training to five out-of-distribution benchmarks. Rows are training arms and columns report benchmark-specific metrics on independent vertical scales. Training progress is normalized within each run. platforms. We fine-tune two model initializations on this same mixture. Qwen3.7-Plus is a fully post-trained starting point, whereas Qwen-Flash-CPT denotes an in-house Qwen-Flash checkpoint after continual pretraining. 3.2

OOD Generalization

Within-metric share (%)

We test whether recreation training transfers across 35 31.8 three coding benchmarks— ProgramBench (Yang 30 et al., 2026a), GameCraft-Bench (Luo et al., 2026), and Vision2Web (He et al., 2026)—and two 24.0 25 interact computer-use benchmarks—OSWorld 2.0 (Yuan 16.7 20.9 20.0 et al., 2026) and WeaveBench (Li et al., 2026b). All 20 five use a Claude Code scaffold with one trial interact 10.7 15 post-edit per task at each checkpoint. ProgramBench mea22.2 post-edit 10.1 18.2 sures test-pass fractions in a cleanroom container; 10 output observe 6.8 4.7 GameCraft-Bench uses Claude Code with native 15.1 output observe 5 2.9 10.2 shell and file tools and reports a build-gated rubric assets assets 5.3 3.9 pre 1.8 pre 1.7 score; Vision2Web covers all three task levels, using 0 First Last First Last First Last visual scores for Level 1 and the mean of visual and Image Code GUI functional scores for Levels 2 and 3. OSWorld 2.0 and verification inspection actions WeaveBench combine cua-driver MCP with shell execution and direct file editing, and report task-specific Figure 6: Behavioral changes averaged equally partial scores and shortcut-audited overall scores, reacross the two training arms. Metrics retain their spectively. Figure 5 presents these graded scores native denominators and sub-types. on a 0–100 scale along two checkpoint sweeps; task counts, budgets, and configuration differences are detailed in Appendix A.5. Both sweeps finish above their first measured checkpoint on all five benchmarks, although their trajectories are not uniformly monotonic. These results provide initial evidence that recreation supervision improves broader artifact-centric coding and interface–code reasoning beyond the training environment. 3.3

Behavioral Transfer

The score improvements in Figure 5 are accompanied by measurable changes in action allocation. Figure 6 gives an equal-weight average of the two arms’ first-to-last ratios. The groups use different estimands: ProgramBench verification per Bash call; per-task image-read fractions averaged across GameCraft-Bench and Vision2Web; and pooled GUI-call fractions averaged across OSWorld 2.0 and WeaveBench. They are not fractions of one action population. Both arms increase all three totals, own-output image reads, and GUI observation and interaction; Qwen3.7-Plus also increases reference/asset reads. Post-edit means after the first edit, not the final mutation. Appendix A.6 gives the detectors and aggregation rules. These descriptive shifts do not establish that the behaviors caused the gains.

R ECREATION B ENCH: Benchmarking Hybrid Computer-Use Agents Across Interfaces and Code

4.1

Platform Coverage: Desktop, Mobile, and Web

Real applications expose their behavior through platform-specific execution and interaction Table 1: Platform-specific delivery contracts and promechanisms, but the capability under study is grammatic test interfaces. Visual assertions supplethe same: recover a behavioral specification ment these APIs on every platform. from a running reference, turn that specificaTest API tion into code, and close the loop by operat- Platform Delivery ing the resulting artifact. R ECREATION B ENCH Ubuntu Source + build/launch AT-SPI therefore applies one observe–build–evaluate macOS Source + build/launch AXUIElement contract across five platforms, organized into Windows Source + build/launch UI Automation desktop (Ubuntu, macOS, and Windows), mo- Android Gradle project → APK UiAutomator bile (Android), and web. In every case the agent Web Pinned React web stack → DOM / ARIA interacts with a fixed reference, produces a sepself-contained index.html arately runnable candidate, and is evaluated on behavior observed through the platform’s programmatic interface and rendered output. The platformspecific delivery contract and programmatic test interface are summarized in Table 1. These contracts preserve each platform’s native build and automation semantics rather than routing all tasks through a shared abstraction. The concrete operating systems and provisioning are specified in Sec. 2.3, the scoring protocol in Sec. 4.4, and benchmark composition in Sec. 4.2. The main protocol asymmetry is reference visibility. Desktop and Android implement a source-blind setting: the agent observes an executable interface while the implementation artifact is withheld. The degree to which individual toolkits expose AT-SPI, AX, or UIA bounds what can be asserted; macOS additionally includes menu-bar applications whose primary surface is a status item and popover rather than an ordinary window. Android further removes remote-service variation by selecting applications that do not request the Internet permission and keep their observable state on device. Web cannot enforce the same source-blind boundary because a static site’s client implementation is precisely what the server sends: the agent may inspect the served HTML, styles, scripts, and assets. There the protected material is the captured ground truth and generated test suite. The delivered page is required not to load the reference at evaluation time; both desktop and mobile viewports are scored, and multi-page tasks must preserve the original path-based URLs rather than substitute hash routing. 4.2

Benchmark Composition

R ECREATION B ENCH contains 250 tasks: 50 applications or websites on each of Ubuntu, macOS, Windows, Android, and Web. We freeze each roster from the corresponding release inventory; the three desktop inventories also record the upstream repository and revision from which each reference was built. Figure 7 compares the four application platforms under a shared nine-domain functional rubric. Web retains its ten site categories, including separate Finance & Business and Education & Reference categories; release metadata preserve the original labels. Independently of genre, the Web suite combines 44 synthetic sites with six sites derived from public websites, balancing controlled coverage and real-world design variation. Appendix C.3 describes its separate real-site and synthetic-site selection procedure. The desktop and Android selections also span substantially different implementation stacks and scales (Figure 7, bottom left). For the three desktop platforms, applying the same source-only cloc policy at pinned upstream revisions reveals several orders of magnitude of variation in production size, both across and within platforms. Ubuntu combines substantial Qt and GTK cohorts with web-wrapped applications; macOS is dominated by AppKit and SwiftUI; and Windows leans toward .NET and JVM stacks while retaining a sizeable Qt cohort. The figure merges versions and language bindings within each framework family while retaining one point per application. The bottom-right plot additionally shows that the four non-Web suites span both niche and widely adopted upstream projects; stars are descriptive popularity metadata rather than a proxy for task difficulty. Android adds a complementary mix of Kotlin and Java applications and XML and Compose interfaces. Its inventory-reported source counts provide the fourth row, with a median of 15.0k LOC; these use platform-specific counting rules described in Appendix C.1.

Desktop & Android

Web Ubuntu 8 10 7 7 8 5

macOS 11 11 8 4 7 4 2 3 0

Productivity & Wellness Developer & Engineering Files & System Graphics & Creative Text & Documents Media (Audio / Video) 1 Games & Leisure 2 Security 2 Science & Data 0

Windows 12 9 6 8 4 2 3 2 4 0

Android 23 1 6 3 3 3 7 2 2

Framework and source scale

13 Developer & Engineering 7 Finance & Business 7 Science & Data 6 Education & Reference 6 Productivity & Wellness 4 Graphics & Creative 3 Media 2 Security 1 Files & System 1 Text & Documents

Repository popularity

AppKit / SwiftUI

.NET

JVM

Web

Views / Jetpack Compose

GTK

Other native

Median

Ubuntu

49.3k

Ubuntu

808

macOS

13.5k

macOS

572.5

Windows

6.7k

Windows

Android

15.0k

Android

701

100

10k

100k

Production source LOC (log scale)

100

10k

100k

Repository stars (log scale)

Figure 7: Benchmark composition. Top: functional domains for desktop and Android applications and release-native categories for Web. Bottom left: native-application source size and framework distributions. Bottom right: repository-star distributions for non-Web tasks. 4.3

Reference and Test-Suite Construction

Construction turns a candidate application or website into a reproducible reference and a hidden behavioral test suite, together with the inputs required for replay. Figure 4 summarizes the process: prepare the reference, author tests of its observed behavior, and validate the cases before freezing the suite. Reference selection and preparation. Candidates come from open-source desktop repositories, Android projects distributed through GitHub or F-Droid, and openly licensed or benchmark-authored websites. We enforce redistribution requirements and exclude libraries, frameworks, host-program extensions, and applications whose core functionality depends on a login or remote service. Each candidate must run reproducibly in its evaluation environment. Desktop references are pinned to upstream commits, built, and exercised in a graphical session. Android references are built and installed in the emulator, with incidental first-run interruptions removed. Web references are frozen as static snapshots, with pages discovered and reviewed before the scored page set is fixed. We reject references that fail to launch, render incompletely, or lack meaningful interaction. Among the survivors, we balance domains, frameworks, source size, feature complexity, and upstream activity; Web additionally balances real and synthetic sites. Platform-specific selection and preparation details appear in Appendix C.3. Behavioral test authoring. An orchestrator combines source or page analysis with exploration of the running reference to build a shared inventory of features, reachable surfaces, triggering actions, and observable responses. Desktop and Android generators divide this inventory among specialized subagents that author tests for complementary aspects of behavior; depending on the platform’s session constraints, runtime probing is performed by the orchestrator, the subagents, or both. Web uses scripted, agent-authored, interaction, and visual tracks. Each case specifies any required fixture and initial state, an interaction sequence, and programmatic assertions, visual assertions, or both. Source inspection may guide coverage, but assertions depend on runtime-observable behavior rather than reference internals, so the same case can run unchanged against an independent recreation. Figure 8 illustrates this contract through Logbert’s fixed log input, interaction steps, and two assertion channels. Across the frozen suites, 81 tasks package 393 files in their fixture trees; this counts packaged files rather than independently addressed test inputs. Web stores its snapshots separately and has no explicit testcase fixture bundle. Appendix C.4 gives the fixture inventory and platform-specific generators. Validation and freezing. We replay every proposed case against a clean reference instance and discard invalid or unstable checks. Uncovered inventory entries guide further authoring, and platform-specific filters prune unsuitable cases, including Web’s discriminativeness and redundancy gates. Human reviewers then inspect each surviving case and its execution on the reference, checking that the fixture, 8

actions, and expected outcomes are valid, unambiguous, and faithful to the observed behavior. Cases that fail review are revised and revalidated or removed. Only after these checks do we freeze the suite and its fixtures for candidate evaluation (Sec. 4.4); Appendix C.4 details the validation gates. Coverage audit. We audit the frozen suites for navigation depth and outcome specificity. Depth counts UI-surface crossings before the assertion; Exact is the subset of Outcome cases requiring a specific expected result. Table 2 shows that, averaged equally across platforms, about 77% of cases leave the start surface and 24% traverse two or more surfaces. Most cases (94.2%) check an interaction outcome rather than mere presence, and 40.7% require an exact expected result. These percentages characterize the navigation and post-action behavior covered by the suites. 4.4

Table 2: Navigation depth and outcome specificity of the frozen test suites across five platforms (%). Platform

Navigation depth

Outcome specificity

Start 1 hop 2+ hops

Outcome (Exact)

Ubuntu macOS Windows Android Web

22.5 34.5 26.9 16.5 13.4

57.8 44.0 54.2 42.2 67.6

19.7 21.6 18.9 41.3 19.0

94.6 (41.6) 90.2 (52.6) 92.9 (51.1) 93.4 (29.2) 100.0 (28.8)

Macro avg.

22.8

53.2

24.1

94.2 (40.7)

Evaluation Protocol

Evaluation replays each application’s reference-validated hidden suite against the candidate in a clean environment. A fixture is task-supplied data or state installed before replay, such as a document, media file, or directory tree; it is an evaluation input rather than an assertion or expected answer. Each retained case fixes its fixture and initial state, interaction sequence, and expected observations; the same setup is run on the reference and every candidate. The suite freezes two complementary inventories, Prog and VLM, so a candidate cannot change its own denominator. An expected case that is absent, crashes, or is never reached fails, and a candidate that cannot be built or launched receives zero. Prog measures behavior exposed through the platform’s structured automation interface. After replaying the interaction, its assertions read exact text, widget state, navigation, persistence, or computed values through AT-SPI, AXUIElement, UI Automation, or UiAutomator. Desktop and Android score the fraction of frozen cases passed. Web follows the same principle but weights its functional checks by specificity, giving behavior and content greater influence than shallow launch or existence checks. VLM measures rendered behavior at frozen visual checkpoints. The harness pairs each candidate screenshot with a reference-grounded natural-language assertion, and a shared Qwen3.7-Plus judge (Qwen Team, 2026a) at temperature zero returns a binary verdict. Desktop and Android score the assertion pass fraction; Web first aggregates within each page and viewport, then combines desktop and mobile views across pages. A missing candidate screenshot fails its assertion, whereas a judge transport or parsing failure is an invalid measurement to rerun rather than a model failure. We report Prog and VLM separately, macro-averaging applications within each platform and weighting platforms equally; where a single summary is useful, we average the two channel scores. Figure 8 shows both channels on one Windows case: the same fixture-driven interaction produces exact UI Automation values and a visual checkpoint of the resulting chart. 4.5

Isolation and Evaluation Integrity

We treat model-generated commands, processes, and artifacts as untrusted. Without isolation, an agent could read reference source code or build artifacts from the local machine, or retrieve the upstream repository from the Internet, reducing reconstruction to source copying. We therefore expose only the running interface, a writable workspace, and necessary tools and inputs, while withholding reference artifacts, hidden evaluation material, and credentials. At handoff, agent-owned processes are terminated and the candidate tree is frozen; a failed isolation check invalidates the run rather than lowering the model’s score. Local permission isolation. Each platform combines a dedicated unprivileged agent principal with POSIX permissions or Windows ACLs, and probes the boundary under that identity before rollout. Only approved fixture copies enter the agent-visible workspace; canonical reference and evaluation assets remain protected. Web necessarily exposes its served client and therefore protects ground truth and tests instead, while Android retains the residual risk that an installed reference package is recoverable in principle. For Web, detected direct repackaging or replay of reference implementation artifacts caps the task aggregate at 0.10; independently authored reconstructions remain permitted. Appendix C.6 gives the complete platform-specific boundaries.

Fixture

controlled input

sample_log4net _mixed.log

Open logger

Select fixture

Loaded rows

depth 0

depth 1

depth 2

use_menu("File", "New Logger")

set_edit_text(MIXED_LOG); OK

wait_until(Number col. populated)

10 messages Info

Error

Debug

Computed statistic

depth 3 (final)

tab("Statistic").click_input()

def test_statistic_percentages_on_mixed_log(app, rb, results_dir):

Programmatic assertions

VLM assertion

labels = [ v for v in (L.statistic_labels(app) or []) if v and v.strip() ] observed = sorted(labels) ok = rb_value(observed, ["20%", "30%", "50%"]) assert ok, ... assert len(labels) == 3, ... assert "17%" not in labels, ...

...read 50%, 30% and 20% ... 50% slice is by far the largest and the 20% slice the smallest. ... 'Debug: 2', 'Info: 5' and 'Error: 3'; no Trace, Warning or Fatal ... 'Statistic' selected. ... ten-row message grid ... 'Logger Tree' ... 'Status: Running' and '10 Messages (all displayed)' are unchanged. Chart detail · final checkpoint

Figure 8: Generated Windows test case for Logbert. Loading a fixed log and selecting Statistic leads to exact UI Automation and visual assertions. Pale panels show intermediate states, and the inset enlarges the final chart; evaluation uses the full application-window capture. Network isolation. Ubuntu filters egress by agent UID, macOS and Windows apply host-wide defaultdeny firewalls, and Web uses a routeless namespace with narrow relays to the reference and model endpoint. Android instead relies on offline dependencies and stripped credentials and does not yet attest a packet-level egress filter. Appendix C.6 gives the exact rules and residual guarantees; these controls prevent run-time retrieval, while Section 6.4 separately audits possible pretraining exposure.

Main Results

5.1

Benchmark Performance

We evaluate the ten frontier models with the inference configurations in Table 8. Table 3 reports aggregate results, and Appendix B gives the per-platform breakdowns. All runs contributing to these benchmark results expose platform actions through the standard direct-MCP interface. The persistent programmable runtime studied in Section 5.2 is a separate experiment and does not contribute to the reported benchmark scores. GPT-6 Astra obtains the highest overall score at 58.06%, followed by Claude Opus 5 at 44.16% and GPT-5.6 Sol at 42.06%. GPT-6 Astra is also the only model with full-suite passes on multiple platforms, with 90% and 100% Prog coverage of 17.6% and 2.8%, versus at most 5.5% and 0.8% for any other model. 5.2

Study: Programmable Interaction Runtime

We examine whether allowing an agent to compose GUI operations inside a persistent runtime can reduce interaction overhead without materially changing observed task quality. Interface. Direct MCP use returns control after every primitive action and repeatedly places raw observations into context. The experimental desktop configuration instead exposes Qwen Code’s qwen-cua-driver, a fork of cua-driver, through a persistent Node.js REPL and typed JavaScript SDK. The agent can compose SDK calls with loops and conditionals, retain state across executions, and select which observations return to context.

Table 3: Main results on R ECREATION B ENCH. Scores and threshold coverage are averaged equally across platforms; the average score is the mean of Prog and VLM. Appendix B reports the per-platform results. Gemini 3.7 Kimi Qwen3.7- Grok Qwen3.8- Claude Claude GPT-5.6 GPT-6 GLM-5.3 Flash K3 Plus 4.6 Max-0902 Opus 5 Opus 4.8 Sol Astra

Metric Programmatic score (%) VLM score (%) Average score (%)

24.91 17.34 21.12

32.07 30.74 31.41

26.30 22.46 24.38

9.18 9.12 9.15

39.02 34.45 36.73

35.53 34.07 34.80

45.99 42.34 44.16

32.40 29.81 31.10

40.63 43.49 42.06

58.19 57.92 58.06

Prog ≥ 90% (% apps) Prog = 100% (% apps)

2.40 0.00

2.00 0.00

0.00 0.00

1.60 0.00

2.00 0.00

5.53 0.80

1.60 0.40

5.20 0.40

17.60 2.80

Comparison protocol. We compare the directMCP and programmable configurations on the same 50 Windows applications with Claude Opus 4.8 and one rollout per task and configuration. We report task-quality metrics over all evaluator-valid task pairs. For interaction and resource metrics, we retain the 39 pairs with non-error terminal results and complete usage records in both configurations, and report the relative change from direct MCP to the programmable configuration; negative values denote reductions.

Direct MCP

Programmable SDK 35.05 35.60

Prog score

31.00 32.29

VLM score 0

Score (%)

Computer-use calls

-39.9%

Agent turns

-35.8%

Input tokens Tool-result text

-40.7% -65.5%

Results. In this auxiliary comparison, the pro+15.7% Output tokens grammable configuration yields similar observed task quality while substantially reducing model–tool -26.1% Wall-clock time calls, incoming context, wall-clock time, and esti-54.1% Model cost mated cost (Figure 9). Output tokens increase by −60 −40 −20 0 +20 15.7%, while tool-result text and input tokens fall Change from Direct MCP (%) by 65.5% and 40.7%, respectively. This pattern is consistent with the model spending more tokens on JavaScript orchestration: the REPL lets it control SDK Figure 9: Direct MCP versus a programmable SDK call sequences, process intermediate results locally, on Windows with Claude Opus 4.8. Top: programand return selected observations within a single ex- matic and VLM scores; bottom: relative changes in ecution, reducing repeated model–tool exchanges. interaction and resource use. Wall-clock time falls from 4.12 to 3.04 hours per task, and estimated model cost from $90.50 to $41.58 per task. Because this is a one-rollout comparison of complete configurations, the result shows lower observed interaction overhead rather than an isolated causal effect of the persistent runtime; Appendix C.7 gives the pairing, measurements, and scope of this comparison.

Analysis

To understand how agents approach recreation and where current systems remain limited, we analyze recorded trajectories, delivered source, and evaluator outcomes. Together, these views characterize how agents investigate references, implement and verify their submissions, how the resulting artifacts differ from reference applications, and which tested behaviors most often remain incomplete. 6.1

Trajectory-Level Analysis

We examine five complementary aspects of the reconstruction process: reference investigation and verification, implementation size over trajectory progress, evaluation-hacking attempts, resource use, and the functional composition of tool calls. Because model identity, harness, and runtime vary together, these comparisons characterize observed workflows but do not isolate the causal effect of any individual behavior. 6.1.1

Reference Investigation, Implementation, and Verification

Recreation requires an agent to infer the reference’s states and transitions, express that evidence in code, and validate the artifact it will submit. Figure 10 summarizes reference exploration, implementation style, 11

GPT-6 Astra

GLM-5.3

Qwen3.8-Max-0902

REFERENCE-APP EXPLORATION

(d) Largest write

Final source (%)

Trajectories (%)

IMPLEMENTATION & VERIFICATION

(b) Windows inspected

Distinct windows

Types used (%)

(a) Interaction breadth

(e) Final-loop closure 60 Trajectories (%)

Claude Opus 5

Figure 10: Reference investigation and final verification. Bars show platform-balanced means with 95% bootstrap intervals. and final-state verification. Here, GUI observation includes screenshots, accessibility-tree queries, and DOM inspection, while GUI interaction includes clicks, typing, and scrolling. Visual input is routine in these trajectories: Claude Opus 5, Qwen3.8-Max-0902, and GPT-6 Astra receive a median of 53 images per trajectory, and 96.8% of their trajectories include at least one image (Appendix C.7). GLM-5.3 receives structured GUI representations but no screenshot pixels, so observation counts do not imply visual access. Broader reference exploration accompanies stronger benchmark performance. GPT-6 Astra achieves the highest interaction breadth, covering 58.1% of the platform-supported reference-side GUI interaction types, while Qwen3.8-Max-0902 follows at 41.8%. The same ordering appears in window coverage, where GPT-6 Astra observes 5.23 distinct reference-window identities per trajectory compared with Qwen3.8Max-0902’s 2.21. The two metrics capture complementary dimensions of exploration—interaction diversity and observed interface states—so their agreement is not specific to a single proxy. This broader reference coverage co-occurs with the strongest benchmark result: GPT-6 Astra averages 58.06% and leads the runner-up, Claude Opus 5, by 13.9 percentage points (Table 3). Controlled input variation is uncommon and similar across models. Controlled input variation— entering at least two distinct nonempty values in the same reference input field—appears in 17.0% of GPT-6 Astra trajectories, close to Claude Opus 5 at 16.8% and only modestly above GLM-5.3 and Qwen3.8-Max-0902 at 13.4%. GPT-6 Astra’s clearest separation is therefore in breadth and window coverage rather than in this particular controlled-probing strategy. Same-field variation captures only one way to test input-dependent behavior and is not a complete measure of exploration quality. Final-loop closure remains low across all models. Under the strict final-loop criterion—final source change, recreation relaunch or reload, and subsequent recreation-side GUI observation—Qwen3.8-Max0902 most often completes the sequence (47.5%), followed by GLM-5.3 (38.0%), GPT-6 Astra (29.1%), and Claude Opus 5 (23.6%). Even the highest rate is below 50%, so most trajectories do not complete a relaunch and inspection after the final source change. Validation before that change does not satisfy the criterion, and a low rate does not imply that no earlier validation occurred. Qwen3.8-Max-0902’s lead also does not coincide with the highest benchmark score, showing that final-loop closure alone is not a proxy for fidelity. 6.1.2

Implementation Size over Trajectory Progress

The largest observed source mutation accounts for 23.3% of reconstructed final source for Claude Opus 5 and 25.3% for GPT-6 Astra, compared with 34.4% for GLM-5.3 and 39.3% for Qwen3.8-Max-0902 (Figure 10(d)). Source changes are therefore less concentrated in one mutation for Claude Opus 5 and GPT-6 Astra. The largest-write share captures how concentrated the edits are, but not when the code is produced. We therefore replay successful, content-bearing source mutations over trajectory progress. At each assistant-turn boundary, the resulting source size is divided by that trajectory’s final replayed source size; 100% thus denotes the source state reconstructed at the end of the trace, not a reference-code target. Figure 11 reports the pointwise median of these normalized curves within each model–platform cell. Most code is written early, followed by smaller edits. By the trajectory midpoint, most model–platform median profiles already exceed half of their final reconstructed source size. Most curves rise sharply soon

Claude Opus 5

GPT-6 Astra

GLM-5.3

Final replayed source (%)

Constant-rate source growth

(a) Ubuntu

(b) macOS

Qwen3.8-Max-0902

Median first code-writing turn

(d) Android

(e) Web

100

0 0

100 0

100

Assistant-turn progress (%)

Figure 11: Normalized source growth over trajectory progress. Curves show model–platform medians; triangles mark the first code-writing turn and the diagonal denotes constant-rate growth. after the first code-writing turn and then grow more gradually across later bins rather than tracking the constant-rate diagonal. This pattern is consistent with a large initial scaffold followed by smaller edits. These aggregate traces do not, however, establish whether each later edit improves fidelity. Implementation timing varies across platforms. We also find that the same model can write code at different stages on different platforms. Qwen3.8-Max-0902 shows the largest contrast. At the midpoint, its median reconstructed source size is 97.4% of its final size on macOS but only 19.9% on Web. The other Web profiles reach 61.3–86.0% at the same point. This within-model span is larger than many between-model differences on a fixed platform. 6.1.3

Hacking Attempts

Prior work has observed tool-using agents searching for benchmark materials during evaluation (Anthropic, 2026c). Recreation presents a similar concern: an agent may seek shortcuts through protected information rather than infer the reference’s behavior through GUI exploration. We therefore examine four classes of executable operations: external-network access, protected reference or evaluation-path access, inspection or extraction of installed reference packages or binaries, and privilege escalation. We count each executed operation matching one of the four detectors as an attempt. Repeated matches within one trajectory count separately, whether they succeed, fail, or are blocked. Figure 12 reports mean attempts per trajectory rather than the fraction of trajectories with at least one attempt. The metric does not establish successful access, malicious intent, or explicit evaluation awareness. Across models, GLM-5.3 has the highest platform-balanced mean in every attempt category. Its networkattempt rate is 1.57× the next-highest rate, and its protected-path rate is nearly twice the next highest. GPT-6 Astra exhibits a different profile: it ranks second on network attempts but lowest on protected-path attempts, while package- or binary-directed matches are concentrated in GLM-5.3. These differences would be obscured by a single aggregate evaluation-hacking statistic. More broadly, agents repeatedly issue executable operations matching network-egress and protected-path detectors during benchmark runs. Evaluation integrity therefore cannot rely on prompt compliance alone: prohibited resources require explicit access controls and auditing. The observed attempts do not imply that any boundary was successfully bypassed. 6.1.4

Token Usage and Estimated Cost

R ECREATION B ENCH is a long-horizon benchmark. Figure 13 jointly reports cumulative input and output tokens, assistant turns, and an API-cost estimate for each model–platform cell. Across the available cells, model–platform means range from 191 to 741 assistant turns and from 23 to 259 million cumulative input tokens per attempt. Token totals are the usage attributed by the corresponding runtime, not unique-token counts or peak context lengths. Assistant turns follow the top-level response boundaries in the corresponding trajectory format and are therefore not a harness-independent unit of reasoning. Recorded usage and cost do not follow the benchmark ranking. GLM-5.3 has the highest mean assistant-turn count on four of five platforms and the highest mean cumulative input-token count on four of five, leading both measures on macOS, Ubuntu, and Windows. On those platforms, its trajectories combine more recorded response boundaries with greater cumulative input-token usage. Yet, under the

Mean operations / trajectory

Claude Opus 5

(a) External-network access attempts

GPT-6 Astra

GLM-5.3

(b) Protected-path access attempts

Qwen3.8-Max-0902

(d) Privilege-escalation attempts

0.9 0.6 0.3 0.0

Figure 12: Detected evaluation-hacking attempts by model and category. Bars show platform-balanced means per trajectory with 95% stratified bootstrap intervals.

(a) Input Tokens

100

0 un Ub

s S id ow cO dro ma Wind An

1250 1000

b We

750 500 250

Qwen3.8-Max-0902

200

GLM-5.3

(b) Output Tokens Mean output tokens per trajectory (thousands)

Mean input tokens per trajectory (millions)

300

GPT-6 Astra

800 600 400 200 0

un Ub

s S id ow cO dro ma Wind An

b We

(d) Estimated API Cost

Mean cost per trajectory (USD)

Claude Opus 5

200 150 100 50 0

un Ub

s S id ow cO dro ma Wind An

b We

un Ub

s S id ow cO dro ma Wind An

b We

Figure 13: Trajectory resource use by model and platform: cumulative input and output tokens, assistant turns, and estimated API cost. Bars show means with 95% bootstrap intervals. Cost applies OpenRouter rates (OpenRouter, 2026), assuming 90% cache-read input; estimates exclude tool and runtime costs. common pricing scenario in Figure 13(d), Claude Opus 5 has the highest estimated API cost on every platform. GPT-6 Astra, by contrast, records fewer input and output tokens than Claude Opus 5 on every platform and remains less expensive in that scenario despite its higher listed per-token rates. Providernative token counts still reflect different tokenizers and accounting conventions, so these comparisons characterize the deployment footprint of each evaluated configuration rather than tokenizer-controlled model efficiency. 6.1.5

Tool-Call Breakdown

An assistant turn is one completed response, and a recorded call is one top-level tool invocation; a turn may therefore contain zero, one, or multiple calls. To characterize how agents use GUI and commandline interfaces, we map model-specific tool names to a common vocabulary and assign each recorded top-level invocation to exactly one of seven functions: GUI observation, GUI interaction, reading, writing/editing, building/execution, planning, or other. File inspection and source mutation map to reading and writing/editing, respectively; compilation, execution, and application launch map to building/execution; explicit planning-tool calls map to planning. Figure 14(a) first converts each trajectory into a seven-part composition, preventing long trajectories from dominating the average. Panels (b)– (e) apply the same partition over normalized assistant-turn progress. Their denominator is the set of actions observed at each progress position, not the number of turns. A shell call that batches several primitive operations still counts as one invocation, whereas dedicated GUI calls are recorded separately. The analysis therefore measures the composition of recorded tool calls rather than a common count of primitive agent actions. Higher GUI-call shares coincide with higher benchmark scores. The highest-scoring model, GPT-6 Astra, allocates 49.5% of its recorded calls to GUI observation and 23.4% to direct GUI interaction, giving it the largest combined GUI share among the four models. This is consistent with its lead in reference-side interaction breadth and window coverage (Figure 10) and characterizes a workflow centered on acquiring interface feedback and manipulating GUI state alongside CLI-based implementation and execution. Qwen3.8-Max-0902 has the highest writing/editing share (21.4%), whereas GLM-5.3 has the highest 14

GUI observation

GUI interaction

Reading

Writing / editing

Build / execution

Planning

Other

(a) Whole-trajectory composition 29

Claude Opus 5

GPT-6 Astra 36

GLM-5.3 Qwen3.8Max-0902

23 14

11 17

Share of tool actions (%) (c) GPT-6 Astra (d) GLM-5.3

(b) Claude Opus 5

100

(e) Qwen3.8-Max-0902

Action share (%)

100

0 0

100

Assistant-turn progress (%)

Figure 14: Tool-call composition and its evolution over normalized trajectory progress. (a) Platformbalanced per-trajectory composition; (b)–(e) call shares by model and progress. build/execution share (17.4%). For GLM-5.3, GUI observation accounts for 35.8% of calls but direct interaction for only 6.5%, an observation-to-interaction ratio of roughly 5.5:1. Claude Opus 5 distributes its core calls more evenly across GUI observation, reading, writing/editing, and build/execution. These are distinct operating profiles, not a common ranking of model quality. GUI use persists as implementation activity increases. Across models, the combined writing/editing and build/execution share rises from 9.3–20.7% in the first progress bin to 26.6–38.5% in the last. GUI observation and interaction still account for 33.2–58.6% of the final bin. At the model-aggregate level, agents therefore shift toward implementation while continuing to use the GUI, consistent with a hybrid workflow. Aggregate shares do not show whether individual trajectories alternate between these call types, and late GUI activity does not establish final verification. The strict final-verification sequence remains incomplete in most trajectories (Figure 10(e)). 6.2

Artifact Structure Analysis

Because R ECREATION B ENCH evaluates observable behavior rather than implementation similarity, agents need not reproduce the reference’s source architecture. This flexibility motivates two questions about the delivered artifacts: whether higher scores coincide with larger implementations, and how closely recreations preserve the reference’s scale, file organization, and framework choice. We include every available delivered project with production source for the four models analyzed above, without filtering by behavioral score. Coverage varies across model–platform cells. Reference source comparisons cover the three desktop platforms and Android because Web references are built snapshots. Appendix C.1 details the source-counting policy and robustness checks. Recreations are usually smaller, and their size is only weakly associated with reference size. Across the three desktop platforms and Android, 89.4% of recreations contain less production source than their corresponding references, and the median recreation-to-reference LOC ratio is 16.9% (Figure 15). The within-platform log–log slope estimates are positive but shallow (0.08–0.23); at these point estimates, a tenfold increase in reference LOC corresponds to only a 1.2–1.7-fold increase in recreation LOC. Codevolume ordering also does not follow benchmark performance: Claude Opus 5 has the highest median recreation LOC on all five platforms, whereas GPT-6 Astra, despite achieving the highest overall score, has the lowest median LOC on four of the five platforms. The model-level comparison therefore does not support a simple account in which generating more code yields higher fidelity. It does not estimate a task-level effect of code volume or identify which implementation choices explain GPT-6 Astra’s advantage. Recreations usually use fewer, more concentrated source files. Across the same native platforms, 92.3% of recreations use fewer production-source files than the corresponding reference, and 83.8% place 15

Claude Opus 5

(b) macOS

GLM-5.3

Qwen3.8-Max-0902

(d) Android

(e) Web

104

103

102 β = 0.08

102

β = 0.20

104

Reference LOC

102

β = 0.23

104

Reference LOC

β = 0.09

104

102

Reference LOC

102

104

Reference LOC

C O lau pu d s e 5 G As PTtr 6 a G LM -5 .3 Q M we ax n -0 3. 90 82

Recreation production LOC

(a) Ubuntu

GPT-6 Astra

Figure 15: Recreation source size. Native-platform points are task–model pairs; dashed lines mark equal size and solid lines show within-platform log–log fits. Web boxes summarize recreation LOC with Tukey whiskers computed in log space. a larger share of source in the single largest file (Figure 16(a)). At the model–platform level, median file counts are 4–43 for recreations and 47–167 for references; median largest-file shares are 10–64% and 5–21%, respectively. The model with the highest concentration differs by platform—GLM-5.3 on Ubuntu and macOS, Qwen3.8-Max-0902 on Windows, and GPT-6 Astra on Android—so concentration is a shared artifact pattern rather than a signature of one model. Web has no comparable reference source; its recreations contain median counts of 26–38 production-source files and median largest-file shares of 15–31%. These measurements establish structural compression, but do not show that it causes lower behavioral fidelity. Recreations frequently substitute platform-native desktop toolkits. They often reimplement applications using GTK on Ubuntu, AppKit on macOS, and WPF or WinForms on Windows. We identify UI frameworks from project files and source code (Figure 16(b)). Among Ubuntu and macOS deliverables whose references use Electron, Tauri, or Wails, 75% switch to the platform-native GTK or AppKit/SwiftUI family. On Windows, 50% of deliverables whose references use a non-.NET family switch to WPF or WinForms. Family-level retention, measured over pairs with a detected framework on both sides, ranges across models from 75–85% on Ubuntu, 63–76% on macOS, and 45–94% on Windows. Android’s implementation contract prescribes XML views: every deliverable with a detected UI framework uses them, although 44% of the references use Jetpack Compose. Web projects with usable source remain in the prescribed Web family, predominantly using Tailwind; Web is omitted from the transition panel because its references are built snapshots rather than source projects. Framework retention is not a requirement for behavioral correctness; the transition data therefore show architectural substitution rather than implementation quality. 6.3

Failure Modes and Error Analysis

To determine what the aggregate Prog scores in Table 3 conceal about application-level completeness, we first examine how often recreations approach or attain full programmatic credit and which tested behavior classes lag. We then use trajectories and audited cases to identify workflow gaps that can leave functional errors undiscovered, focusing on controlled input variation and post-edit validation. Few applications pass the full programmatic suite. Figure 17(f) shows the complementary cumulative distribution of Prog scores, averaged equally across platforms. At 90%, coverage is 17.6% for GPT-6 Astra, 5.5% for Claude Opus 5, 2.0% for GLM-5.3, and 2.0% for Qwen3.8-Max-0902. At 100%, GPT-6 Astra remains highest at 2.8%, while every other model reaches at most 0.8%; full-suite success therefore remains rare even for the leader. An aggregate Prog mean should therefore not be read as the probability that a model reconstructs an application completely. Evaluated recreations reproduce what an interface contains more reliably than what it does. Across all five platforms, tests that require an action-dependent state change or computed output receive lower pass rates than tests centered on static interface structure or content (Figure 17(a–e)). Among the six displayed native categories on Ubuntu, macOS, Windows, and Android, structure has the highest mean pass rate in 15 of 16 model–platform comparisons; the exception is GPT-6 Astra on Windows, where 16

Reference Claude Opus 5

GPT-6 Astra GLM-5.3

Qwen3.8-Max-0902

Ubuntu

60%

40%

GTK

Largest-file share (bars)

File count (points, log)

80%

macOS Windows Android

AppKit / SwiftUI

.NET

Qt wxWidgets Electron

Win32

Undetected

Gecko

Pascal

Electron

Wails

Undetected

Tauri

JVM

Tauri

Electron

JVM

Electron wxWidgets

Ubuntu

.NET

AppKit / SwiftUI

20%

100

Windows

GTK

100%

102

macOS

Win32

Undetected

Web

(a) File organization

(b) Framework-family transitions

Figure 16: Implementation structure of recreations. (a) Median source-file counts and largest-file LOC shares for recreations and references. (b) Reference-to-recreation framework transitions on desktop platforms; ribbon width counts task–model pairs, with pale ribbons for retention, solid ribbons for changes, and gray for undetected frameworks. Claude Opus 5

GPT-6 Astra

(a) Ubuntu

GLM-5.3

Qwen3.8-Max-0902

(b) macOS

Structure

Text

State

Button

Computation

Computation 0

100

Computation 0

(d) Android

100

(e) Web

100

(f) Complementary CDF 90% 17.6

100

Structure

100%

Text

Applications (%)

Static content

Navigation State Interaction

Computation Interaction

5.5

2.0

2.8 0.8 0.0

50 25 0

100

Programmatic score (%)

Programmatic pass rate (%)

Figure 17: Programmatic outcomes by platform and score threshold. (a–e) Application-macro category pass rates under each platform’s native taxonomy. (f) Platform-balanced complementary CDF of application-level Prog scores, with a 90–100% inset. menu is higher by only 0.2 percentage points. The weakest category is always button, computation, or interaction, trailing structure by 9.1–34.4 points. Web shows the same pattern using different test categories: static-content scores exceed interaction scores by 28.2–36.7 points across the four models. These gaps identify a consistent weakness among the scored outcomes: the evaluated recreations more often preserve visible interface elements than the state transitions and outputs produced by using them. Because categories contain different assertions and sometimes different application subsets, this comparison locates a suite-level gap rather than isolating interaction as its cause. The PhotoCollage audit on Ubuntu provides a concrete example (Appendix D.2.1). After a photo is rotated and the user invokes Undo, the recreation restores the photo arrangement but leaves the photo itself rotated. Its initial screen can still look correct, and the application still launches; only the action sequence exposes that the history state is incomplete. The case illustrates the distinction behind the aggregate category gap: a recreation can pass a launch or static-screen check while still implementing the wrong state transition. Controlled variation can expose input-dependent behavior. Same-field input variation appears in only 13.4–17.0% of the observed native trajectories (Figure 10(c)). Thus, the broader exploration measures above do not by themselves show that an agent has recovered input–output rules. The Privacy-Friendly Pedometer case on Android in Section 6.5 illustrates the complementary success mode: varying steps, 17

weight, and gender exposes persistence, conversion, and rounding behavior that a single static state cannot reveal. The last edit often reaches submission without a final check. The strict final edit–relaunch–inspection sequence appears in only 23.6–47.5% of trajectories (Figure 10(e)); its absence does not exclude validation before the final edit. The Plus Plus Battery case on Android shows the resulting blind spot (Appendix D.2.2). Its first launch exposes a missing cycle count and a broken row layout; the agent edits the Kotlin and XML but the retained trajectory ends before the patched application is relaunched and inspected. The inspected case artifact scores 100.0% visually but only 55.1% programmatically (Table 12), showing that a plausible final screen can coexist with substantial functional errors. Without rerunning the patched build, the agent has no opportunity to learn whether its final edits fix those errors or introduce new ones. Evaluation outcomes establish a consistent gap between static structure and action-conditioned behavior. Separately, trajectories show that same-field controlled input variation is uncommon and that the final edit is often not followed by relaunch and inspection; audited cases show what these process measurements mean in concrete successes and failures. These observations identify two concrete improvement targets— collect targeted evidence about state transitions and validate the exact artifact submitted—without claiming that the trajectory measurements explain the aggregate score gap. 6.4

Contamination and Evaluation Integrity

Benchmark scores would be misleading if a model recovered protected implementation artifacts during evaluation or reproduced a reference implementation from prior exposure. We audit these risks at two complementary levels: exact textual overlap in the delivered source and executable operations directed at enforced evaluation boundaries during rollout. The analyses use different cohorts and denominators, so we report them separately rather than collapse them into a single integrity rate.

Table 4: Noise-filtered exactline overlap (% of reference valid LOC), pooled over models. Platform

Median Maximum

Ubuntu macOS Windows Android

0.014% 0.060% 0.088% 0.044%

1.728% 3.188% 1.380% 1.866%

Noise-filtered exact-line overlap is small. On the four platforms with available reference source, we measure exact line-level overlap between each recreation and its corresponding reference for four models. After removing comments, recognizable interface and framework boilerplate, and common language syntax from both sides, we match identical lines one-to-one and divide the result by the reference’s original valid LOC. Pooling models within each platform, the median proportion of matched reference lines ranges from 0.014% on Ubuntu to 0.088% on Windows (Table 4), and every platform maximum remains below 3.2%. These consistently small values show that the delivered implementations are overwhelmingly textually distinct from their references rather than large verbatim copies. Inspection of the platform maxima likewise finds short API interactions, declarations, control fragments, and isolated pieces of domain logic rather than wholesale implementation reuse. Large-scale verbatim source recall is therefore unlikely to account for the observed benchmark performance. However, exact matching cannot detect memorization expressed through renamed identifiers, refactored control flow, reused interfaces, or short code fragments. We exclude Web from this analysis because its references are built snapshots and the served client is visible, making source-line overlap incomparable to that on the source-hidden native platforms. The complementary evaluation-hacking audit in Section 6.1.3 records executed operations matching detectors for external-network access, protected paths, installed reference packages, and privileged interfaces. Figure 12 reports attempts rather than successful access. Taken together, the two audits separate attempted operations from artifact-level outcome: agents sometimes issue evaluation-hacking attempts, while the delivered source shows no large-scale verbatim reproduction of the references. 6.5

Case Studies

The two cases below place the quantitative patterns in context without estimating their prevalence: one trajectory uses targeted probes to recover input-dependent behavior, while another uses shell scripts as a batch interface for both exploration and paired verification. Appendix D presents additional cases spanning specification recovery, failure-driven repair, layered verification, long-horizon state management, and integrity failures.

Measured comparisons close the refinement loop. In the Android Privacy-Friendly Pedometer task, GPT-6 Astra saves aligned screenshots and UI hierarchies, measures pixel error over the full application and named regions, continues implementation, and repeats the same comparison. The excerpt below preserves the original trajectory text; omitted spans and fields are marked in brackets. Selected transcript. Assistant: The skeleton builds offline. I’m comparing its first render. [Install, launch, screenshot, and UI−dump events omitted.] Command: /bin/bash −lc "python3 − <<'PY' from PIL import Image import numpy as np [UI−hierarchy and reference−color inspection omitted.] ref=np.array(Image.open('/workspace/recreation/reference_day.png').convert('RGB')).astype(int) clone=np.array(Image.open('/workspace/recreation/clone_day_1.png').convert('RGB')).astype(int) for label,region in [('app',(0,128,1080,2337)),('toolbar',(0,128,1080,401)),('summary',(21,422,1059,951)),('chart ',(21,972,1059,1854))]: left,top,right,bottom=region difference=np.abs(ref[top:bottom,left:right]−clone[top:bottom,left:right]) print(label,'MAE',round(difference.mean(),3),'within 8',round((difference.max(axis=2)<=8).mean()∗100,2)) PY" Tool result: [UI−hierarchy bounds and reference−color counts omitted.] app MAE 2.259 within 8 96.48 toolbar MAE 0.989 within 8 97.72 summary MAE 1.319 within 8 98.66 chart MAE 4.192 within 8 94.51 Assistant: The first render matches closely; I’m adding persistence and settings. [Implementation, rebuild, reinstall, and second screenshot capture omitted.] Command: /bin/bash −lc "python3 − <<'PY' from PIL import Image import numpy as np import xml.etree.ElementTree as ET reference=np.array(Image.open('/workspace/recreation/reference_day.png').convert('RGB')).astype(int) clone=np.array(Image.open('/workspace/recreation/clone_day_2.png').convert('RGB')).astype(int) for label, bounds in [('dashboard',(0,128,1080,2337)),('summary',(21,422,1059,951)),('chart',(21,972,1059,1854))]: left,top,right,bottom=bounds difference=np.abs(reference[top:bottom,left:right]−clone[top:bottom,left:right]) print(label, 'RGB MAE:', round(difference.mean(),3), 'pixels within 8:',round((difference.max(axis=2)<=8).mean() ∗100,2)) for filename in ['main_reference.xml','clone_day_2.xml']: root=ET.fromstring(open('/workspace/recreation/'+filename).read().split('</hierarchy>')[0]+'</hierarchy>') print(filename,'hierarchy nodes',len(list(root.iter('node')))) PY" Tool result: dashboard RGB MAE: 2.005 pixels within 8: 97.4 summary RGB MAE: 0.113 pixels within 8: 99.85 chart RGB MAE: 4.848 pixels within 8: 94.09 main_reference.xml hierarchy nodes 59 clone_day_2.xml hierarchy nodes 54 Assistant: Summary geometry now matches exactly. I’m refining dialogs and charts.

The aligned crops below are the exact summary region used in both comparisons.

Reference summary comparison target

First render: misplaced label

summary: 1.319 MAE; 98.66% within 8

After refinement: aligned label summary: 0.113 MAE; 99.85% within 8

The first comparison identifies a three-pixel vertical displacement and a misplaced Steps label in the summary card. After refinement, the summary region’s mean absolute RGB error falls from 1.319 to 0.113 and the fraction of pixels within tolerance rises from 98.66% to 99.85%. The rollout later applies controlled probes to behavior as well: it enters 0, 1, 1,000, 10,000, and 1,234 steps, checks persistence after relaunch, varies weight and gender, and corrects energy conversion and rounding. The submitted artifact 19

passes all 85 rule-based tests and 34 of 35 visual assertions, for 100.0% Prog and 97.1% VLM. This case documents a measured visual discrepancy, a subsequent localized correction, and a high-scoring artifact; the temporal sequence does not by itself establish the correction’s cause or estimate the population-level effect of pixel comparison. Batch exploration and reusable paired captures. GUI exploration and verification can combine dedicated tools with shell scripts that batch interactions and reuse checks. A separate paired comparison provides trajectories for both models on 249 application–platform pairs and gives the five platforms equal weight. Across all five platforms, Claude Opus 5 uses Bash more than Claude Opus 4.8. Averaged across platforms, it makes nearly three times as many Bash calls and allocates almost twice the share of its tool calls to Bash, while the share of dedicated GUI tools decreases (Table 5). Because the model version and recorded runtime differ, this comparison describes the observed workflows rather than isolating the cause of the change. In the Ubuntu TeXworks task, Claude Opus 5 uses Bash as a GUI exploration and verification interface. In Figure 18, it first batches eight menu captures, then reuses a capture script on both the reference and recreation. Claude Opus 4.8 instead opens menus and captures windows through separate desktop-control calls, while using Bash to process the saved screenshots (Appendix D.1.2). Each shell call can contain many GUI actions: exploration collects a sequence of menu states, and verification applies the same capture procedure to both applications. Dedicated GUI-tool counts alone therefore miss part of the agent’s GUI work.

Assistant: ImageMagick is available for fast root captures. Let me script the menu walk with keyboard navigation. [Bash: reference exploration] cd /tmp/shots && for i in 1 2 3 4 5 6 7 8; do xdotool key −−window 41943046 Right 2>/dev/null || xdotool key Right sleep 0.5 import −window root /tmp/shots/menu_$i.png done; ls /tmp/shots/ [Result lists menu_1.png through menu_8.png.] [Later Bash call writes and runs /tmp/menucap.sh. Window activation and coordinate lookup omitted.] for i in 0 1 2 3 4 5 6 7; do import −window root /tmp/shots/${PFX}_$i.png xdotool key Right; sleep 0.7 done echo "mine:"; /tmp/menucap.sh 0x00c00006 mm echo "ref:"; /tmp/menucap.sh 0x02800006 rr

Figure 18: Claude Opus 5 batches menu exploration and paired captures for TeXworks on Ubuntu. Selected trajectory excerpts; setup omitted as marked. Table 5: Mean tool use across five platforms, weighted equally. Parentheses give mean per-trajectory shares. GUI MCP includes desktop, mobile, and browser tools; Bash includes all purposes, including builds and edits. Model Claude Opus 4.8 Claude Opus 5

Bash calls (share)

GUI MCP calls (share)

61.2 (23.6%) 179.6 (45.5%)

106.3 (40.6%) 108.1 (28.6%)

Related Work

Scalable environments for agentic tasks. Scalable agent training requires diverse executable tasks, reusable environments, and reliable feedback on outcomes. Gym-Anything (Aggarwal et al., 2026) automates software setup and auditing, producing long-horizon tasks and trajectories for distillation. CUA-Gym (Wang et al., 2026) jointly generates task instructions, environment states, and executable reward functions, and synthesizes mock web applications to expand its training environment pool. GUIGENESIS (Cao et al., 2026) reconstructs real applications as lightweight environments with code-native rewards, while InfiniteWeb (Zhang et al., 2026), AutoWebWorld (Wu et al., 2026), and VeriEnv (Chae et al., 2026) synthesize or recreate functional websites with programmatic verification. OSGym (Qin et al., 2026) addresses the infrastructure cost and reliability of parallel operating-system rollouts. Complementary efforts scale the experience used to learn agent policies: OpenCUA (Wang et al., 2025b) builds crossplatform demonstrations and reflective trajectories, while OS-ATLAS (Wu et al., 2024) develops crossplatform GUI grounding data and action models. UI-TARS (Qin et al., 2025) develops a native GUI agent with screenshot-based perception, unified action modeling, and iterative training on reflective interaction traces; UI-TARS-2 (Wang et al., 2025a) extends this line with multi-turn reinforcement learning and a hybrid GUI–terminal sandbox. ComputerRL (Lai et al., 2025) scales online reinforcement learning over hybrid API–GUI desktops, and SCALECUA (Lv et al., 2026) couples verifiable task synthesis with efficient online reinforcement learning. EvoCUA (Xue et al., 2026) combines verifiable task synthesis, large-scale rollouts, and iterative policy improvement. Qwen-CUA (Lu et al., 2026) scales screenshotonly native computer use with long-horizon verifiable tasks and also studies Bash-augmented operation. R ECREATION W ORLD connects scalable task construction, automatic verification, and trajectory generation within a common framework for agent training and evaluation. Executable references provide behavioral supervision without prescribing an implementation, while reproducible environments support diverse, long-horizon experience that can be scored and selected by its outcomes. Our transfer experiments on 20

external coding and computer-use benchmarks provide initial evidence that this experience develops agent capabilities beyond the source tasks. Hybrid computer use. OSWorld (Xie et al., 2024) and OSWorld 2.0 (Yuan et al., 2026) evaluate agents operating desktop applications, while WebArena (Zhou et al., 2024) and AndroidWorld (Rawles et al., 2025) provide web and mobile environments with execution- and state-based evaluation. WeaveBench (Li et al., 2026b) and PhoneHarness (Li et al., 2026a) explicitly study workflows that coordinate GUI interaction with command-line, code, or structured tool actions. CoAct-1 (Song et al., 2025) delegates between GUI operation and executable code, whereas StateAct (Yang et al., 2026b) makes program-state access primary and reserves GUI interaction for visually grounded subgoals. Qwen-UI-Agent (Zhou et al., 2026) learns mixed GUI and command-line actions across mobile, desktop, and web environments. EdgeBench (Zhu et al., 2026) measures within-run improvement over day-scale executable tasks, while Agents’ Last Exam (Sun et al., 2026) evaluates expert-authored professional workflows with verifiable outcomes. R ECREATION W ORLD focuses on autonomous coordination of the explore–implement–verify loop: observations inform code and tool use, while executing and inspecting the agent’s own outputs reveals what to explore or revise next. Its unified GUI–code harness and five-platform benchmark, R ECREATION B ENCH, provide a reproducible setting for studying this long-horizon coordination across heterogeneous environments, including whether agents use execution and visual feedback to correct their own outputs. Visual and interactive coding. Visual coding connects visual observations with code generation, modification, and verification. Design2Code (Si et al., 2025) and DesignBench (Xiao et al., 2025) evaluate screenshot-conditioned front-end generation, while Web2Code (Yun et al., 2024) and WebCode2M (Gui et al., 2024) provide large-scale data for visual front-end development. SWE-bench Multimodal (Yang et al., 2025) extends this scope to bug fixing in visual JavaScript software, with images in problem statements or unit tests. Interaction2Code (Xiao et al., 2024) extends evaluation to dynamic webpage interactions, and WebGen-Bench (Lu et al., 2025) evaluates instruction-driven website construction through agent-executed functional tests. Vision2Web (He et al., 2026) spans static pages, interactive multi-page frontends, and full-stack development, combining GUI-agent verification with visual judging. Beyond websites, APPFORGE (Ran et al., 2025) studies Android application development from naturallanguage specifications and constructs functional tests by navigating reference applications, followed by expert verification; RealDevWorld (Bian et al., 2025) evaluates generated software through interactive GUI testing of function, appearance, and runtime behavior; and GameCraft-Bench (Luo et al., 2026) evaluates complete game artifacts through executable gameplay and multimodal assessment. These works establish functional behavior and visual fidelity as complementary evaluation targets. R ECREATION W ORLD extends visual coding to tasks in which the target must be actively discovered through interaction. Agents infer visual and behavioral requirements from a running reference, then execute and inspect their own outputs to guide further exploration and revision. Frozen, reference-validated programmatic and visual tests score both appearance and action-conditioned behavior, allowing diverse implementations to be evaluated without prescribing their architecture or the agent’s workflow.

Conclusion and Limitations

We introduced R ECREATION W ORLD, a framework for studying hybrid computer-use agents that must move repeatedly among GUI exploration, implementation, execution, and verification. Across Ubuntu, macOS, Windows, Android, and Web, application recreation turns a running reference into both an interactive specification and a source of automatically scored training experience. Reference-grounded programmatic and visual tests measure action-conditioned behavior without prescribing the candidate’s implementation. This combination makes it possible to study the full interface–code loop in reproducible environments rather than evaluating GUI operation and software construction in isolation. Using these environments, we selected a balanced set of 35,000 trajectories, with 7,000 from each platform, to train two model initializations. Both training runs improved from their first evaluated checkpoints across five out-of-distribution coding and hybrid computer-use benchmarks, with gains of up to 17.9 percentage points, while their trajectories shifted toward more active code and visual verification and increased GUI interaction. For held-out evaluation, R ECREATION B ENCH contributes 250 applications, 50 per platform. Frontier agents remain well below reference fidelity: interactions and computed outputs are harder to reproduce than static interface structure, and recreated applications remain smaller and more structurally concentrated than their references. We release the benchmark, environments, and test suites to support further work on agents that can use execution feedback to improve what they build. Limitations. R ECREATION B ENCH emphasizes pinned, reproducible environments, which limits coverage of workflows involving live services, changing external state, or real-user interaction. Its hidden 21

suites sample a finite set of visual and behavioral outcomes, so passing them cannot establish equivalence across all states and interaction paths. Public reference implementations may also have appeared in pretraining data, and source-overlap screening cannot rule out prior memorization or all forms of source reuse. The scores therefore measure fidelity within the specified environments and frozen test suites. Future work could incorporate controlled online and multi-user workflows, expand hidden-test coverage across states and interaction paths, and introduce newly constructed references to reduce contamination risk.

Contributors1 Shuai Bai, Jiayong Deng, Yikun Fu, Chang Gao, Xuhao Hu, Mianqiu Huang, Yizhen Jiang, Yuheng Jing, Dehui Kong, Keliang Li, Ning Li, Wanli Li, Dayiheng Liu, Dunjie Lu, Changwei Luo, Que Shen, Zheyuan Wang, Zijian Wang, Jie Wu, Gao Wu, Zhihui Xie, Rui Xie, Haiyang Xu, An Yang, Jiakang Yuan, Yanming Zhang, Jiajun Zhang, Xi Zhang, Zhenru Zhang, Zhuo Zhen, Mingkang Zhu, Bowen Zhou.

1 Contributors are listed in alphabetical order by last name.

References Pranjal Aggarwal, Graham Neubig, and Sean Welleck. Gym-anything: Turn any software into an agent environment, 2026. URL https://arxiv.org/abs/2604.06126. Anthropic. Claude opus 4.8 system card, 2026a. URL https://www.anthropic.com/ claude-opus-4-8-system-card. Official system card, May 2026. Anthropic. Claude opus 5 system card, 2026b. URL https://www.anthropic.com/ claude-opus-5-system-card. Official system card, July 2026. Anthropic. Eval awareness in Claude Opus 4.6’s BrowseComp performance, March 2026c. URL https: //www.anthropic.com/engineering/eval-awareness-browsecomp. Anthropic Engineering, March 6, 2026. Yutong Bian, Xianhao Lin, Yupeng Xie, et al. You don’t know until you click: Automated GUI testing for production-ready software evaluation, 2025. URL https://arxiv.org/abs/2508.14104. Yuan Cao, Dezhi Ran, Mengzhou Wu, et al. GUI-GENESIS: Automated synthesis of efficient environments with verifiable rewards for GUI agent post-training, 2026. URL https://arxiv.org/abs/2602.14093. Hyungjoo Chae, Jungsoo Park, and Alan Ritter. Safe and scalable web agent learning via recreated websites, 2026. URL https://arxiv.org/abs/2603.10505. Google DeepMind. Gemini 3.7 Flash model card, 2026. URL https://deepmind.google/models/ model-cards/gemini-3-7-flash/. Official model card, updated August 13, 2026. Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Yi Su, Bohua Chen, Dongping Chen, Siyuan Wu, Xing Zhou, Wenbin Jiang, Hai Jin, and Xiangliang Zhang. Webcode2m: A real-world dataset for code generation from webpage designs, 2024. URL https://arxiv.org/abs/2404.06369. Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, and Jie Tang. Vision2web: A hierarchical benchmark for visual website development with agent verification, 2026. URL https: //arxiv.org/abs/2603.26648. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024. URL https://arxiv.org/abs/2310.06770. Kimi Team. Kimi k3: Open frontier intelligence, 2026. URL https://arxiv.org/abs/2607.24653. Hanyu Lai, Xiao Liu, Yanxiao Zhao, et al. ComputerRL: Scaling end-to-end online reinforcement learning for computer use agents, 2025. URL https://arxiv.org/abs/2508.14040. Chenxin Li, Zhengyao Fang, Zhengyang Tang, Pengyuan Lyu, Xingran Zhou, Xin Lai, Fei Tang, Liang Wu, Yiduo Guo, Weinong Wang, Junyi Li, Yi Zhang, Yang Ding, Huawen Shen, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Chengquan Zhang, and Han Hu. Phoneharness: Harnessing phone-use agents through mixed gui, cli, and tool actions, 2026a. URL https://arxiv. org/abs/2606.14832. Wanli Li, Bowen Zhou, Yunyao Yu, Zhou Xu, Yifan Yang, Dongsheng Li, and Caihua Shan. Weavebench: A long-horizon, real-world benchmark for computer-use agents with hybrid interfaces, 2026b. URL https://arxiv.org/abs/2606.09426. Dunjie Lu, Shuai Bai, Tianyi Bai, et al. Qwen-cua: Native computer use for (almost) everything, 2026. URL https://arxiv.org/abs/2608.02352. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch, 2025. URL https://arxiv.org/abs/2505.03733. Tongxu Luo, Rongsheng Wang, Jiaxi Bi, Chenming Xu, Zhengyang Tang, Jianlong Chen, Juhao Liang, Ke Ji, Shuqi Guo, Yuhao Du, Fan Bu, Wenyu Du, Xiaotong Zhang, Kyle Li, Shaobo Wang, Linfeng Zhang, Yuxuan Liu, Xin Lai, Chenxin Li, Yiduo Guo, Zhexin Zhang, Xinyuan Wang, Tianyi Bai, Ziniu Li, and Benyou Wang. Gamecraft-bench: Can agents build playable games end-to-end in a real game engine?, 2026. URL https://arxiv.org/abs/2606.17861. Bowen Lv, Xiao Liu, Yanyu Ren, et al. SCALECUA: Scaling computer use agents with verifiable task synthesis and efficient online RL, 2026. URL https://arxiv.org/abs/2607.11185. 23

OpenAI. GPT-5.6 system card, 2026. URL https://deploymentsafety.openai.com/gpt-5-6. Official system card. OpenRouter. Models and pricing, 2026. URL https://openrouter.ai/models. Accessed September 11, 2026. Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, and Guang Shi. UI-TARS: Pioneering automated GUI interaction with native agents, 2025. URL https://arxiv.org/abs/2501.12326. Zengyi Qin, Jinyuan Chen, Yunze Man, Shengcao Cao, Ziqi Pang, Zhuoyuan Wang, Han Fang, Ling Zhu, Zixin Xie, Zibu Wei, Tianshu Ran, Haoran Geng, Ray Pan, Qizhen Sun, Zachary Bright, Yuyang Cai, Chongye Yang, Jiace Zhao, Tianrui Liu, Han Cao, Yeyang Zhou, Rui Wang, Song Wang, Xiang Ren, Bo Zhang, Yutong Ban, Pieter Abbeel, and Brian Anthony. Osgym: Scalable os infra for computer use agents, 2026. URL https://arxiv.org/abs/2511.11672. Qwen Team. Qwen3.7-Plus: Multimodal agent intelligence, 2026a. URL https://qwen.ai/blog?id= qwen3.7-plus. Official model release. Qwen Team. Qwen3.8-Max: A new bar for coding and cowork, 2026b. URL https://qwen.ai/blog?id= qwen3.8. Official model release. Dezhi Ran, Yuan Cao, Mengzhou Wu, Simin Chen, Yuzhe Guo, Jun Ren, Zihe Song, Hao Yu, Jialei Wei, Linyi Li, et al. Appforge: From assistant to independent developer–are gpts ready for software development? arXiv preprint arXiv:2510.07740, 2025. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. Androidworld: A dynamic benchmarking environment for autonomous agents, 2025. URL https://arxiv.org/abs/2405.14573. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: Benchmarking multimodal code generation for automated front-end engineering, 2025. URL https://arxiv.org/abs/2403.03163. Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J. Liu, James Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi, Abhishek Kumar, Alex Alemi, Alex Rizkowsky, Azade Nova, Ben Adlam, Bernd Bohnet, Gamaleldin Elsayed, Hanie Sedghi, Igor Mordatch, Isabelle Simpson, Izzeddin Gur, Jasper Snoek, Jeffrey Pennington, Jiri Hron, Kathleen Kenealy, Kevin Swersky, Kshiteej Mahajan, Laura Culp, Lechao Xiao, Maxwell L. Bileschi, Noah Constant, Roman Novak, Rosanne Liu, Tris Warkentin, Yundi Qian, Yamini Bansal, Ethan Dyer, Behnam Neyshabur, Jascha Sohl-Dickstein, and Noah Fiedel. Beyond human data: Scaling self-training for problem-solving with language models, 2024. URL https://arxiv.org/abs/2312.06585. Linxin Song, Yutong Dai, Viraj Prabhu, et al. CoAct-1: Computer-using multi-agent system with coding actions, 2025. URL https://arxiv.org/abs/2508.03923. Yiyou Sun, Xinyang Han, Weichen Zhang, et al. Agents’ last exam, 2026. URL https://arxiv.org/abs/ 2606.05405. Bowen Wang, Dunjie Lu, Junli Wang, Tianyi Bai, Shixuan Liu, Zhipeng Zhang, Haiquan Wang, Hao Hu, Tianbao Xie, Shuai Bai, Dayiheng Liu, Que Shen, Junyang Lin, and Tao Yu. Cua-gym: Scaling verifiable training environments and tasks for computer-use agents, 2026. URL https://arxiv.org/abs/2605. 25624. Haoming Wang, Haoyang Zou, Huatong Song, et al. UI-TARS-2 technical report: Advancing GUI agent with multi-turn reinforcement learning, 2025a. URL https://arxiv.org/abs/2509.02544. Xinyuan Wang, Bowen Wang, Dunjie Lu, et al. OpenCUA: Open foundations for computer-use agents, 2025b. URL https://arxiv.org/abs/2508.09123. Jason Wei. Asymmetry of verification and verifier’s rule. https://www.jasonwei.net/blog/ asymmetry-of-verification-and-verifiers-law, July 2025. Blog post, published July 15, 2025. Yifan Wu, Yiran Peng, Yiyu Chen, et al. AutoWebWorld: Synthesizing infinite verifiable web environments via finite state machines, 2026. URL https://arxiv.org/abs/2602.14296. 24

Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. Os-atlas: A foundation action model for generalist gui agents, 2024. URL https://arxiv.org/abs/2410.23218. xAI. Model card: Grok 4.6, 2026. URL https://media.x.ai/v1/website/card-4p6-4cd2dc57.pdf. Official model card. Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R. Lyu. Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping, 2024. URL https://arxiv.org/abs/2411.03292. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R. Lyu. Designbench: A comprehensive benchmark for mllm-based front-end code generation, 2025. URL https://arxiv.org/abs/2506.06251. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024. URL https://arxiv.org/abs/2404.07972. Taofeng Xue, Chong Peng, Mianqiu Huang, Linsen Guo, Tiancheng Han, Haozhe Wang, Jianing Wang, Xiaocheng Zhang, Xin Yang, Dengchang Zhao, Jinrui Ding, Xiandi Ma, Yuchen Xie, Peng Pei, Xunliang Cai, and Xipeng Qiu. Evocua: Evolving computer use agents via learning from scalable synthetic experience, 2026. URL https://arxiv.org/abs/2601.15876. John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWEbench multimodal: Do ai systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=riTiq3i21b. John Yang, Kilian Lieret, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, and Ofir Press. Programbench: Can language models rebuild programs from scratch?, 2026a. URL https://arxiv.org/abs/2605.03546. Yan Yang, Xiangru Jian, Ziyang Luo, et al. StateAct: Program state, before pixels, for long-horizon computer-use agents, 2026b. URL https://arxiv.org/abs/2607.22798. Mengqi Yuan, Zilong Zhou, Xinzhuang Xiong, Weiming Wu, Jiayang Sun, Jiamin Song, Kaiqian Cui, Bowen Wang, Haoyuan Wu, Yitong Li, Dunjie Lu, Haikong Lu, Qi Zhen, Xinyuan Wang, Jiaqi Deng, Yuhao Yang, Cheng Chen, Boyuan Zheng, Alex Su, Xiao Yu, Hao Zou, Saaket Agashe, Xing Han Lu, Manpreet Kaur, Zhengyang Qi, Vincent Sunn Chen, Frederic Sala, Dayiheng Liu, Junyang Lin, Zhou Yu, Yu Su, Siva Reddy, Xin Eric Wang, Peng Qi, Tianbao Xie, and Tao Yu. Osworld 2.0: Benchmarking computer use agents on long-horizon real-world tasks, 2026. URL https://arxiv.org/abs/2606. 29537. Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, and Zhiqiang Shen. Web2code: A large-scale webpageto-code dataset and evaluation framework for multimodal llms, 2024. URL https://arxiv.org/abs/ 2406.20098. Z.ai. GLM-5.3: Frontier coding with emergent cyber capabilities, 2026. URL https://z.ai/blog/glm-5. 3. Official technical report. Ziyun Zhang, Zezhou Wang, Xiaoyi Zhang, et al. InfiniteWeb: Scalable web environment synthesis for GUI agent training, 2026. URL https://arxiv.org/abs/2601.04126. Hanzhang Zhou, Panrong Tong, Xu Zhang, et al. Qwen-UI-Agent technical report: Toward nextgeneration real-world centric foundation GUI agents, 2026. URL https://arxiv.org/abs/2607. 28227. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024. URL https://arxiv.org/abs/2307.13854. Deyao Zhu, Xin Zhou, Shengling Qin, et al. Edgebench: Unveiling scaling laws of learning from real-world environments, 2026. URL https://arxiv.org/abs/2607.05155.

Appendix Contents A Release Artifacts and Reproducibility

A.1 Public Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.2 Licensing and Redistribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.3 Relation to the In-house Benchmark . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.4 RecreationBench Evaluation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.5 Out-of-Distribution Evaluation Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

A.6 Behavioral Metric Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

B Per-Platform Results

C Benchmark and Environment Details

C.1 Source-Size Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.2 Platform Runtime and Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.3 Benchmark App Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.4 Test Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.5 Concrete Test Cases Across Five Platforms . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.6 Isolation and Sandboxing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

C.7 Agent Harness Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D Trajectory Case Studies

D.1 Exploration and Specification Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.2 Repair Driven by Observed Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.3 Layered Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.4 Long-Horizon State Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.5 Integrity Failures in Web Recreation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

D.6 Cross-Case Observations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Release Artifacts and Reproducibility

We release the resources needed to inspect, run, and reproduce R ECREATION W ORLD. These resources include the project website, an interactive application-comparison interface, the recreation and evaluation code, the benchmark dataset, and the execution environments. Table 6 provides the corresponding public entry points. A.1

Public Resources

A.2

Licensing and Redistribution

R ECREATION W ORLD is a composite release rather than a single-license archive. Project-authored framework and benchmark files will carry the licenses declared at their release roots, while each third-party reference remains under its upstream terms; inclusion in the benchmark does not relicense that material or grant rights to upstream names, logos, or other trademarks. For every task, the Hugging Face release will provide machine-readable task metadata containing the source repository, pinned revision, exact SPDX expression (or a scoped LicenseRef), applied patches, and paths to the corresponding license and notice files. When source or a runnable reference is redistributed, those files and attribution notices are included beside it.

Table 6: Public resources accompanying R ECREATION W ORLD. Resource

Contents

Access

Project website Interactive comparison

Benchmark overview, public leaderboard, and visualizations Side-by-side interaction with selected reference and recreated applications Recreation harness, platform runners, evaluation code, configurations, and reproduction instructions Public task metadata, prompts, and reference repositories Mirror of the public RecreationBench dataset

Website Live comparison

GitHub repository Hugging Face dataset ModelScope dataset

GitHub Hugging Face ModelScope

Table 7: License audit of the 250 selected reference tasks. Counts are per task, so a repository represented on two platforms is counted twice. The family labels group -only and -or-later variants for readability; the released metadata preserve the exact expression for each pinned artifact. Platform

Tasks

Ubuntu

macOS

Windows

Android

Web

License distribution of reference artifacts GPL-2.x family (19); GPL-3.x family (19); MIT (9); Artistic-2.0 (1); MPL-2.0 OR GPL-3.0-or-later (1); GPL-3.0-or-later with additional project terms (1). MIT (22); GPL-3.x family (10); AGPL-3.0 family (6); GPL-2.x family (4); LGPL family (3); Apache-2.0 (1); BSD family (1); CC0-1.0 (1); MPL-2.0 OR GPL-3.0-or-later (1); modified MIT terms (1). MIT (20); GPL-3.x family (17); Apache-2.0 (4); GPL-2.x family (3); BSD family (2); AGPL-3.0 family (1); LGPL family (1); MPL-2.0 (1); MPL-2.0 OR LGPL-family terms (1). GPL-3.x family (33); Apache-2.0 (8); MIT (6); BSD-3-Clause (1); AGPL-3.0 family (1); Unlicense (1). Benchmark-authored sites (44): MIT code AND CC-BY-4.0 authored content and evaluation data. Openly licensed sites (6): MIT AND CC-BY-4.0, MIT, Apache-2.0, PostgreSQL, MIT OR Apache-2.0, and CC-BY-4.0 (one each).

The six openly licensed Web snapshots are, respectively, climatewatch.org (MIT AND CC-BY-4.0), detox.github.io (MIT), keycloak.org (Apache-2.0), postgresql.org (PostgreSQL), rust-lang.org (MIT OR Apache-2.0, with CC-BY-3.0 documentation where marked), and webpack.js.org (CC-BY-4.0). Their release copies remove standalone brand marks, third-party promotional media, and images of identifiable people, replace them with synthetic artwork under CC0-1.0, substitute fonts with OFL-1.1licensed alternatives, and record these modifications in per-site NOTICE files. For the 44 benchmarkauthored .example sites, authored implementation code is MIT, whereas authored page content, task metadata, ground truth, and evaluation files are CC-BY-4.0. Bundled fonts, icons, libraries, and media retain their own licenses in both groups. Compound and project-specific terms are not flattened in the released task metadata. SQLiteBrowser retains its MPL-2.0 OR GPL-3.0-or-later choice, ATSynEdit its MPL/LGPL choice, and Letos its GPL3.0 OpenSSL exception; Sigma File Manager carries GPL-3.0-or-later together with its additional rules, while JSONExport uses modified MIT terms that require attribution when incorporated into paid software. A.3

Relation to the In-house Benchmark

The version of R ECREATION B ENCH used in this paper supersedes the preliminary in-house version reported in our earlier blog post2 . We revised the benchmark application set to provide broader and more balanced coverage of task domains, implementation stacks, source sizes, and interaction complexity. Applications with limited, unstable, or insufficiently discriminative behavior were replaced with higherquality references. We also regenerated the evaluation suites to exercise deeper multi-step interactions, state transitions, persistence, and end-to-end outputs, while strengthening reference validation and filtering to remove flaky, redundant, or shallow checks. Together, these changes make the revised benchmark more challenging and more discriminative, and therefore better suited to tracking the capabilities of frontier models on hybrid computer-use tasks. Because both the application set and the evaluators changed, scores reported in the blog are not directly comparable with those in this paper; the results here correspond to the revised benchmark described in this version. 2 https://qwen.ai/blog?id=qwen3.8

A.4

RecreationBench Evaluation Settings

Table 8 summarizes the evaluated models and inference settings, and Table 9 records the interaction servers used on each platform. The Codex runs use version 0.145.0. Each recreation rollout has a 20-hour wall-clock budget. Table 8: Models and inference configurations evaluated in R ECREATION B ENCH (OpenAI, 2026; Anthropic, 2026b;a; Google DeepMind, 2026; Kimi Team, 2026; Z.ai, 2026; xAI, 2026; Qwen Team, 2026b;a). Model ID

Agent scaffold

Context window

Max output

Reasoning setting

gpt-6-astra gpt-5.6-sol

Codex Codex

272k 272k

128k 128k

max max

claude-opus-5 claude-opus-4-8 gemini-3.7-flash kimi-k3 glm-5.3 grok-4.6 qwen3.8-max-0902 qwen3.7-plus

Claude Code Claude Code Claude Code Claude Code Claude Code Claude Code Claude Code Claude Code

1M 1M 1M 1M 1M 500k 256k 256k

128k 128k 64k 128k 128k 128k 32k 32k

max max high max max xhigh xhigh enabled

Table 9: Interaction servers pinned by the released R ECREATION W ORLD configuration. Behavioral assertions use the platform-native automation substrates independently. Ubuntu

macOS

Windows

Android

Web

Control server qwen-cua-driver qwen-cua-driver qwen-cua-driver mobile-mcp @playwright/mcp Interface Direct MCP Direct MCP Direct MCP Direct MCP Direct MCP Version3 0.7.3 0.7.3 0.7.3 0.1.5 0.0.79

A.5

Out-of-Distribution Evaluation Settings

The transfer experiments in Section 3.2 evaluate each checkpoint once per task using a Claude Code scaffold with benchmark-specific tools and budgets. Figure 5 reports each benchmark’s graded task score on a 0–100 scale. All output-token caps below apply to individual model requests. ProgramBench. We evaluate the 200 tasks in the full split using Claude Code inside a cleanroom container, with shell execution and file editing for implementation and testing. The configuration permits 2,000 turns, caps output at 64,000 tokens per request, and sets a 24-hour container timeout; the task prompt specifies a budget of 1,000 steps and six hours. We report Partial, the macro-average of the fraction of tests passed within each task. GameCraft-Bench. We evaluate 140 tasks with Harbor’s LocalClaudeCode harness, using native shell and file tools without cua-driver. The agent timeout is six hours, with a separate two-hour verification timeout and GPT-5.5 as the rubric judge. We report Overall, which averages the build-gated task rubric over mechanics, content depth, functional visuals, and art. Vision2Web. We evaluate 193 tasks through its Claude Code adapter: 100 Level-1 static-page tasks, 66 Level-2 frontend tasks, and 27 Level-3 website tasks. The inference timeout is eight hours, and the configured output caps are 64,000 tokens for Qwen3.7-Plus and 128,000 for Qwen-Flash-CPT. The GUI evaluator and visual judge use GPT-5.4. Each Level-1 task receives its visual score (VS), while each Level-2 or Level-3 task receives the mean of its visual and functional scores, (VS + FS)/2. The reported overall score averages these task scores across all three levels. 3 Pinned interaction-server versions: Qwen CUA driver 0.7.3 (https://github.com/QwenLM/qwen-code/ tree/cua-driver-rs-v0.7.3/packages/cua-driver); Qwen mobile MCP 0.1.5 (https://github.com/QwenLM/ qwen-code/tree/cb483092561a3606dfd53447a26f3718380f0f59/packages/mobile-mcp); and Playwright MCP 0.0.79 (https://github.com/microsoft/playwright-mcp).

OSWorld 2.0. We evaluate 108 tasks from the v2 split using Claude Code with cua-driver 0.19.0 in a hybrid setting. The agent can use GUI, browser, and accessibility tools alongside shell execution and direct file editing. It has a five-hour execution timeout and a 128,000-token output cap per request. We report the mean task-specific partial score, using GPT-5.5 for model-based evaluation checks. WeaveBench. We evaluate the 114 tasks in the v2 test split using Claude Code with cua-driver 0.7.1, retaining native shell and file tools alongside GUI interaction. The agent timeout is five hours and the output cap is 32,000 tokens per request. We report Overall, the mean shortcut-audited task score produced by its GPT-5.5 judge. The recorded runs include differences in budgets and harness revisions. We therefore interpret the curves as exploratory evidence of transfer under these settings. A.6

Behavioral Metric Definitions

Section 3.3 reports three behavioral ratios, each measured on the benchmark category where it is most relevant. This section specifies how each ratio and its sub-types are computed from the raw tool-call trajectories. Code verification (ProgramBench). A Bash tool call counts as code verification if it invokes a known test runner or executes a script file that the agent itself created during the current trajectory. Test-runner detection covers 62 head patterns across 12 language ecosystems, listed in Table 10. A script counts as agent-created if its file path was written via an Edit or Write tool call, produced by a shell heredoc, or resides under /tmp/. To avoid false positives, classification operates on execution-position heads of shell fragments rather than on substrings of the full command. Figure 6 further divides verification calls by timing: pre-edit calls occur before the first Edit or Write in a trajectory, and post-edit calls occur after it. The denominator is all Bash calls pooled over the observed ProgramBench trajectories, not all tool calls. This split concerns the first source edit, not verification after the final mutation. Table 10: Complete list of test-runner head patterns used to identify code verification calls (62 patterns across 12 ecosystem groups). Each pattern is matched against the execution-position head of a shell fragment after stripping transparent wrappers. Ecosystem

Matched head patterns

Python JavaScript / TS Rust Go C / C++ / Make Ruby PHP .NET Java / JVM Swift / Zig / Elixir Lua Shell conventions

pytest, py.test, python -m pytest, python -m unittest, python -m nose2, python -m tox, python -m nox, tox, nox npm test, pnpm test, yarn test, bun test, jest, mocha, vitest, ava, tap, karma, playwright, cypress cargo test, cargo nextest go test, gotestsum make test, make tests, make check, make checks, gmake test, gmake check, ctest rspec, minitest, rake test, bin/rails test, rails test phpunit, vendor/bin/phpunit dotnet test mvn test, gradlew test, gradlew check, ./gradlew test, ./gradlew check swift test, zig build test, mix test busted, luaunit bats, ./test.sh, ./tests.sh, ./run tests, run tests.sh, run tests.py, test.py, scripts/run tests, scripts/run-tests, scripts/test.sh, bash tests.sh, sh test.sh

Visual verification (GameCraft-Bench, Vision2Web). Every tool call that reads an image file is counted, including explicit Read calls on files with image extensions such as .png and .jpg, as well as screenshot tool calls that return an image in the tool result. The ratio is ∑ image reads/ ∑ all tool calls. The sub-type split classifies each image read by its file path. Reads whose path matches a whitelist of reference directories, including /workspace/assets/, /prototypes/, and /resources/, are labeled asset. All remaining reads, which typically correspond to the agent’s own rendered output, screenshots, or files written to /tmp/, are labeled render. GUI actions (OSWorld 2.0, WeaveBench). Every tool call whose name carries an MCP CUA prefix is counted. The two recognized prefixes are mcp cua-computer-use * and mcp cua *. The ratio is ∑ GUI calls/ ∑ all tool calls. The sub-type split distinguishes observe from interact. Observe covers state-reading tools such as get window state, get desktop state, and screenshot. Interact covers state-changing tools such as click, type text, press key, hotkey, and scroll, as well as GUI utility operations such as application launch, window resizing, and configuration changes.

Per-Platform Results

Table 11 expands the aggregate results in Table 3 into the five constituent platforms. Table 11: Per-platform results on R ECREATION B ENCH. The average score is the mean of Prog and VLM; threshold rows report application-level Prog coverage. Platform / metric

Gemini 3.7 Kimi Qwen3.7- Grok Qwen3.8- Claude Claude GPT-5.6 GPT-6 GLM-5.3 Flash K3 Plus 4.6 Max-0902 Opus 5 Opus 4.8 Sol Astra

Ubuntu Programmatic score (%) VLM score (%) Average score (%) Prog ≥ 90% (% apps) Prog = 100% (% apps)

18.15 6.92 12.54 0.00 0.00

28.03 23.39 25.71 0.00 0.00

15.33 10.46 12.90 0.00 0.00

9.28 2.23 5.76 0.00 0.00

28.25 26.64 27.45 2.00 0.00

29.15 28.71 28.93 2.00 0.00

35.83 30.75 33.29 2.00 0.00

24.04 18.15 21.10 2.00 2.00

37.45 35.91 36.68 8.00 0.00

54.34 52.64 53.49 14.00 2.00

macOS Programmatic score (%) VLM score (%) Average score (%) Prog ≥ 90% (% apps) Prog = 100% (% apps)

24.26 13.04 18.65 0.00 0.00

30.79 19.94 25.37 4.00 0.00

27.83 20.88 24.36 0.00 0.00

9.58 5.92 7.75 0.00 0.00

33.63 24.06 28.85 0.00 0.00

41.41 29.19 35.30 2.00 0.00

39.80 26.44 33.12 2.00 0.00

33.35 25.81 29.58 0.00 0.00

40.62 34.53 37.58 4.00 2.00

53.71 42.65 48.18 20.00 6.00

Windows Programmatic score (%) VLM score (%) Average score (%) Prog ≥ 90% (% apps) Prog = 100% (% apps)

13.29 7.54 10.42 0.00 0.00

29.31 26.60 27.96 2.00 0.00

23.16 16.08 19.62 2.00 0.00

9.06 7.09 8.08 0.00 0.00

36.13 32.13 34.13 4.00 0.00

31.47 26.27 28.87 4.00 0.00

50.33 46.28 48.31 8.00 4.00

35.05 31.00 33.03 4.00 0.00

47.85 46.59 47.22 10.00 0.00

61.41 57.80 59.61 22.00 4.00

Android Programmatic score (%) VLM score (%) Average score (%) Prog ≥ 90% (% apps) Prog = 100% (% apps)

34.46 34.05 34.26 0.00 0.00

43.32 50.55 46.94 2.00 0.00

20.54 22.34 21.44 2.00 0.00

9.23 18.82 14.03 0.00 0.00

51.60 57.84 54.72 0.00 0.00

41.09 48.20 44.65 2.00 0.00

50.56 56.74 53.65 2.00 0.00

33.01 40.51 36.76 0.00 0.00

50.00 56.93 53.47 2.00 0.00

62.95 64.98 63.97 26.00 2.00

Web Programmatic score (%) VLM score (%) Average score (%) Prog ≥ 90% (% apps) Prog = 100% (% apps)

34.38 25.14 29.76 12.00 0.00

28.90 33.23 31.07 2.00 0.00

44.66 42.52 43.59 6.00 0.00

8.74 11.56 10.15 0.00 0.00

45.50 31.56 38.53 2.00 0.00

34.54 37.99 36.27 0.00 0.00

53.44 51.47 52.46 13.64 0.00

36.54 33.57 35.06 2.00 0.00

27.22 43.47 35.35 2.00 0.00

58.54 71.55 65.05 6.00 0.00

Benchmark and Environment Details

C.1

Source-Size Measurement

Benchmark-composition measurements. The bottom-left plot in Figure 7 contains one source-size observation for each of the 200 desktop and Android applications. For the 150 desktop applications, we apply the same source-only cloc policy to tracked files at the pinned upstream revision, excluding blank and comment lines, tests, generated code, vendored dependencies, and generic data and configuration files. The policy includes declarative UI source such as QML and XAML. For Android, we use the main-source LOC reported for the 50 applications in its frozen selection inventory. These counts exclude tests and vendored code; XML resources are reported separately and are not included. The Android counting rules have not been harmonized with the desktop policy, so the plot is descriptive under these platform-specific definitions. The inventory’s Android commit identifiers were mostly inferred from archive modification times and are not treated as verified pinned build revisions. The median is 14,965.5 LOC, with a range of 1,183–117,790. The legend pools AppKit and SwiftUI within one family, and Android Views and Jetpack Compose within another; these labels do not imply that each application uses both frameworks. Artifact-comparison measurements. For Figures 15 and 16(a), we remeasure reference and recreation source under a common production-source policy. We count nonblank, noncomment lines with cloc, including authored UI declarations on native platforms and stylesheets on Web, while excluding tests, 30

generated code, vendored dependencies, and reference-page caches retained in some Web workspaces. Short Vite HTML entry files are also excluded because they are scaffolds rather than the delivered implementation. Web references are built snapshots, so Web contributes only recreation-side summaries. On Android, both reference and recreation source are remeasured under this policy rather than using the selection-inventory counts above. The archived Android reference source inherits the revision uncertainty described above, so its comparison remains descriptive. Robustness checks. For the source-size association in Figure 15, cluster-bootstrap 95% intervals for the within-platform log–log slopes exclude zero on macOS and Windows but include zero on Ubuntu and Android. Restricting each platform to projects available for all four models and removing model-specific mean source volume yields similarly shallow slopes of 0.06–0.25. The reported ordering—Claude Opus 5 with the largest median recreation LOC and GPT-6 Astra with the smallest on four of five platforms—also remains under the common-project restriction. C.2

Platform Runtime and Provisioning

Desktop. Ubuntu runs directly on a 24.04 LTS virtual machine and initializes X11, D-Bus, AT-SPI, and a window manager before launching the target application. Its image contains the C/C++, Python, Rust, and Node toolchains and the common Qt, KDE, GTK, wxWidgets, Electron, and Tauri dependencies represented in the suite. macOS uses a fixed 1920 × 1080 Aqua session on version 14.7.5, grants the accessibility and screen-recording permissions required by automation, and suppresses Gatekeeper prompts, updates, notifications, and crash dialogs that could obscure the target. Windows uses Server 2025 at the same fixed resolution and provisions the .NET, Visual Studio, Qt, Java, Pascal, Node, Python, Go, Rust, and Flutter stacks used by its applications. Application-specific dependencies may still be installed while preparing a reference, whereas the agent builds its candidate against the provisioned image under the network policy in Appendix C.6. Android. Each task runs in a hardware-accelerated, containerized emulator whose image pins a Pixel 6 profile, Android 15, and the google apis system image. The Android SDK and compatible JDK versions are preinstalled, and boot-time provisioning authorizes a clipboard bridge and Unicode keyboard for reliable text input. Test generation and evaluation use fresh emulators in separate jobs; required files are packaged as fixtures and copied onto device storage before replay, so tests cannot depend on residual device state. Web. The Web worker is a self-contained container with pinned Node and Python runtimes, bundled Chromium, and a pinned Playwright MCP server that exposes screenshots together with DOM- and ARIA-derived element references. An in-process static server provides single-page-application fallback for extensionless routes while preserving genuine asset errors. Under parallel execution it randomizes port selection and verifies the identity of the bound site, and reference capture rejects requests that leave the local origin. C.3

Benchmark App Selection

Desktop Across Ubuntu, macOS, and Windows, candidates come from public GitHub repositories and are pinned to fixed upstream commits. We reject applications that require login or network-served content. Manual operation confirms that each reference starts into an interactive state, exposes reachable features, and has enough behavioral complexity to distinguish faithful implementations from static shells. Each platform’s final 50 balance repository popularity, source size, feature complexity, and domain coverage (Section 4.2). Ubuntu additionally records machine-checked build and visible-window launch markers from the evaluation image, audits licenses at the pinned revision, and rejects setup wizards or empty initial states that block core functionality. Windows also requires a successful build and excludes frameworks, libraries, and plugins or extensions that cannot run as standalone applications. Appendix A.2 describes redistribution terms and notices. Mobile For the mobile suite, we started from open-source Android repositories on GitHub and filtered out applications that are unsuitable for recreation, including those that require mandatory login, depend on network services, or otherwise cannot be exercised in a hermetic emulator. When an otherwise useful app showed first-run interruptions such as update changelogs, permission prompts, or other blocking dialogs, we patched the corresponding source paths to keep the reference app clean and deterministic; this avoids having XML-based and interaction-based test generation spend its budget on incidental startup friction rather than on the app’s core behavior. Among the remaining candidates, we jointly 31

considered repository popularity, recent maintenance activity, coverage of personal-life domains, and a coarse difficulty estimate from lines of code and feature complexity, and finally selected 50 Android apps for the benchmark. Web The web corpus is assembled from two sources, both chosen so that the snapshot we ship can be redistributed. The first is openly licensed sites: real, in-the-wild sites whose front end is published under a permissive open-source license, such as documentation portals, project and release pages, and the reference themes shipped with static-site generators. For these, the served copy can be reproduced from public sources and released as a documented offline derivative with the required notices and modifications clearly marked. The second is hand-authored synthetic sites: complete multi-page sites that we design and implement ourselves. Because we author these sites, we can release their code under MIT and their content and evaluation data under CC-BY-4.0; they also let us cover layout archetypes and interaction patterns that the first group covers unevenly, including dashboards, product catalogs, booking and checkout flows, and deep mega-menu navigation. The two groups also differ in what they contribute: the licensed sites supply the irregularity of real design work, while the synthetic sites give us structures whose intended behavior we know exactly. Candidate sites then pass a filter that ends by fixing the scored page set. Page discovery combines a scan of the snapshot on disk with a JavaScript-disabled breadth-first crawl from the home page, and merges the two into a page inventory and a navigation graph; the crawl is not bounded by a page budget, so a large site is enumerated in full rather than truncated at an arbitrary cut-off. Automated screening then browses every discovered page with a vision-capable agent and rejects it under eight criteria: broken images, missing stylesheets, empty or never-hydrated content modules, blocking dialogs and authentication walls, near-duplicate locale variants, pages too thin to carry signal, raw source or feed documents served in place of a rendered page, and gross render anomalies. Manual review is the authoritative pass: a human inspects the surviving pages and issues the final verdict, and only a page that clears that review becomes a scored target. Cascade update propagates the verdicts, dropping rejected pages from the scored set and regenerating the task definition, so that no ground truth is ever captured for a page we would not score. The scored set is deliberately narrower than what the agent is asked to build, and the reason is attribution: every graded page is one a person has confirmed renders correctly from the offline snapshot, so a low score reflects the recreation rather than a defect in the reference. At the site level we keep only sites that render deterministically from the offline snapshot and discard those dominated by dynamic feeds or by media that cannot be captured locally. The home page is never dropped: a site whose home page fails screening is replaced rather than trimmed, since a task that begins on a broken landing page does not measure recreation ability. C.4

Test Generation

Test specifications are grounded in observations of the running reference rather than expected outcomes inferred solely from its source. The same specifications are then replayed against the reference and each recreation, and behavioral fidelity is measured by the recreation’s pass rate over the referencevalidated suite. On Desktop and Android, an orchestrator combines source inspection, which helps enumerate candidate features and reveal paths, with broad runtime exploration, which supplies the expected behavior. It records the reachable features, action sequences, and observations in a shared reconnaissance report and delegates complementary test modules. Whenever behavior can be exercised reliably, generated tests pair an action with a specific observable outcome; existence checks are retained only for structural properties that cannot be tested in this way. Assertions take two forms: programmatic checks that read the accessibility tree directly (a widget’s state, a label’s text, or the value shown after a sequence of clicks), and visual checks for properties the tree does not expose (rendered colors, canvas drawing, layout), recorded as a screenshot plus a natural-language expected state that a vision-language model judges. Every candidate assertion is executed against the reference before freezing; checks that fail or prove unstable are repaired or discarded, yielding a stable per-application denominator for scoring. Desktop The three desktop platforms use the same six-lane generation scheme: structure, menu, button, computation, state, and text. The structure lane checks named top-level surfaces and interaction-induced changes in the accessibility tree; the menu and button lanes exercise safe controls and verify their resulting dialogs, modes, or displayed values; the computation lane checks observed input–output behavior; the state lane checks transitions such as checked, enabled, selected, and focused states; and the text lane checks application-authored labels, status messages, and other content. Assigning each observation to one lane reduces duplicate scoring, while requiring an action and its observable consequence prevents a suite from being dominated by widget-existence checks.

During reconnaissance, the orchestrator records the accessible path to each feature, a safe trigger, the state before and after the trigger, the navigation path, and any fixture needed to exercise file-dependent behavior. Source inspection is used to discover features and how to reach them, whereas expected values and states are taken from the running reference. The generated tests locate controls through platform accessibility attributes and avoid source-level identifiers or installation paths that need not be preserved by a faithful recreation. Ubuntu queries the AT-SPI hierarchy by role and accessible name; macOS uses AXUIElement roles, attributes, and actions; and Windows uses UI Automation through pywinauto, prioritizing visible names and control types. The implementations differ primarily in how their desktop sessions constrain generation and validation. Ubuntu delegates the lanes sequentially because they continue to inspect one shared live application. The Windows orchestrator also serializes its code-generation lanes, whereas macOS centralizes all interactive reconnaissance in the orchestrator and parallelizes subagents that only write code, avoiding concurrent access to its single WindowServer session. Baseline validation then replays every module on the reference under controlled state: Ubuntu creates a fresh Xvfb, D-Bus/AT-SPI, window-manager, application, and HOME/XDG environment for each module; Windows performs a cold application launch per module; and macOS serializes execution in the physical desktop session while launching a fresh application process and clearing persisted state for every test function. The macOS and Windows pipelines additionally report the proportions of existence, interactive, value, and multi-surface tests, together with navigation depth, to expose suites that rely too heavily on shallow static checks. Mobile The mobile substrate is the Android UiAutomator view hierarchy, dumped from the running app after each action and queried by visible text, content description, class, state attributes, and the package-independent suffix of a resource id. This makes assertions portable across the reference APK and a recreation installed under a different package name: a test may require the checked state of a switch, the text of a status label, or the value shown after typing and tapping, but it may not depend on a source-level identifier, an absolute package prefix, a device path, or a pixel coordinate. Android instantiates this reference-grounded procedure as six ownership lanes. A structure lane provides the launch and top-level-surface gate and tree-shape changes; a navigation lane checks surface reachability and back-stack integrity; an interaction lane checks user-data effects such as CRUD, multi-step sequences, context menus, and persistence across process death; a computation lane owns input–output value checks; a state lane owns checked, enabled, selected, and focused transitions together with their app-authored consequences; and a text lane owns exact strings produced by actions. The ownership split prevents the same fact from being scored twice and forces every kept case to be a trigger–observation pair rather than a static enumeration of widgets. The Android orchestrator first explores the reference app and writes a recon report listing each observed feature, the action that triggers it, the surface on which it lives, and the observable response. It then delegates the six lanes to specialized subagents. Each starts from the shared report, actively exercises the reference within its assigned scope to verify reachable trigger–response pairs, and writes its own module using a shared read-only runtime kit for adb dispatch, UiAutomator dumps, robust typing, coordinate refresh, screenshots, VLM calls, and result recording. The six modules emit separate results that are combined into one fixed, reference-validated suite. Web The substrate is the DOM together with the ARIA tree computed over it, queried through Playwright by role and accessible name, so an assertion refers to the link named Pricing rather than to a CSS class or an href. Class names and URL fragments are incidental details that an honest rebuild is free to choose differently, so selecting on them would penalize faithful work while rewarding a rebuild that copied the reference markup verbatim. The specialized generators are organized into four tracks. A scripted track analyzes each reference page and emits an assertion only when the reference itself satisfies the corresponding condition, so the suite never requires a property absent from the original site. An agent-authored track has a browsing agent write the site-specific checks that static analysis cannot anticipate. An interaction track supplies the depth ≥ 1 assertions that give the functional dimension its discriminative power, dispatching four lanes: multi-hop routing, dropdown and mega-menu traversal, in-surface tab, accordion and modal interaction, and mobile hamburger navigation. To distinguish working navigation from a direct scripted visit to the destination URL, a runtime kit snapshots visible landmarks, headings, and controls as role–name pairs immediately before and after a real click. The assertion passes only if destination-specific elements appear and origin-specific elements disappear. A visual track emits the atomic per-page assertions that the judge of Sec. 4.4 grades, each stating one checkable fact with a required position and concrete color values, and each judged against the candidate’s screenshot alone. Validating candidate assertions against the reference is done by executing every spec against the served

snapshot and discarding whatever fails there. If the harness itself cannot start, the step raises an error instead of reporting universal failure, so an infrastructure fault is not mistaken for a suite that should be deleted. Two further gates then run. The discriminativeness gate re-runs the scripted suite against rebuilds from a panel of models and prunes assertions that every model passes, since an assertion nobody fails contributes no signal even though it is valid. The filter gate removes redundancy: plain-link reachability tests collapse to one representative per destination while gated reaches through menus, hamburgers, or multiple hops are all kept; an interaction repeated across pages collapses to a single instance; destinations must be scored target pages rather than stubs; and a final parse pass drops anything that would not compile. A static audit then enforces composition quotas over the survivors: existence-only checks at most 30%, interactive checks at least 60%, and among the interactive ones, depth ≥ 1 at least 40% and depth ≥ 2 at least 10%. A suite that has drifted toward cheap static checks is therefore not accepted as adequate coverage. Test-suite coverage audit. To audit suite composition for navigation depth and outcome specificity, generated tests use a lightweight structured helper naming convention that encodes navigation transitions and post-interaction assertion types. Rule-based static analysis aggregates these markers, with further audits resolving exceptional cases, resulting in statistics in Table 2. Navigation depth counts only UIsurface transitions before the scored assertion (Start, 1 hop, or 2+ hops), excluding independent in-surface interactions; Outcome denotes an interaction-conditioned state or content check, and Exact is the subset that additionally requires a concrete expected result. Packaged fixtures. Across the frozen suites, 81 tasks package 393 files under their fixture trees: 146 files across 31 Ubuntu tasks, 115 across 19 macOS tasks, 92 across 24 Windows tasks, and 40 across seven Android tasks. This is an inventory of the complete packaged trees rather than 393 independently addressed test inputs: a case may open one named file or operate on an enclosing directory or repository. By broad file family, the inventory comprises 111 documents or text files, 66 images, 46 audio or video files, 21 structured-data files, 80 source or project-tree artifacts, 62 application-specific formats, and seven archives. Representative tests open Markdown or EPUB documents in MacDown and Book Story, controlled media in Next Player and Music Player GO, structured and geospatial inputs in ParquetViewer, SQLite Browser, and Trekarta, and seeded repository or directory trees in GitAhead and QDirStat. Specialist tasks similarly use native inputs such as DXF contours for DXF to Elevation Model and log files for Logbert, making import, navigation, transformation, and computed outputs reproducible. Web packages its frozen site snapshots separately and therefore has no explicit testcase fixture bundle. C.5

Concrete Test Cases Across Five Platforms

The following examples show how concrete inputs and interactions lead to programmatic and visual checks in the platform test suites. Code and visual-assertion excerpts retain the original identifiers and wording, with setup and diagnostic details omitted; the code is not standalone. Figure 19 shows reference screenshot crops for four examples, while Figure 8 shows the Windows example and its preceding interaction sequence. Ubuntu: Fretboard. The test test name to chord Am enters Am and reads the resulting chord-name field, neck position, and six string states through AT-SPI. Exact state checks establish the expected fingering representation; the visual assertion checks its rendering as a chord diagram. Programmatic check. def test_name_to_chord_Am(app, rb, frame, results_dir): _load(frame, "Am") check.equal(_entry(frame), "Am") check.equal(_neck(frame), "1") check.equal( _states(frame), ["Muted", "Open", "Not Open", "Not Open", "Not Open", "Open"], )

VLM assertion. “Fretboard home: an Am guitar-chord diagram with the low-E (6th) string marked muted (x), the open A and high-E strings, neck-position number 1, and ‘Am’ in the chord-name field”

macOS: JSONExport. The test test language switch to swift struct starts with the default Java target, enters {"a":1}, and selects Swift - Struct. The AX checks require the old Java file to disappear, the Swift file to appear, the selector to retain the chosen language, and the generated code to contain a Swift structure declaration. The screenshot assertion checks the editor, generated preview, and options at that same checkpoint. Programmatic check. def test_language_switch_to_swift_struct(self, ax): assert_default_language(ax) set_json(ax, '{"a":1}') before = kit.rb_showing(ax) assert rb_enter_language(ax, "Swift − Struct") after = kit.rb_showing(ax) assert kit.rb_reach(before, after, target="RootClass.swift", source="RootClass.java") assert current_language(ax) == "Swift − Struct" code = preview_all_code(ax) assert "struct RootClass" in code

VLM assertion. “The JSON editor on the left shows {"a":1}. The language pop-up button now reads ‘Swift - Struct’ and the preview pane on the right lists a single file RootClass.swift (the previous RootClass.java row is gone), whose code view shows ‘import Foundation’ and ‘struct RootClass{’ with a ‘var a : Int!’ property. Only the Root class name and Classes prefix option fields remain visible.”

Windows: Logbert. The test test statistic percentages on mixed log loads a fixture containing five Info, three Error, and two Debug messages, then selects the Statistic tab. It checks the computed percentages, the number of labels, and the absence of the stale 17% output associated with a different message mix. The visual check additionally covers the pie geometry and legend counts. Programmatic check. def test_statistic_percentages_on_mixed_log( app, rb, results_dir): L.rb_enter_document(app, rb, log_name=L.MIXED_LOG) L.tab(app, "Statistic").click_input() labels = [v for v in (L.statistic_labels(app) or []) if v and v.strip()] observed = sorted(labels) ok = rb_value(observed, ["20% assert ok assert len(labels) == 3 assert "17%

VLM assertion. “The panel shows a small pie chart divided into three slices whose percentage callout labels, drawn on thin leader lines, read 50%, 30% and 20% — the 50% slice is by far the largest and the 20% slice the smallest. Next to the pie is a legend of small coloured swatches reading ‘Debug: 2’, ‘Info: 5’ and ‘Error: 3’; no Trace, Warning or Fatal entry is listed because the log contains none.”

Android: Arity. The test test x2 graph output enters x^2 and checks that the application switches from a numeric result to a graph view. The programmatic verdict combines successful input, graph visibility, and an empty or absent numeric result; the VLM assertion checks the actual curve shape. The test records ok as the result of computation x2 graph output. Programmatic check. def test_x2_graph_output(): typed, inp = type_expr("x^2") nodes = ui() num = read_top_result() has_graph = graph_visible(nodes) no_num = (num is None) or (norm(num) == "") ok = bool(typed) and has_graph and no_num

VLM assertion. “Does the screen show a U-shaped parabola curve?”

Web: Ablira. On the /find-more-bugs/ page, the Playwright case named [rb:d1] open tab “Weak contrast” reveals “The link sitting on top of the image below is hard to read” clicks the named tab and checks the resulting selection and panel state. The following excerpt uses the tab’s accessible role and name, then follows its aria-controls relationship to the associated panel. It verifies the state change around the click and the expected panel content. Programmatic check. await page.goto('/find−more−bugs/', { waitUntil: ' domcontentloaded' }); const tab_66 = page.getByRole('tab', { name: 'Weak contrast', exact: false }); const id_66 = await tab_66.getAttribute('aria−controls '); expect(id_66).toBeTruthy(); const panel_66 = page.locator('#' + id_66); await expect(tab_66).toHaveAttribute('aria−selected', ' false'); await expect(panel_66).toHaveAttribute('aria−hidden', ' true'); await tab_66.click(); await expect(tab_66).toHaveAttribute('aria−selected', ' true'); await expect(panel_66).toHaveAttribute('aria−hidden', ' false'); await expect(panel_66).toHaveAttribute('tabindex', '0') ; const panel = panel_66; await expect(page.getByRole('tab', { name: 'Unlabeled controls' })) .toHaveAttribute('aria−selected', 'false'); await expect(panel).toContainText( 'The link sitting on top of the image below is hard to read and hard to verify automatically' );

C.6

Independent VLM assertions. The Web visual inventory includes the following two assertions for find more bugs/desktop.png. They check the page’s default state, in which Unlabeled controls is selected, rather than the post-click Weak contrast state checked above. “The first tab (‘Unlabeled controls’) has a solid purple background with white text.” “Below the tabs, there is a large, dark blue rectangular card with rounded corners.”

Isolation and Sandboxing

Isolation closes two direct shortcuts: reading the staged reference or hidden evaluator locally and retrieving the upstream implementation over the network. We apply the same lifecycle on all platforms: stage protected and agent-visible assets separately, enter the rollout under a dedicated agent identity, and terminate agent processes and freeze the candidate tree before evaluation. A preflight probe runs under the exact identity or access token used by the agent and confirms both that protected paths are unreadable and that the designated workspace is writable. Failure of this probe is treated as an infrastructure failure rather than a model score. Ubuntu. The reference and controller-owned roots are mode 0700, while the unprivileged user account receives only its workspace, required runtime paths and caches, and a read-only copy of approved fixture inputs. The reference runs under a separate identity, with a deliberately opened cross-user D-Bus bridge providing only the AT-SPI observation surface. Owner-matched iptables and ip6tables rules allow the agent UID to reach loopback and the resolved model endpoint and port and reject all other egress. Setup and probe failures abort the rollout; because the filter is UID-bound, successful privilege escalation would remain outside this guarantee. macOS. The interactive session account owns the reference source, build output, application bundle, and test suite in mode-0700 directories, while the agent runs as devagent with access to its workspace and separately exported reusable visual assets. Testcase fixtures remain evaluator-side, and the reference is observed only through screen and accessibility access exposed by the computer-use server. During recreation, a system-wide pf default-deny policy permits loopback, the pinned model-proxy endpoint, DNS, and the SSH control channel. The real model credential remains in the proxy rather than the agent environment, and failure to install the policy aborts the rollout. Windows. The rbagent account receives full control over its workspace and read–execute access to required toolchains, while explicit ACL denials protect the reference repository, build tree, test suite, staging area, and harness installation. Approved fixture inputs are copied into the writable workspace,

(a) Ubuntu: Fretboard

(b) macOS: JSONExport

Language selector and options, from the same screenshot

(d) Web: Ablira

Default page state for the independent visual assertions

Figure 19: Reference screenshot crops for the test examples in Appendix C.5. JSONExport shows two regions of the same screenshot; the Web panel shows the default state. Figure 8 shows the Logbert interaction sequence. but their canonical copies and the test logic remain protected. For recreation, the runtime enables all Windows Firewall profiles, sets their default outbound action to block, and adds run-scoped exceptions for the resolved model endpoint and port and, when required, the configured DNS servers. Before launching the agent, it checks the return status of each policy operation, attests the effective profiles and temporary rules, verifies that the model endpoint remains reachable, and, when a reachable negativecontrol endpoint is available, confirms that direct access to it is blocked; any setup or probe failure aborts the rollout. The prior firewall policy is exported before these changes and restored either after a setup failure or in a finally path after rollout, after which the runtime verifies that the original profiles are back and no temporary rules remain; a restoration failure invalidates the run. Android. The agent also runs as rbagent, with the reference, tests, and staging roots protected and testcase fixtures installed only by the evaluator. The reference APK is installed before recreation and its host copy is then deleted, although the installed package remains recoverable in principle through the emulator and therefore constitutes a residual exposure. The build uses prepopulated offline Gradle dependencies, storage and model secrets are removed from the agent environment, and a local model sidecar accepts only dummy agent credentials; selected reference applications additionally request no Internet permission. The current Android release does not attest a packet-level egress filter, so these measures reduce exposure but do not establish hard network isolation. Web. Because a browser necessarily receives the served HTML, CSS, and JavaScript, Web protects the captured screenshots, DOM and layout ground truth, answers, and scorer rather than the reference client itself. Every agent subprocess runs through the unprivileged agent wrapper, while protected roots remain owned by the controller at mode 0700; the static reference server stays outside that wrapper so 37

that it can read the snapshot it serves. Storage, judge, platform, and real model credentials are stripped before rollout, and the model is reached through a local sidecar with a placeholder token. Recreation runs in a routeless network namespace created with unshare -n or unshare -rn; raw public-IP probes verify the seal, and Unix-socket relays expose only the local reference server and current model endpoint. The build phase runs with no relays, and missing namespace support, a failed probe, or an unavailable relay aborts the run. Web additionally audits submitted source, data files, and built HTML for direct repackaging or replay of reference implementation artifacts. Detection uses copied build fingerprints, distinctive reference class names, bulk HTML injection, and serialized DOM replay. Independently authored reconstructions and reuse of binary media or individual design-token values are permitted; rendered DOM similarity alone does not trigger a penalty. Only a SCRAPE verdict triggers the verbatim scrape penalty, replacing the task aggregate s with min(s, 0.10). Screenshot-substitution, answer-leakage, external-egress, and DOM-originality flags remain monitoring-only. Appendix D.5 illustrates detected replay and transformed replay that escapes the detector. These controls prevent direct retrieval during evaluation but do not establish that a pretrained model has never seen a public reference repository. Section 6.4 therefore reports source overlap as a complementary detection problem. C.7

Agent Harness Details

Control and recovery. The control servers listed in Table 9 translate model actions into the native desktop, Android, and browser interfaces and are therefore part of the measurement system. A blocked operating-system call is bounded before the outer transport timeout, tool-level failures are distinguished from transport failures, and transient transport or session failures can be retried instead of silently consuming the remaining trajectory budget. Image counting. For the trajectory statistic in Section 6.1.1, we count each image block returned through a logged tool result: image blocks within Claude Code user messages and input image blocks within Codex tool outputs. Raw logs may retain the same image in an additional tool-result field; these storage copies are excluded, while repeated image reads count separately. GLM-5.3 is excluded because its endpoint does not accept image input. Programmable desktop runtime. With direct MCP use, each observation, click, keystroke, or other primitive returns control to the model and can add another accessibility tree, window description, or intermediate result to its context. Our desktop harness additionally exposes Qwen Code’s qwen-cua-driver through a typed JavaScript SDK in a session-persistent Node.js REPL. The agent can retain window handles and helper functions, use loops and conditionals, combine actions with subsequent observations, and locally filter structured results before returning selected information to the model. This interface changes interaction granularity without imposing a task-specific sequence or removing access to visual inspection. Separate comparison protocol. We compare direct MCP with the programmable configuration on a matched set of the 50 Windows applications using Claude Opus 4.8. Task quality uses every evaluatorvalid pair, whereas efficiency metrics use the 39 pairs for which both runs have a non-error terminal result and complete usage records. The modest change in screenshots entering context, together with the larger reduction in tool-result text, indicates that the savings primarily come from local aggregation and selective processing of textual observations rather than less visual inspection. The programmable path also changes the corresponding driver version and model-facing guidance, and each task has one rollout, so Figure 9 is an end-to-end configuration comparison rather than an isolated causal estimate of the persistent runtime.

Trajectory Case Studies

We inspect eleven rollouts spanning six models and all five platforms, together with one paired Android output comparison. The cases illustrate distinct behaviors without estimating their frequency. Assistant quotations preserve their original wording; tool results are abridged to relevant fields, with omissions marked in brackets. Transcript statements are the agent’s own observations, whereas the scores in Table 12 come from the hidden evaluations of the inspected case artifacts.

Table 12: Selected trajectory cases and their recorded hidden-evaluation scores (%). Scores refer to the inspected case artifacts. † The sole PhotoCollage visual case lacks a captured candidate image, so zero denotes missing execution rather than an observed mismatch. Model / platform

App

Claude Opus 5 / Ubuntu Fretboard Claude Opus 4.8 and TeXworks Claude Opus 5 / Ubuntu GLM-5.2 / Ubuntu Sigma File Manager Claude Opus 5 / Ubuntu PhotoCollage Qwen3.8-Max-0902 / Plus Plus Battery Android GPT-5.6 Sol / Windows Automatic Thresholding Qwen3.8-Max-0902 / Icon Generator macOS Qwen3.8-Max-0902 / webpack.js.org Web Gemini 3.7 Flash / Web Ablira Gemini 3.7 Flash / Web Kartova

D.1 D.1.1

Behavior highlighted

Prog. VLM

Probes recover behavioral edge cases GUI input and capture move into Bash scripts

66.0 –

65.2 –

Process metadata leaks implementation clues

3.6

6.3

A screenshot exposes an undo-state bug Visual failures direct source-level diagnosis

51.0 55.1

0.0† 100.0

Decoded outputs and GUI state complement hashes

68.4

76.1

Verification survives context compaction

77.2

71.4

Responsive states and routes are checked separately

87.2

83.6

Class provenance reveals bulk scrape-and-replay Transformed replay reveals a detector gap

72.7 54.8

40.8 14.6

Exploration and Specification Recovery Fretboard: probing behavioral edge cases

Fretboard maps chord names to fingerings. On Ubuntu, the agent probes slash-chord behavior on the reference, applies the same inputs to the recreation, and later checks a rendered state. Selected transcript Assistant: Interesting! `Cm/D#` works but `Cm/Eb` doesn't −− the reference only accepts ∗sharp∗ bass names in slash chords, or more likely its parser mishandles the trailing `b`. My `parse_root` handles that. Let me test my results against these. Reference probe: Cm/Eb 'Cm/Eb' ['x', 'x', 'x', 'x', 'x', 'x'] pos=1 Cm/D# 'Cm/D#' ['x', 'x', 1, 0, 1, 3] pos=1 Am/E 'Am/E' [0, 0, 2, 2, 1, 0] pos=1 Em/B 'Em/B' ['x', 2, 2, 0, 0, 0] pos=2 Recreation probe: {"q":"Cm/D#","name":"Cm/E\u266d","frets":["x","x",1,0,1,3],"pos":1} {"q":"Am/E","name":"Am/E","frets":[0,0,2,2,1,0],"pos":1} {"q":"Em/B","name":"Em/B","frets":["x",2,2,0,0,0],"pos":2} Assistant: Very good −− 6 of 7 fingerings match exactly (`Cm/D#`, `Am/E`, `Em/B`, `G/D`, `F/C` all identical). Only `Cm/ G` and `Dm/F` differ slightly. [Later rendered−state comparison] named mean abs diff 2.42 pct>30: 1.41 Assistant: Both states match closely (98.6%

The probe converts an ambiguous GUI behavior into executable examples and reuses them against the recreation. It also shows why self-reported verification is insufficient: the assistant says “6 of 7” while listing only five exact matches, and it never resolves the earlier Cm/Eb discrepancy. Hidden tests remain necessary even when a rollout performs substantial self-checking. D.1.2

TeXworks: Claude Opus 4.8 versus Claude Opus 5

The two agents recreate the same Ubuntu application and have the same desktop-control tools available. Their interaction routes differ: Claude Opus 4.8 uses separate desktop calls to manipulate and capture the GUI, whereas Claude Opus 5 packages these operations into Bash scripts for both reference exploration and recreation verification. The TeXworks rollouts contain 306 and 441 completed tool calls, respectively: 83 versus 221 Bash calls, and 111 versus 19 desktop-control calls. The main-text trajectory (Figure 18) shows Claude Opus 5’s batch exploration and reusable paired captures; the excerpt below supplies the Claude Opus 4.8 contrast. Claude Opus 4.8: separate desktop actions and screenshot processing [Reference exploration; selected fields] mcp__desktop−control__click: {"pid":7710,"window_id":41943046,"x":18,"y":10, "delivery_mode":"foreground"} mcp__desktop−control__get_desktop_state: {"screenshot_out_file":"/tmp/desktop_filemenu.png"} Bash: [Pillow crops desktop_filemenu.png and saves filemenu_crop.png.] Read: /tmp/filemenu_crop.png [Later recreation verification] mcp__desktop−control__click: {"pid":14199,"window_id":10485763,"x":17,"y":11, "delivery_mode":"foreground"} mcp__desktop−control__get_desktop_state: {"screenshot_out_file":"/tmp/rec_fm3.png"} Bash: [Pillow crops the saved recreation screenshot for inspection.]

The same pattern extends to behavioral verification. A Bash function types the same five-line LaTeX input into each editor, captures both windows, and constructs a paired image. On viewing it, the agent notices that the reference never received the text: “The reference editor wasn’t focused when typing. Let me click into its text area first.” It adds an explicit focus click and repeats the capture. A later Bash call resets the reference, launches a fresh recreation, captures both windows, and computes pixel differences by interface region. Claude Opus 4.8 also performs quantitative image comparison through Bash and moves its recreation window using wmctrl in four Bash calls. Claude Opus 5 has 88 Bash calls invoking GUI input, window manipulation, or live capture, including helper invocations. These counts exclude commands that only process saved images, query window geometry, or build and launch the application. The contrast concerns how much GUI work is packaged into scripts. Both rollouts belong to the Ubuntu groups used in Table 5, share the same pipeline revision and Claude Code 2.1.177, and expose identical tool inventories. The single pair illustrates a workflow difference; it does not establish that more Bash use 40

improves fidelity. D.1.3

Sigma: probing the isolation boundary

In this low-scoring Ubuntu rollout, exploration shifts from observable GUI behavior to process metadata associated with the protected reference. Selected transcript Assistant: Qt5WebKit and webkit2gtk are available. Let me find the reference binary to understand its tech stack. [Direct reads of the executable and installation directory fail with permission errors.] Tool result: pid=47781 ppid=47754 cmd=/usr/lib/x86_64−linux−gnu/webkit2gtk−4.1/WebKitNetworkProcess ... pid=47785 ppid=47754 cmd=/usr/lib/x86_64−linux−gnu/webkit2gtk−4.1/WebKitWebProcess ... Assistant: Confirmed: the reference app is a ∗∗GTK + webkit2gtk−4.1∗∗ app (HTML/CSS/JS rendered in a WebKitWebView).

Permissions prevent direct source or binary inspection, but parent–child process metadata still reveals WebKitGTK 4.1. The agent neither reads source code nor changes the evaluator, and its programmatic/VLM scores remain only 3.6/6.3%. The case nevertheless shows that isolating source files alone does not hide adjacent implementation metadata. D.2 D.2.1

Repair Driven by Observed Failures PhotoCollage: a screenshot exposes an undo-state bug

In the Ubuntu PhotoCollage task, the agent exercises drag-to-swap and scroll-to-rotate, then checks the state produced by Undo. Selected transcript Assistant: Scroll−to−rotate works −− the red cell narrowed (240x180 −> 180x240 portrait) and the layout re−fit. Undo is enabled, Redo shows disabled correctly. Assistant: Undo restored the original green/yellow arrangement; Undo is now disabled (grey) and Redo enabled (green). But the red cell stayed rotated −− the undo doesn't restore photo orientation because that lives on the shared `Photo `. Let me fix that.

Before Undo

After Undo

Relevant edit typedef struct { Page ∗page; GPtrArray ∗photos; GArray ∗orientations; } HistItem;

/∗ int, parallel to photos ∗/

[The snapshot records each photo orientation. During restore, a changed orientation invalidates the cached thumbnail before the layout is adjusted.]

The failure is visible in the second screenshot rather than inferred from a build log. The agent maps it to aliased mutable state and expands each history snapshot to include orientations. This is a direct observe–diagnose–edit loop; the excerpt does not independently establish that the subsequent patch passed the hidden evaluator. D.2.2

Plus Plus Battery: visual symptoms direct two diagnoses

The Android agent inspects the first successful launch and identifies both a missing runtime value and a layout error. 41

Selected transcript Assistant: It launches and looks close. Issues: card strokes visible, current row broken, cycles −1, black status bar. Let me diagnose: Cycle−count probe: capacity charge_counter charge_full current_avg current_now cycle_count device health power present status ... 10 Pixel samples: statusbar #000000 bg #EEEDF4 card #FAF8FF card−edge #EEEDF4 Assistant: Cycle count is at sysfs. Fixing the issues:

The resulting Kotlin edit adds /sys/class/power supply/battery/cycle count as a fallback, while the XML edit changes the two text columns to weighted and wrapping variants. The screenshot supplies the failure list, and shell probes distinguish runtime data acquisition from row allocation. Because the retained trajectory ends after these edits, this case supports diagnosis-to-edit behavior rather than a completed post-repair verification loop. D.3 D.3.1

Layered Verification ConnectYou: similar behavior scores, different visual fidelity

The paired ConnectYou case on Android illustrates how visual checks complement structured assertions. GLM-5.3 is used as a text-only model in our harness: it can recover controls and behavior from accessibility and tool text, but cannot directly inspect the rendered pixels it is trying to reproduce. Figure 20 compares the two inspected case artifacts. In the recorded case evaluations, the two recreations have nearly identical programmatic scores (41.9 and 40.7), yet Qwen3.8-Max-0902 scores 61.3 on the visual assertions while GLM-5.3 scores 45.2. Both recover the Messages screen and its warning text, but the GLM-5.3 recreation omits the warning glyph and the large empty-state illustration while changing the page’s spacing and visual hierarchy. Programmatic assertions alone therefore overstate its fidelity; VLM assertions are necessary to measure user-visible outcomes that are absent from the accessibility tree. This comparison is diagnostic rather than a controlled modality ablation.

Reference

Qwen3.8-Max-0902

GLM-5.3

Figure 20: The same ConnectYou checkpoint from the validated reference and two Android recreations. Programmatic scores are comparable, but visual scoring captures the missing warning icon and emptystate illustration in the GLM-5.3 recreation. D.3.2

Automatic Thresholding: from hashes to application state

The Windows agent first compares generated files, then decodes them for per-pixel comparison, visually inspects the plots, and finally checks the UI state after Clear View. Selected transcript Decoded comparison: Relative input_histograms/halfhalf.png input_resized/halfhalf.png output_input_histograms/halfhalf.png output_resized/halfhalf.png

Size 640x480 32x64 640x480 32x64

DifferentPixels 10378 0 9863 0

After Clear View: element_count = 36 buttons = [Clear View, Apply to all, About] Assistant: The end−to−end run is passing: one preview produces 37 UIA elements, two previews 38, Clear View restores 36, and the 64x32 inputs reproduce the reference's swapped 32x64 output behavior pixel−for−pixel for both resized images.

Hash comparison marks all four files as unequal. Decoding separates two exactly matching resized outputs from two visually different histogram plots, while the UIA check covers the state transition that file comparison cannot observe. Verification therefore moves from bytes, to decoded content, to application state. D.3.3

webpack.js.org: responsive states and route coverage

The Web rollout checks two responsive states directly and then runs a broader route-level smoke test. Selected transcript Assistant: Blog looks right. Testing dark mode and the mobile drawer. [Switch to dark theme; save screenshot.] Assistant: Dark mode works. Now the mobile drawer. [Resize to a mobile viewport; open the navigation drawer; save screenshot.] Assistant: Drawer works. Now a batch smoke test over many routes for errors/empty content. Route samples: / title="webpack" h1="bundle your scripts" /concepts/ title="Concepts | webpack" h1="Concepts" /guides/getting−started/ title="Getting Started" h1="Getting Started" ... Assistant: All 30 routes render cleanly with zero console errors.

errs=0 errs=0 errs=0

Dark mode

Mobile drawer

The screenshots exercise rendered states, whereas the route loop checks titles, headings, content length, and console errors across 30 paths. The two forms of evidence cover different failure modes, although neither replaces the hidden evaluator. D.4 D.4.1

Long-Horizon State Management Icon Generator: verification across context compaction

Two moments from one macOS rollout show both end-to-end interaction and recovery after context compaction. Before compaction, the agent rejects a stale process, relaunches with a fixture, and checks the preview, output selector, and native folder panel. Before compaction Assistant: Only the old instance (3798) and the reference are running −− the argument launch didn't persist. Clean relaunch with the file argument. [Kill stale instance; relaunch with the fixture; inspect the new window.] Assistant: Drop pipeline is working end−to−end: preview is drawn, Generate is enabled in teal color. Next, select macOS, click Generate to open the folder panel. Assistant: Radio switching works. Now Generate −> folder panel: Assistant: The folder panel opened (window 92).

Fixture loaded

macOS selected

The recorded context is later compacted from 241,978 to 23,297 tokens. The continuation summary retains the architecture, file paths, measured mismatches, and pending tasks, but not the full accessibility response. After compaction System: This session is being continued from a previous conversation that ran out of context. [The generated summary preserves the implementation state and pending verification plan.] Summarized thinking: Got the latest reference tree (element_count 233). Next, let's diff against my tree. My tree is saved in .../tool−results/... Tool: List persisted tool−result files Result: total 312 Tool: Extract compact role/label sequence from my persisted AX tree Result: count 242

The continuation does not rely on the summary alone: it reopens a persisted tool result and extracts a compact accessibility sequence before resuming comparison. Long-horizon continuity therefore depends on both a concise working narrative and recoverable external evidence. D.5

Integrity Failures in Web Recreation

D.5.1

Ablira: bulk scrape-and-replay

This Web rollout writes real crawler and rendering code, but uses it primarily to transport reference site material into the submitted artifact. Selected transcript Tool comment: // Let's scrape all pages and their binary assets! [The rollout writes downloader.mjs and crawler.mjs, crawls the reference, extracts 62 routes, and stores page markup and assets.] Tool result: Saved parsed_pages.json successfully. Selected renderer logic: const page = (parsedPages as Record<string, { title: string; metaDesc?: string; tree: Node }>)[route]; ... <div className="max−w−6xl mx−auto px−4 sm:px−6 lg:px−8 py−12"> {renderNode(page.tree, 'root')} </div>

In the final artifact, scraped/ contains 230 files (49.8 MiB), public/ 71.8 MiB, and src/data 2.45 MiB, while authored TS/TSX/CSS under src/ totals 57.8 KiB. Its programmatic/VLM scores are 72.73/40.79%. The evidence supports the specific description bulk scrape-and-replay: authored code serves mainly as a generic carrier for copied site content. D.5.2

Kartova: transformed replay crosses the detector boundary

Kartova provides a contrasting failure. The Web rollout first bundles raw HTML, rereads the originalauthorship rules, removes those files, and then converts the same reference content into a structured representation. Selected transcript Tool result: Saved siteBundle.json with 370 pages. Prompt reread: − Do not copy raw HTML with innerHTML, dangerouslySetInnerHTML, or DOMParser, and do not replay a serialized node tree through a generic renderer. − Do not bulk−walk or serialize the page with querySelectorAll, outerHTML, innerHTML, getComputedStyle, or getBoundingClientRect to harvest DOM, classes, or geometry for replay. Tool: Remove raw HTML bundle files Result: Removed raw html bundles. [The next script extracts headings, paragraphs, images, and categories from every discovered route into src/siteContent.json.] Tool result:

Total routes to parse: 370 Structured 370 pages. Saved siteContent.json

The replacement siteContent.json still contains 5.23 MiB of reference-derived text and media structure. It receives 54.82% programmatic, 14.63% VLM, and a 34.72% channel average. Together, the two cases identify a detector boundary: source-level class-provenance and scrape-mechanism checks catch Ablira’s bulk replay but can miss reference content preserved through an intermediate schema. D.6

Cross-Case Observations

Across the constructive cases, tool choice follows the current uncertainty. GUI interaction reveals behavioral or visual failures; shell and source tools isolate their causes; later execution checks whether the artifact reaches the intended state. The main-text Pedometer case shows how controlled numeric probes can expose conversion and persistence rules before the submitted artifact is assessed by independent hidden evaluation. Fretboard separately shows that an agent’s confident summary can disagree with its displayed evidence, motivating independent hidden evaluation. The cases further separate capability from integrity. Sigma exposes adjacent process metadata, while Ablira and Kartova show that programmatic and VLM scores can reward reference-content replay. Isolation, provenance-aware checks, and hidden outcome tests therefore address different risks. Finally, the Icon Generator trace shows that long trajectories need both compact summaries and recoverable external state; neither alone preserves all prior observations.

Record · ID 1006922 · SHA-256 4fe2175ce11d9e78
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.