UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation Grace Man Chen1 , Litao Guo1 , Yifan Wu1 , Yiyu Chen1 , Yenchi Tseng1 , Sicheng Liu1 , Yuyu Luo1,2,† , Ying-Cong Chen1,2,† 1
The Hong Kong University of Science and Technology (Guangzhou) 2 The Hong Kong University of Science and Technology †
Corresponding author
Project
§ Code
Dataset
arXiv:2607.06306v1 [cs.SE] 7 Jul 2026
Abstract Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2×. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.
1
Introduction
Large language models (LLMs) have made notable progress in code generation, task planning, and reasoning [Roziere et al., 2023, Guo et al., 2024, 2025]. Building on these advances, recent work has demonstrated the feasibility of end-to-end software development from natural-language instructions [Jimenez et al., 2024, Hong et al., 2024, Qian et al., 2024]. However, existing text-driven paradigms largely rely on carefully designed or highly detailed prompts as input [Lu et al., 2025, Zhang et al., 2025a], which introduces practical limitations. First, textual descriptions are limited in precisely specifying detailed visual requirements, making it difficult to accurately define page layouts and maintain visual coherence across multiple pages. Second, functional requirements involving cross-page interactions, such as state management and data synchronisation, are difficult to specify precisely and consistently in natural language. To alleviate these limitations, image-to-webpage generation has emerged as an alternative paradigm. Instead of relying on textual specifications, this approach takes UI designs as input and reconstructs webpage structure and appearance directly from visual signals. This paradigm better aligns with Preprint.
Input: static UI screenshots alone (no behavior spec) → generate a runnable app
Build & Render (EXEC)
Navigation (NRS)
Visual fidelity (VFS)
Click 'Add to Cart' (size M)
Internal state (simplified)
Switch to size L, click again
Navigate to /cart
cart = [ { id: "tshirt-01", size: "M", color: "Black", qty: 1 }, { id: "tshirt-01", size: "L", color: "Black", qty: 1 }, ]
Build & Render (EXEC)
Navigation (NRS)
Click 'Add to Cart' (size M)
Internal state (simplified)
Interaction inference - IIS (ours)
Inferred ✓ — stateful interactions work
Visual fidelity (VFS)
Interaction inference - IIS (ours)
Cart: 0
Cart: 0
Switch to size L, click again
Navigate to /cart
Frozen façade — looks right, but state never updates
cart = [] // no items, no schema // CartContext.add() not implemented
Figure 1: UI2App task overview: visual fidelity and interaction inference are independent capabilities. Same e-commerce screenshots, same action sequence (Add to Cart twice, then navigate to /cart). Claude Sonnet 4.6 reconstructs a fully dynamic application (cart counter increments, cross-route state propagates); Gemini 3.1 Pro Preview produces a higher-fidelity but frozen façade in which every interaction is a no-op. Only IIS catches this dichotomy. the common designer workflow of turning mockups into functional prototypes. It also addresses the demand from non-technical users who want functional apps directly from screenshots. Several benchmarks have been developed along this line [Beltramelli, 2018, Si et al., 2025, Laurençon et al., 2024, Li et al., 2025]. However, these benchmarks largely focus on visual fidelity. They measure only how closely the rendered output matches a reference image, not whether the artifact works as an application. A generated interface may appear correct yet remain a behaviorally inert façade. As illustrated in Figure 1, different models might both produce runnable applications with high visual fidelity and basic navigation capabilities. However, while one model merely reconstructs static layouts, the other accurately infers and realizes the underlying dynamic interactions purely from static screenshots. This gap reflects a deeper distinction in capability. When interaction behavior is supplied explicitly, through textual instructions or demonstration videos [He et al., 2026, Chen et al., 2025a, Dai et al., 2026], the model performs specification-following: implementing a target that has already been named. When the input is screenshots alone, the model must perform interaction inference: recovering the missing behavior from visual evidence. No existing benchmark measures interaction inference under image-only input. We introduce UI2App, a benchmark that makes interaction inference measurable. Each task supplies a set of screenshots and asks the model to emit a runnable multi-page artifact. Screenshots arrive without captions, instructions, or interaction descriptions. The model must therefore reconcile structure, palette, and inferred behavior across multiple routes simultaneously rather than translate one canonical view. The released benchmark comprises 327 screenshots organized into 45 carefully curated state-coherent screenshot sets. Rather than isolated page images, each set spans multiple routes and captures consistent cross-page state, providing the visual evidence required to reconstruct a complete runnable web application. To evaluate the resulting artifacts, we design an end-toend evaluation chain that scores each along four dimensions: whether the application builds and renders (EXEC@1 / EXEC@3), whether expected routes are reachable (NRS), how faithfully the rendered UI matches the reference (VFS), and whether the interactions implied by the screenshots are realized (IIS).
2
Of the four, IIS is the core measure and the hardest to design: static screenshots underdetermine behavior and admit multiple valid realizations, since the same visual state may correspond to different correct implementations of interaction logic (e.g., a search box that can be triggered on Enter, button click, or real-time query). To address this challenge, we organize interactions commonly found in web applications into a taxonomy of seven categories and evaluate each with a rubric-based protocol along three complementary axes. Each inferred interaction is assessed against a category-specific rubric that checks whether the expected trigger, state transition, and resulting interface behavior are correctly realized. This taxonomy-driven design enables structured assessment of inferred behaviors without requiring a unique ground-truth execution trace, while providing fine-grained diagnostic signals beyond a single aggregate score. We evaluate six frontier VLMs on UI2App. Even the strongest reaches an overall IIS of only 39.3, indicating that inferring complete interaction behavior from static screenshots remains a frontier challenge. Moreover, visual fidelity does not imply interaction-inference capability: the leader on visual fidelity (VFS) places fourth on IIS, 5.2× behind the IIS leader. The gap is widest on cross-route state: on S3-scope interactions (cross-route persistence), three of six models score exactly zero, and even the best reaches only 21.6 out of 100. Our contributions are as follows: • Benchmark. We release UI2App: 327 screenshots in 45 state-coherent screenshot sets for runnable, multi-route web applications. UI2App is the first benchmark targeting interaction inference rather than specification-following from image-only input. • End-to-end evaluation protocol and IIS taxonomy. We introduce a four-metric protocol spanning build executability (EXEC@1 / EXEC@3), navigation reachability (NRS), visual fidelity (VFS), and interaction inference (IIS), with IIS built on the interaction taxonomy. • Extensive evaluation and analysis. We benchmark six frontier VLMs and a Qwen2.5-VL scaling ladder, establishing the first baselines for image-only interaction inference and showing that visual fidelity does not imply interaction-inference capability, with cross-route state persistence a frontier-wide bottleneck.
2
Related Work
Static Visual-to-Code. Building on advances in screenshot and UI understanding [Lee et al., 2023, Baechler et al., 2024, Liu et al., 2024, Wu et al., 2024], vision-language models have made rapid progress on translating UI visuals into frontend code. Early benchmarks established the screenshot-tocode task with real-world webpages and visual similarity scoring [Si et al., 2025, Xiao et al., 2025b], then scaled the paradigm with synthetic and large-scale corpora [Yun et al., 2024, Gui et al., 2025]. Subsequent work shifted the output target from HTML to framework-specific components [Ge et al., 2025, Wu et al., 2025, Chen and Chen, 2025, Chen et al., 2025b, Zhang et al., 2025b, Wu et al., 2026] and extended generation to multi-page sites by pairing screenshots with a structured list of navigation links and resources [Wan et al., 2024]. The underlying paradigm treats UI generation as a static rendering problem: tasks specify what an interface looks like, not how it behaves, and runtime interaction is left outside evaluation. In contrast, UI2App inherits the screenshot-based input but reframes the task around behavior, asking models to produce a runnable application whose interactions are tested at execution time rather than a visual replica. Interaction Specified by Text. Once interaction becomes part of the task, the most straightforward way to specify it is through natural language. WebGen-Bench [Lu et al., 2025] tasks an LLM agent with building a multi-file website codebase from a textual instruction and grades it through automated functional tests. ArtifactsBench [Zhang et al., 2025a] extends this paradigm to interactive artifacts spanning web interfaces, visualisations, and mini-games, scored by an LLM judge. An image-augmented variant further pairs multi-screen prototypes with textual descriptions of intended interactions [He et al., 2026]. These benchmarks explicitly specify the interaction and evaluate compliance with that specification. Instead, UI2App provides only static screenshots, leaving the intended behavior to be inferred from visual cues alone. Interaction Demonstrated Visually. Another line replaces text with additional visual demonstrations. Interaction2Code [Xiao et al., 2025a] provides state-pair screenshots and evaluates interaction
3
Table 1: Benchmark comparison for web application generation. Benchmark
Input modality
Output format
Image-only input
Multi-page app
Interaction eval
Inferred interaction
WebGen-Bench [Lu et al., 2025] Design2Code [Si et al., 2025] Interaction2Code [Xiao et al., 2025a] MRWeb [Wan et al., 2024] Vision2Web L2 [He et al., 2026] IWR-Bench [Chen et al., 2025a]
Text Single image Paired images Image + Text Images + Text Video
Framework HTML HTML HTML Framework HTML
✗ ✓ ✓ ✗ ✗ ✗
✓ ✗ ✗ ✓ ✓ ✗
✓ ✗ ✓ ✗ ✓ ✓
✗ – ✗ – ✗ ✗
UI2App (ours)
Images
Framework
✓
✓
✓
✓
categories via Selenium-driven probes. IWR-Bench [Chen et al., 2025a] instead supplies userinteraction videos that record state transitions over time, scored on functional and visual fidelity. In these benchmarks, the target interaction trajectory is provided beforehand and serves as a behavioral oracle against which model outputs are evaluated. UI2App, however, provides a single canonical screenshot per route and no target transition, making multi-state inference an intrinsic part of the task rather than information supplied in the input. Table 1 summarises this positioning across the closest prior benchmarks.
3
UI2App
3.1
Task Definition
A UI2App task supplies only a set of M screenshots (M = 4–14, mean 7.3), each capturing a distinct route or visual state of the target application. No action trace or interaction description is provided. Given this input, the model must produce the source code of a runnable application that reproduces the screenshots. All tasks and models share a fixed React + TypeScript scaffold, chosen as the most widely deployed front-end stack so that the benchmark reflects current production practice. 3.2
Dataset Construction
Source pool and filtering. We curate UI2App’s reference screenshots from opensource GitHub projects. To obtain a broad initial pool, we use 24 archetype-aware GitHub search queries spanning 12 application categories, yielding 2,013 candidate repositories. We then apply a four-stage automated pipeline that checks permissive licensing, structural validity, buildability, and authentication-wall detection. This leaves 164 repositories for expert review.
Intranet (2.2%) Learning App (2.2%) Todo (2.2%) Music Player (2.2%) Analytics Dashboard (6.7%) Vertical Admin (2.2%) CRM (4.4%) Mantine / Other Admin (4.4%) Tailwind Admin (4.4%) MUI Admin (6.7%) Ant Design Admin (6.7%) shadcn Admin (6.7%)
Portfolio (15.6%) Blog (8.9%) Specialty 15.6%
Admin 35.6%
Content 35.6%
Transaction 13.3%
SaaS Landing (6.7%) Documentation (4.4%) E-commerce (6.7%) Crypto Exchange (2.2%) NFT Marketplace (2.2%) Booking (2.2%)
Figure 2: UI2App dataset diversity.
Three-level expert selection. From the 164 usable repositories, expert reviewers select the 45 runnable multi-route reference applications using a three-level rubric. The rubric follows a progressive selection logic: page-level discriminability, application-level complexity, and corpus-level diversity. Rather than maximizing corpus size, we prioritize applications whose screenshots contain sufficient visual evidence for route recognition and latent interaction logic inference. • Page-level discriminability. Each retained screenshot must provide enough visual and semantic evidence for evaluation. Reviewers prefer pages with non-trivial layouts, diverse content regions, both atomic and composite UI components, and real text or image content. Placeholder pages, loading states, error pages, skeleton screens, empty states, and lorem-ipsum pages are removed. • Application-level complexity. Each retained application must form a genuine multi-route application rather than a set of near-duplicate pages. Reviewers filter out pseudo multi-page apps, require sufficiently rich screenshot-implied interactions across multiple IIS categories, and check
4
that the app renders stably at the fixed capture resolution without overflow, unhydrated content, or dev-server artifacts. • Corpus-level diversity. The final collection is selected to ensure diversity across application categories, subcategories, and design languages. Reviewers avoid repeated visual themes, duplicated upstream starters, and near-identical templates, so that evaluation reflects model capability rather than memorization of a small set of recurring layouts or scaffolds. Capture pipeline. Each application is captured by a headless-browser script at 1440×900, with timing and content-validation rules producing post-hydration screenshots free of dev-server overlays (full pipeline in Appendix A). Captures are deduplicated by perceptual hashing and expert-reviewed. Stratification and statistics. Applications are organised into four application categories by userfacing function: Content, Admin, Transaction, and Specialty. Subcategory listings appear in Table 8 (Appendix). 3.3
Evaluation Protocol Generate & Execute
INPUT Plan & Generate
M screenshots
Build & Render
LLM emits a file plan, then per-file code
Self-Debug
EXEC@1
Round 1 src/App.tsx import React from 'react" import ...
{
"files": [ {"path": ... {"path": ... ]
export default function App() { return ( <Router> <Routes> <Route path="/" } /> </Routes> </Router> > >
}
EXEC@3
Up to 3 stderr-fed repair rounds
Vite build + home-route render
stderr (truncated)
Error: Cannot find module "./Header' ...
Generate & Execute >_ $ vite build rendered ✔
Round 2
TypeError: Cannot read ...
}
Round 3
fail
stderr (truncated)
fail
stderr (truncated)
/ Build succeeded.
pass
Evaluate & Score Visual Fidelity
Coverage Aligns screenshots to generated routes Input screenshots
IIS & NRS
VFS
DOM match: Size/Text/Position/Color
Interaction taxonomy
Generated routes
IIS (Ours)
/(home)
/cart
著
...
/product/:id
Route reachability
-
...
-
...
Size
Text
Position
NRS
Color
Figure 3: The evaluation protocol of UI2App. The protocol decomposes evaluation into a sequence of automated stages for reproducibility, reserving a final human-annotation stage for behavioral judgments that cannot be reliably captured by runtime automated probes. Figure 3 illustrates the full chain. (1) Plan & Generate splits generation into a file-list pass followed by per-file content. (2) Build & Render attempts a production build and a home-route render, producing the per-attempt EXEC pass/fail signal. (3) Self-Debug on EXEC failure feeds the build’s error messages back to the same model for up to three repair attempts. (4) Coverage maps each input screenshot to its generated-app route and emits the page pairs consumed by VFS. (5) Visual Fidelity computes DOM-to-DOM similarity on the matched pairs, defining VFS. (6) Human annotation verifies navigation reachability for NRS and labels each interaction along the interaction taxonomy. 3.4
Standard Metrics: EXEC, NRS, VFS
EXEC@k (Executability). EXEC measures whether the generated source code actually runs, the protocol’s entry gate. An app passes when the production build completes without error and the home route renders meaningful content. EXEC@1 is the one-shot pass rate. EXEC@3, the pass rate after up to three rounds of error-feedback retry, separately measures iterate-fix capability. 5
NRS (Navigation Reachability Score). NRS measures route-level connectivity: the fraction of input-screenshot routes whose pages are reachable from the home route via visible navigation in the generated application, N
nreach a,m N RSa,m = min 1, , ntot a
N RSm =
100 X N RSa,m , N a=1
(1)
reach where ntot a is the number of input screenshots of application a, and na,m counts how many of these routes are reachable in model m’s artifact. Human verification is used because such navigation (nested menus, modal-triggered routing, conditional links) cannot be reliably enumerated programmatically.
VFS (Visual Fidelity Score). VFS scores what is rendered where via judge-free block-level matching: visible content blocks from each page are paired by optimal bipartite matching, and four sub-metrics (Size, Text, Position, Color) on the matched pairs are averaged unweighted. This design is robust to implementation differences in the underlying DOM. Algorithm and sub-metric formulas in Appendix F. Conventions. URL-direct loading: VFS and IIS are evaluated by loading each route directly via URL, isolating page implementation from routing. Zero-imputation: apps failing EXEC@3 contribute zero to NRS/VFS/IIS, keeping models with different EXEC@3 rates directly comparable. 3.5
Interaction Inference Score
Definition. IIS quantifies how well a model infers and realizes screenshot-implied interactions, jointly considering interaction coverage, implementation result, and state-logic complexity. Rather than comparing against a single reference behavior, IIS adopts a rubric-based evaluation in which each interaction is judged against the functional criteria. This implementation-agnostic formulation accommodates diverse yet semantically equivalent realizations of the same screenshot-implied interaction while enabling consistent evaluation across different applications. Interaction taxonomy and axes. To make interaction inference measurable, we organize interactions commonly found in web applications into seven categories: toggle, expand/collapse, list operations, data CRUD, form validation, notification, and cross-route state (C01–C07, cataloged in Appendix B.1). Each category is evaluated along three axes: coverage, indicating whether the interaction category is implied by the screenshots; result, indicating whether the generated application realizes the interaction category (fully realized, partially realized, or failed); and scope (S1 UI-state, S2 data-state, S3 cross-route persistence), reflecting increasing state-management complexity in real front-end engineering. Separating these axes enables failures to be attributed to specific interaction types and aspects, rather than being collapsed into a single aggregate score. Scoring and aggregation. For each application, we first construct a Reference Interaction Inventory (RII) from the input screenshots by labeling each interaction category with its coverage (whether the interaction is implied by the screenshots) and, when covered, its reference scope sref i . The covered categories form the reference set Ga , ensuring that only interactions inferable from the screenshots are evaluated. For each generated application, every category is annotated along the same three axes: coverage yi ∈ {0, 1}, implementation scope sgen ∈ {1, 2, 3}, and result ri ∈ {1, 0.5, 0} (working, i partial, or failed). Each scope level carries a linear weight, wS1 = 1, wS2 = 2, wS3 = 3. Combining the three axes yields the per-application quality score and the model-level IIS: P N gen ri 100 X i∈Ga yi · min wsi , wsref i P QSa,m = , IISm = QS , (2) N a=1 a,m i∈Ga wsref i with N the number of applications. The denominator is the sum of reference scope weights, which makes IIS recall-oriented. Annotation and validity. Each generated application is evaluated over all seven interaction categories, yielding 315 IIS items per model and 1,890 annotated items across the six evaluated models. Two front-end engineering experts annotate the reference side to construct the RII. Three annotators independently evaluate the generation side by running the generated applications, exercising each 6
interaction category, and scoring the observed runtime behavior, with disagreements resolved through arbitration. Manual annotation remains necessary because runtime probing alone cannot reliably determine whether different implementations satisfy the same interaction rubric. Inter-annotator agreement is substantial (Krippendorff α in 0.72–0.84). Further details are provided in Appendix B.
4
Experiments
4.1
Experimental Setup
We evaluate six frontier VLMs: Claude Sonnet 4.6 [Anthropic, 2026], Kimi K2.5 [Kimi Team, 2026] (thinking mode enabled), GPT-5.4 [OpenAI, 2026], Gemini 3.1 Pro Preview [Google DeepMind, 2026], Qwen3.5-397B-A17B [Qwen Team, 2026], and GLM-4.6V [Z.ai, 2025]. Each model is queried through its vendor’s official API. To study how performance scales within a single model family, we additionally evaluate a Qwen2.5-VL ladder (3B, 7B, 32B, 72B) under the same protocol. 4.2
Main Results
Overall ranking across the four metrics. Table 2 shows the four-metric panorama. Claude Sonnet 4.6 leads EXEC@1 (95.6%) and ties Gemini 3.1 Pro Preview at EXEC@3 (100%). Gemini leads VFS (78.1) and Claude leads IIS (39.3), while GLM-4.6V is lowest on every metric. Among the four metrics, IIS is where the six models diverge most sharply. Table 2: Per-category main results on UI2App. All scores except EXEC (%) are on a 0–100 scale. Best per (category, column) shown with sage cell + bold; worst with blush cell. Category
EXEC (%)
Model @1
NRS ∆
@3
VFS components
VFS
IIS
Size Text Pos. Color
Content
Claude Sonnet 4.6 100.0 100.0 0.0 Gemini 3.1 Pro Preview 100.0 100.0 0.0 100.0 100.0 0.0 Kimi K2.5 GPT-5.4 62.5 75.0 +12.5 Qwen3.5-397B-A17B 75.0 81.2 +6.2 56.2 56.2 0.0 GLM-4.6V
95.4 91.0 88.6 68.5 69.8 47.0
67.6 71.6 68.2 50.7 43.7 34.8
85.2 88.4 84.7 64.5 60.7 45.1
76.3 80.1 75.8 60.4 52.5 38.0
68.8 71.6 73.4 56.6 51.6 38.6
74.5 46.4 77.9 7.6 75.6 22.7 58.0 8.0 52.1 26.1 39.1 6.2
Admin
Claude Sonnet 4.6 Gemini 3.1 Pro Preview Kimi K2.5 GPT-5.4 Qwen3.5-397B-A17B GLM-4.6V
93.8 100.0 +6.2 81.2 100.0 +18.8 68.8 68.8 0.0 50.0 75.0 +25.0 50.0 62.5 +12.5 25.0 25.0 0.0
94.2 72.9 52.9 59.8 43.0 11.5
76.1 74.7 52.8 54.5 50.5 12.3
85.1 88.5 59.9 64.1 55.3 14.5
80.2 81.9 56.8 61.0 51.6 12.7
63.5 61.3 37.8 54.0 41.5 13.0
76.2 25.0 76.6 7.4 51.8 13.9 58.4 2.0 49.7 4.2 13.1 4.0
Claude Sonnet 4.6 83.3 100.0 +16.7 Gemini 3.1 Pro Preview 83.3 100.0 +16.7 Kimi K2.5 100.0 100.0 0.0 Transaction GPT-5.4 100.0 100.0 0.0 Qwen3.5-397B-A17B 66.7 83.3 +16.7 GLM-4.6V 33.3 33.3 0.0
69.7 68.0 59.3 58.3 38.3 9.4
76.5 77.6 73.3 69.3 60.6 24.6
85.6 90.4 85.8 89.2 69.3 30.5
68.5 72.1 68.1 71.9 58.4 24.0
79.2 87.4 82.7 87.2 65.6 27.6
77.5 81.9 77.5 79.4 63.5 26.7
51.8 11.6 30.7 13.5 11.0 0.5
Claude Sonnet 4.6 100.0 100.0 0.0 Gemini 3.1 Pro Preview 85.7 100.0 +14.3 Kimi K2.5 85.7 85.7 0.0 GPT-5.4 71.4 100.0 +28.6 Qwen3.5-397B-A17B 85.7 100.0 +14.3 GLM-4.6V 0.0 14.3 +14.3 Claude Sonnet 4.6 95.6 100.0 +4.4 Gemini 3.1 Pro Preview 88.9 100.0 +11.1 Kimi K2.5 86.7 86.7 0.0 GPT-5.4 64.4 82.2 +17.8 Qwen3.5-397B-A17B 66.7 77.8 +11.1 GLM-4.6V 33.3 35.6 +2.2
67.8 69.5 41.4 85.1 67.3 1.4 87.3 78.2 64.6 66.6 55.7 22.3
69.0 68.2 54.8 72.8 65.9 2.7 72.0 73.0 61.4 57.9 51.8 20.4
85.8 88.9 70.5 92.3 82.2 3.6 85.3 88.8 73.8 72.0 63.3 25.8
77.8 84.2 65.0 87.4 75.9 2.6 76.8 80.3 66.3 66.4 56.6 21.7
70.1 74.8 57.3 81.0 67.6 3.9 68.5 70.5 59.5 63.5 52.4 22.6
75.7 79.0 61.9 83.4 72.9 3.2 75.7 78.1 65.3 65.0 56.0 22.6
45.1 4.1 23.3 8.6 5.9 4.8 39.3 7.5 20.7 6.7 13.2 4.5
Specialty
Overall
EXEC self-debug heterogeneity. Self-debug benefit varies sharply across models (Table 2, ∆ column). GPT-5.4 recovers the most (roughly +18 points), Gemini and Qwen3.5 recover moderately, while Kimi shows essentially no gain and GLM-4.6V recovers only marginally (+2.2); Claude is already ceiling-bounded. EXEC@1 alone therefore substantially understates models that depend on error-feedback recovery, motivating the dual report. Despite its large EXEC gain, GPT-5.4 still ranks 5/6 on IIS, suggesting that self-debug improves build success rather than interaction inference. 7
VFS spread is largely build-driven. Conditioning on EXEC@3-passing apps (VFS⋆ , Table 6) shrinks the headline VFS range to under a 20-point gap and lifts GLM-4.6V’s visual-fidelity score from 22.6 to 59.8. The headline spread is thus largely build-driven; the remaining VFS⋆ differences trace to the visual-attribute sub-metrics (Size/Text/Position/Color), with the per-component decomposition reported in Appendix E. 4.3
Interaction Inference Deep Dive
Through a fine-grained analysis of IIS across interaction categories and scope tiers (Figure 4, Table 3), we derive the four findings below. 67.4 42.4 30.6 34.7 17.4 51.2 29.9 21.3 17.1 10.4 28.1 34.4 8.3 5.2 7.3 18.5 3.7 0.0 0.0 0.0 35.1 22.3 8.1 5.4 6.8 25.0 8.9 0.0 0.0 0.9 21.6 5.7 0.0 0.0 5.7
6.2 6.1 4.2 0.0 2.7 0.0 0.0
ude 2.5 3.5 ini -5.4 .6V Cla Kimi K Qwen Gem GPT LM-4 G
70
S1 (typ.)
60
Per-cell success rate (%)
C02 expand / collapse C01 toggle C05 form validation C06 notification C03 list operations C04 data CRUD C07 cross-route state
50 40
S2 (typ.) S3 (typ.)
30 20 10
Table 3: S-restricted IIS (capped, zeroimputation, §3.5). Population per scope: apps with ≥ 1 S-typical RII-positive item (n in header). Best per column bold, worst underlined. Model
0
Figure 4: IIS per-cell heatmap. Per-(category, model) success rate (×100, capped, zero-imputation) over 7 categories × 6 baselines; rows grouped by typical scope (S1/S2/S3); columns ordered by overall IIS.
S1
S2
S3
n=44 n=38 n=22
Claude Sonnet 4.6 Kimi K2.5 Qwen3.5-397B-A17B Gemini 3.1 Pro Prev. GPT-5.4 GLM-4.6V
48.5 29.4 19.9 15.9 9.2 6.2
31.6 16.4 6.9 4.3 5.6 1.3
21.6 5.7 0.0 0.0 5.7 0.0
spread
42.3
30.3
21.6
Finding 1 — Visual fidelity does not imply interaction-inference capability. The VFS leader and the IIS leader are different models: Gemini 3.1 Pro Preview ranks first on VFS (78.1) but fourth on IIS (7.5), while Claude Sonnet 4.6, second on VFS at 75.7, leads IIS at 39.3 (5.2× ahead of Gemini). Per-cell row-means (Figure 4) put Gemini at exactly 0.0 on C04 (CRUD), C06 (notification), and C07 (cross-route state), showing that pixel-faithful reconstruction can still miss core interaction logic. Finding 2 — The closed-vs-open quality hierarchy does not transfer to IIS. Two of the three open-weight entries (Kimi 20.7, Qwen3.5 13.2) outscore the two closed-frontier entries below Claude (Gemini 7.5, GPT-5.4 6.7). The split is IIS-specific, since Gemini leads VFS and ties Claude on EXEC@3 but trails only on IIS. The closed-vs-open ordering observed on VFS therefore does not carry over to interaction inference. Finding 3 — Cross-route state is a frontier-wide gap. S-restricted IIS (Table 3) decreases as scope becomes harder: S1 > S2 > S3 holds for nearly all models. The S3 end is a shared frontier limitation: three of six models score exactly zero and only Claude meaningfully clears the floor (at 21.6). Between-model spread also roughly halves as scope hardens, contracting from S1 to S3. S3 remains the benchmark’s most saturation-resistant signal as model families improve. Finding 4 — Admin apps are the consistent IIS bottleneck. Admin apps yield the lowest sixmodel mean IIS (9.4), roughly half of Content and Transaction. This gap is not explained by item count or cross-route incidence, since Transaction has comparable item count and more cross-route state but scores higher. The narrow model spread on Admin further suggests a shared limitation across current models. 4.4
Failure Analysis
Across the six frontier models, distinct failure mechanisms emerge on the executability, navigation, and interaction dimensions. EXEC@3 losses are dominated by scaffold-respect violations: 50% of failures violate one of four explicit plan-prompt constraints. Instruction compliance ranges from 100% (Claude / Gemini / GPT-5.4) down to ∼53% (GLM-4.6V). NRS reductions trace to two recurring patterns: mismatched navigation (a link points to no registered route) and hallucinated routes (pages 8
absent from the input screenshots). IIS zeros concentrate on S2 CRUD and S3 cross-route state. More detailed failure analysis is provided in Appendix G. 4.5
Within-Family Scaling (Qwen2.5-VL)
We complement the cross-family comparison with a within-family scaling analysis on the Qwen2.5VL ladder (3B, 7B, 32B, 72B). All four checkpoints are queried through the vendor’s official API with the same prompt templates. We report the three automatic metrics (EXEC@1, EXEC@3, VFS). NRS and IIS are not collected on this ladder because their human-evaluation cost over four additional ladder points exceeds our annotation budget. Macro means use zero-imputation for EXEC@3 failures, consistent with our main results. Non-gradual scaling: phase transition between 32B and 72B. Across the 24× size range, EXEC and VFS are essentially flat below 32B and jump sharply at 72B (Figure 5a): EXEC@3 from ≤ 2.2% to 62.2%, VFS from ≤ 0.2 to 35.2. Usable React-app generation thus emerges between 32B and 72B and is absent below it. Even at 72B, 37.8% of generations still fail to build, indicating that within-family parameter scaling alone does not saturate UI2App’s executability requirement.
EXEC@1 EXEC@3 VFS
Score (%)
80 60 40
35.2 33
20 3B
7B
9
32B
60
0
72B
(a) Scaling on three automatic metrics.
40
22
51
other hallucination syntax inconsistency pass
83
40 20
13
15
80 62
0
100
phase transition
Apps (%)
100
49
62
38 9
3B
7B 32B 72B
(b) Failure-mode composition.
Figure 5: Qwen2.5-VL scaling on UI2App (3B → 72B). a: EXEC@1 (circles), EXEC@3 (squares), and VFS (triangles, dashed); the shaded band marks the 32B → 72B phase transition. b: per-app build-failure category, stacked to 100%. Categories: syntax, inconsistency, hallucination, other, pass. Build-failure composition shifts with size, not just rate. The dominant build-error category shifts monotonically with size (Figure 5b). The smaller checkpoints fail at basic source-code competence (3B on syntax, 7B on inconsistency), while the larger ones fail in more semantically loaded ways (32B on a mix of syntax and hallucinated dependencies, 72B at deeper build stages). Each successive size escapes the previous bottleneck only to surface a more semantically loaded one, a pattern EXEC pass-rate alone does not reveal. Per-size counts and interpretation caveats are in Appendix G.2.
5
Conclusion
In this paper, we introduced UI2App, the first benchmark to measure interaction inference: recovering application behavior from image-only multi-page screenshots, with no textual or demonstration specification. Each artifact is scored along four dimensions (executability, navigation reachability, visual fidelity, and IIS), with IIS grounded in the interaction taxonomy that tiers interactions by state-logic complexity. Across six frontier vision-language models, visual fidelity proves a poor proxy for interaction realization: the VFS leader trails the IIS leader by 5.2×. Cross-route state persistence is a frontier-wide bottleneck, with three of six models scoring exactly zero and the strongest reaching only 21.6 on S3-scope interactions. UI2App provides a testbed for closing this S3 headroom. We hope it will drive vision-language models toward both stronger inference and more faithful realization of screenshot-implied interactions.
9
References Anthropic. Claude 4.6 sonnet, 2026. URL https://www.anthropic.com/news/claude-sonnet-4-6. Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cărbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: a vision-language model for ui and infographics understanding. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 3058–3068, 2024. Tony Beltramelli. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, pages 1–6, 2018. Yang Chen, Minghao Liu, Yufan Shen, Yunwen Li, Tianyuan Huang, Xinyu Fang, Tianyu Zheng, Wenxuan Huang, Cheng Yang, Daocheng Fu, et al. Iwr-bench: Can lvlms reconstruct interactive webpage from a user interaction video? arXiv preprint arXiv:2509.24709, 2025a. Yongxi Chen and Lei Chen. Psd2code: Automated front-end code generation from design files via multimodal large language models. arXiv preprint arXiv:2511.04012, 2025. Yunnong Chen, Shixian Ding, YingYing Zhang, Wenkai Chen, Jinzhou Du, Lingyun Sun, and Liuqing Chen. Designcoder: Hierarchy-aware and self-correcting ui code generation with large language models. arXiv preprint arXiv:2506.13663, 2025b. Yuhong Dai, Yanlin Lai, Mitt Huang, Hangyu Guo, Dingming Li, Hongbo Peng, Haodong Li, Yingxiu Zhao, Haoran Lyu, Zheng Ge, et al. Webvr: Benchmarking multimodal llms for webpage recreation from videos via human-aligned visual rubrics. arXiv preprint arXiv:2603.13391, 2026. Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, and Chao Wang. Advancing vision-language models in front-end development via data synthesis. arXiv preprint arXiv:2503.01619, 2025. Google DeepMind. Gemini 3.1 pro model card, 2026. model-cards/gemini-3-1-pro/.
URL https://deepmind.google/models/
Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Bohua Chen, Yi Su, Dongping Chen, Siyuan Wu, Xing Zhou, et al. Webcode2m: A real-world dataset for code generation from webpage designs. In Proceedings of the ACM on Web Conference 2025, pages 1834–1845, 2025. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, YK Li, et al. Deepseek-coder: when the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196, 2024. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Zehai He, Wenyi Hong, Zhen Yang, Ziyang Pan, Mingdao Liu, Xiaotao Gu, and Jie Tang. Vision2web: A hierarchical benchmark for visual website development with agent verification. arXiv preprint arXiv:2603.26648, 2026. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. Metagpt: Meta programming for a multi-agent collaborative framework. In The twelfth international conference on learning representations, 2024. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. Kimi Team. Kimi k2.5: Visual agentic intelligence, 2026. URL https://arxiv.org/abs/2602.02276. Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset. arXiv preprint arXiv:2403.09029, 2024. Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893–18912. PMLR, 2023.
10
Ryan Li, Yanzhe Zhang, and Diyi Yang. Sketch2code: Evaluating vision-language models for interactive web design prototyping. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3921–3955, 2025. Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebbench: How far have multimodal llms evolved in web page understanding and grounding? In First Conference on Language Modeling, 2024. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Webgen-bench: Evaluating llms on generating interactive and functional websites from scratch. Advances in Neural Information Processing Systems, 2025. OpenAI. Gpt 5.4, 2026. URL https://openai.com/index/gpt-5-4-thinking-system-card/. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), pages 15174–15186, 2024. Qwen Team. Qwen3.5: Towards native multimodal agents, 2026. URL https://qwen.ai/blog?id=qwen3.5. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950, 2023. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3956–3974, 2025. Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, and Michael R Lyu. Mrweb: An exploration of generating multi-page resource-aware web code from ui designs. arXiv preprint arXiv:2412.15310, 2024. Xian Wu, Ming Zhang, Zhiyu Fang, Fei Li, Bin Wang, Yong Jiang, and Hao Zhou. Vsa: Visual-structural alignment for ui-to-code. arXiv preprint arXiv:2512.20034, 2025. Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. ChartInsights: Evaluating multimodal large language models for low-level chart question answering. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12174–12200, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-emnlp.710. URL https://aclanthology.org/2024.findings-emnlp. 710/. Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zijie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, Liang Chen, Yuyao Zhai, Bang Liu, Chenglin Wu, and Yuyu Luo. Autowebworld: Synthesizing infinite verifiable web environments via finite state machines, 2026. URL https://arxiv. org/abs/2602.14296. Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zixin Wang, Xinyi Xu, Wenxuan Wang, Zhiyao Xu, Yuhang Wang, and Michael R Lyu. Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping. In 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 241–253. IEEE, 2025a. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R Lyu. Designbench: A comprehensive benchmark for mllm-based front-end code generation. arXiv preprint arXiv:2506.06251, 2025b. Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Q Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, et al. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms. Advances in neural information processing systems, 37:112134–112157, 2024. Z.ai. Glm-4.6v, 2025. URL https://huggingface.co/zai-org/GLM-4.6V. Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, et al. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation. arXiv preprint arXiv:2507.04952, 2025a.
11
Houston H Zhang, Tao Zhang, Baoze Lin, Yuanqi Xue, Yincheng Zhu, Huan Liu, Li Gu, Linfeng Ye, Ziqiang Wang, Xinxin Zuo, et al. Widget2code: From visual widgets to ui code via multimodal llms. arXiv preprint arXiv:2512.19918, 2025b.
12
A
Dataset construction details
Source-pool filter (§3.2). The four-stage filter referenced in the main text applies the following per-stage criteria to each of the 2,013 candidate repositories: 1. Permissive license. The project’s upstream LICENSE file (not package-manager metadata) must be MIT, Apache-2.0, BSD, or ISC. 2. Structural validity. The repository exposes a recognizable mainstream front-end framework layout with detectable component and route directories. 3. Buildability. Dependency installation and the production build both complete within a 180-second timeout on a clean environment. 4. Authentication-wall detection. Static heuristics over README/config files combined with HTTPredirect probing of the served root. Repositories surviving all four stages enter expert review. The cumulative pass rate is 164/2,013 ≈ 8%.
Screenshot capture pipeline. Captures use Playwright with headless Chromium at viewport 1440×900 and device_scale_factor=1. The four-stage timing waterfall referenced in §3.2 is: 1. domcontentloaded plus a 3-second hydration settle; 2. injection of dev-overlay suppression CSS (Next.js HUD, Vite, webpack); 3. wait_for_function polling for asset readiness (6-second timeout); 4. animation completion via document.getAnimations() (3-second timeout), followed by a 13selector loading-indicator detector with a 2-second recheck. A per-route login-redirect detector complements the repo-level authentication-wall gate. Surviving screenshots are deduplicated by perceptual hashing on the content area (32×32 average-hash, Hamming distance < 6%).
B
IIS taxonomy, and inter-annotator agreement
B.1
Category catalog
The seven interaction families spanned by IIS and their typical scopes are listed below.
B.2
Category
Description
Typical scope
C01 toggle C02 expand/collapse C03 list operations C04 data CRUD C05 form validation C06 notification C07 cross-route state
show/hide, style switch, on/off accordion, dropdown, drawer search, filter, sort, paginate create, read, update, delete required fields, format checks toast, banner, confirmation feedback state persistence across navigation
S1 S1 S2 S2 S1 S1 S3
State management and scope design
In production front-end engineering, interactive state is managed at increasing levels of architectural scope. The cheapest is component-local state (e.g., React useState/useReducer): state owned by a single component, bound to its mount lifecycle and not shared externally, such as whether a dropdown is open. When a piece of state is read or mutated by several sibling components, it must be lifted to a shared owner (a parent component, a React Context, or a store) so that updates stay consistent across the subtree; typical cases are a list with an active filter, or a selection shared between a toolbar and a table. The most demanding tier is global state that must survive client-side route changes: a single-page app re-mounts the route subtree on navigation, so any state that must persist (a cart, an auth session, a draft) cannot live in the page that gets unmounted and must instead live in a global store whose lifecycle is independent of routing (a Context at the app root, Redux/Zustand/Jotai, or storage-backed). Implementation cost broadly increases along this hierarchy (a local handler, then crosscomponent state lifting, then a routing-independent global store), and the difficulty of inferring the requirement from the screenshots alone rises with it. We turn this hierarchy into the three-level scope axis used by IIS, with an operational classification rule. An interaction is S1 (UI-state) when its effect is confined to the immediate rendered subtree (e.g., a toggle); S2 (data-state) when it performs a collection-level transformation (filter/sort/group/CRUD) that other components
13
must observe; and S3 (cross-route persistence) when its effect must outlive a route change (e.g., a cart that stays non-empty after navigating to /checkout). Because implementation cost broadly rises across S1<S2<S3, we weight each item by its scope with the linear ordinal weights wS1 =1, wS2 =2, wS3 =3 in Eq. 2, the simplest choice that respects this ordinal ranking without imposing an arbitrary curvature. The six-model IIS ordering is insensitive to this choice: recomputing model-level IIS under equal weights (1:1:1), the default (1:2:3), and a ratio-preserving alternative (1:2:4) leaves the full ranking unchanged (Table 4; Kendall τ =1.0 between every pair of weightings).
Table 4: IIS is insensitive to the scope-weight choice. Model-level IIS recomputed under three weightings of (S1, S2, S3); the six-model ranking is identical, and Kendall τ =1.0 between every pair of weightings. The 1:2:3 column reproduces the headline IIS of Table 2.
B.3
Model
1:1:1
1:2:3 (default)
1:2:4
Claude Sonnet 4.6 Kimi K2.5 Qwen3.5-397B-A17B Gemini 3.1 Pro Preview GPT-5.4 GLM-4.6V
42.1 23.8 15.4 10.3 7.9 4.8
39.3 20.7 13.2 7.5 6.7 4.5
38.9 20.1 13.0 7.2 6.6 4.4
Outcome rubric and paired-merge interpolation
For each cell, each annotator assigns the categorical outcome label W (working), P (partial), or F (failed), defined as: W, every behavior the category implies is realized at runtime; P, some implied behaviors are realized while others are not, or one is only partially realized; and F, none of the implied behaviors are realized (the affordance produces no effect). These map to the per-item score ri ∈ {1, 0.5, 0} used in Eq. 2. Under the paired design (§B, two annotators per cell), the cell’s ri is set as follows: agreement uses the agreed value; a one-step disagreement (W/P or P/F) is merged by interpolation to 0.75 or 0.25, so that a partial implementation flagged by one annotator does not collapse to an extreme; a two-step disagreement (W/F) triggers arbitration by the third annotator, after which ri takes the arbitrated label.
B.4
Inter-annotator agreement
Three annotators (A, B, C) labeled in a paired design: 15×{A+B}, 15×{A+C}, 15×{B+C}. Arbitration was triggered on coverage mismatch, result gap > 1, or scope gap > 1. Agreement values across the four labeled annotation fields are summarized in Table 5. 95% bootstrap CIs are computed by resampling the 45 paired-labeled (app, model) runs (15+15+15) with B=1,000, seed 42.
Table 5: Inter-annotator agreement on the four labeled annotation fields. gen_appl (the coverage axis) is binary so reports nominal α and Fleiss κ. gen_result (result) and gen_scope (scope) are ordinal so report αordinal . reachable_routes (the number of input routes reachable from the home page) is a count so reports ICC(2, 1).
C
Axis
Statistic
gen_appl gen_result gen_scope reachable_routes
αnominal / Fleiss κ αordinal αordinal ICC(2, 1)
Value
95% CI
0.77 / 0.73 0.72 0.84 0.86
[0.71, 0.83] [0.66, 0.78] [0.79, 0.88] [0.81, 0.90]
Provider-rejected projects
For GPT-5.4, one project was rejected by the OpenAI content-safety filter before the model received any input screenshots. The main table (Table 2) counts the rejected project as 0 in EXEC@1, EXEC@3, and VFS (zero-imputation, denominator N =45). Two alternative conventions yield: (i) N =44 zero-imputation (rejected project excluded from the denominator; remaining build/runtime failures still scored 0 under zero-imputation): EXEC@3 =84.1% (37/44), VFS=66.4; (ii) N ⋆ =37 conditional on EXEC@3-pass (the VFS⋆ convention restricted to the 44 evaluable projects). GPT-5.4’s EXEC@3 ranking and the broad strong/weak tiering are invariant across all three conventions (it stays fourth, between Kimi and Qwen).
14
D
Conditional metrics: VFS⋆ and IIS⋆
Two denominator conventions. UI2App reports two complementary scores for VFS and IIS. The headline scores (main table) average over the full curated set N =45 with zero-imputation: an application that fails to build after three self-debug rounds (EXEC@3=F) contributes 0 to the model’s average, penalizing buildability and the underlying capability jointly. This is the most direct measure of end-to-end usability. The conditional variants VFS⋆ and IIS⋆ restrict the average to the N ⋆ projects that pass executability (EXEC@3=P, i.e., the build and home-route render both succeed), isolating visual-language understanding and interaction-inference capability from buildability. We report both because the two factor different concerns into the same score: • Headline (zero-imputation, N =45): is the rendered output of a complete model run usable on average? • Conditional (VFS⋆ / IIS⋆ , N ⋆ ≤N ): given the model produced something at all, how visually faithful or interaction-faithful is it? Table 6 reports VFS/VFS⋆ and IIS/IIS⋆ per model alongside their headline→conditional gaps. The headline→conditional gap is exactly the contribution of build failures to the headline score. Models with no build failures (Sonnet 4.6, Gemini 3.1 Pro Preview at EXEC@3=100%) have ∆=0 by construction; highbuild-failure models open a large gap. GLM-4.6V’s +37.2-point gap on VFS (22.6 → 59.8) is the most extreme: it has substantial visual-language ability (N ⋆ =16 projects average to 59.8), but only 35.6% of its outputs build at all, so its end-to-end VFS sits last. On IIS, the conditional score IIS⋆ similarly lifts GLM-4.6V from headline-last to mid-pack (4.5 → 12.6). The top three preserve their order (Claude 39.3 → 39.3, Kimi 20.7 → 23.9, Qwen3.5 13.2 → 17.0); the bottom three reshuffle because GLM’s small 16-app build-pass subset yields a higher per-app IIS than its full-set headline, moving it past Gemini (7.5 → 7.5) and GPT-5.4 (6.7 → 8.1) on the conditional view. The top-three preservation confirms that the IIS gaps among the strongest models in Table 2 reflect interaction-inference capability rather than executability alone.
Table 6: Conditional metrics VFS⋆ and IIS⋆ . Both restrict the average to projects with EXEC@3=pass; N ⋆ is the pass-conditioned project count. The headline→conditional gap separates executability from intrinsic capability. Best per column in bold, worst underlined; the largest absolute ∆ (most build-constrained model) is bolded. Model
N EXEC@3 (%) N ⋆ VFS VFS⋆
Claude Sonnet 4.6 Gemini 3.1 Pro Preview Kimi K2.5 GPT-5.4 Qwen3.5-397B-A17B GLM-4.6V
45 45 45 45 45 45
100.0 100.0 86.7 82.2 77.8 35.6
45 45 39 37 35 16
75.7 78.1 65.3 65.0 56.0 22.6
∆V F S
IIS IIS⋆ ∆IIS
75.7 +0.0 39.3 39.3 +0.0 78.1 +0.0 7.5 7.5 +0.0 75.3 +10.0 20.7 23.9 +3.2 79.0 +14.0 6.7 8.1 +1.4 72.0 +16.0 13.2 17.0 +3.8 59.8 +37.2 4.5 12.6 +8.1
Choice for the main table. We headline VFS and IIS (zero-imputation) in Table 2 because end-to-end usability is the property the benchmark targets: a generation run that does not build is unusable to a real user, and its visual or interaction quality on hypothetical-builds-only is not actionable. VFS⋆ and IIS⋆ are reported as diagnostics that isolate visual-language and interaction-inference capability from build success: they measure output faithfulness over only the projects a model builds, independent of how often it builds.
E
Conditional sub-metric breakdown (VFS⋆ decomposition)
The four zero-imputation VFS sub-metrics (Size, Text, Position, Color) are reported as columns of the main results table (Table 2); (SIZE+T EXT +P OSIT ION +COLOR)/4 reconstructs the headline VFS exactly. That zero-imputation view conflates intrinsic visual sub-skill with build success: Gemini 3.1 Pro Preview leads on all four sub-metrics in line with its VFS lead, and GLM-4.6V is lowest on all four, reflecting that its weak VFS stems from buildability (35.6%) rather than a sub-skill-specific deficit. Table 7 reports the conditional decomposition that isolates sub-skill from build success: each sub-metric is averaged across the matched pages of the model’s EXEC@3-pass projects only (N ⋆ apps, npages pages per model), so it decomposes VFS⋆ rather than VFS. The conditional view reveals two patterns invisible in the zero-imputation main table: (i) Sub-skill gaps narrow once buildability is controlled for: Position spread shrinks from 58.6 points (zero-impute) to 9.4 points (conditional), Color from 47.9 to 11.9. Visual sub-skill differences across the six models are several-fold smaller than overall VFS differences would suggest; most of the spread comes from build success, not from visual proficiency. (ii) GLM-4.6V’s signature is strong palette plus weak spatial composition: it is last on Size and Position but first on Color (80.8, ahead of GPT-5.4’s 77.7). The Color advantage is partly a selection-bias
15
Table 7: Conditional sub-metric breakdown (VFS⋆ decomposition). Per-page average over matched pages of EXEC@3-pass projects only; npages varies per model and is reported alongside. Best per column in bold (ties shown in bold for both rows), worst underlined. Note the unequal npages across models: GLM-4.6V’s reported scores draw from 76 pages (16 build-pass apps) versus Sonnet’s 318 pages (45 apps), so cross-model comparisons here are sub-skill estimates conditional on build success and should be read as upper bounds on what each model could deliver if its build failures were eliminated. Model
N⋆
npages
Size
Text
Position
Color
Gemini 3.1 Pro Preview Claude Sonnet 4.6 Kimi K2.5 GPT-5.4 Qwen3.5-397B-A17B GLM-4.6V
45 45 39 37 35 16
315 318 262 271 228 76
75.6 74.1 74.3 73.1 74.8 70.6
91.7 87.0 88.7 90.1 90.0 89.2
83.3 79.2 80.4 83.3 81.2 73.9
71.4 68.9 69.5 77.7 74.3 80.8
artifact (npages =76 from 16 build-pass apps), but the cross-validation across the four sub-metrics within those 16 apps still suggests GLM-4.6V allocates representational capacity preferentially to color over geometry.
F
VFS algorithm: DOM-alignment matching and cascade fallback
Overview. VFS is a fully judge-free metric: no VLM, LLM, or pixel-classifier is invoked at any stage. tot
Given a reference application R and a model-generated application G on the same input set I={(si , ri )}n i=1 (screenshot si at reference route ri ), VFS is produced by the pipeline of Algorithm 1: (A) extract the route set of G from src/App.tsx; (B) boot the generated dev server and render each candidate route in headless Chromium under an auth/gate-dismissal protocol; (C) extract reference-side DOM text-blocks from R, falling back to OCR on the input PNG when R cannot be reached at ri (auth-locked routes, missing dependencies); (D) match input screenshots to generated routes by global Hungarian assignment on the M ×N matrix of reference-vs-generated DOM-block alignment scores, with a path-token Jaccard tie-breaker and an absolute floor; (E) score the aligned-block set of each matched pair on four sub-metrics (Size, Text, Position, Color); (F) aggregate to page-, application-, and model-level. A legacy cascade T1 –T5b matcher is retained as a fallback for the rare case where Sref fails to start (§F.1). All hyperparameters are held constant across all models and applications. Specific values are stated inline at each point of use.
Auth and gate handling. React applications in UI2App routinely guard non-public routes behind authentication or splash gates that, if not dismissed, cause every captured route to render the same login/intro view. We apply three source-derived mitigations on the generated dev server (no per-app human configuration), invoked at Algorithm 1 lines 2–4: (i) localStorage auth seeding: scan src/**/*.[jt]s* for storage call sites whose key matches the regex /(auth|session|login|signin|loggedin|token|user)/i and inject a permissive auth payload via Playwright add_init_script; (ii) in-memory auth form-fill: when (i) finds no localStorage hooks but a useState(false) auth flag exists, navigate to /login, fill synthetic credentials, submit, and switch subsequent navigation to pushState so React state survives; (iii) splash/intro gate dismissal: AST-detect the useState(false) + early return pattern in App.tsx, regex-extract the dismiss-button label, and dismiss on every non-/ route (the home is captured with the gate visible to match the reference). The same localStorage seeding is also applied to Sref ; routes that seeding cannot bypass fall through to OCR.
Render-validity probe. For each rendered route we evaluate a four-signal probe on #root: all (descendant element count), text (innerText length), visuals (svg/img/canvas/video count), and interactive (button/input/textarea/select/a[href]/[role="button"] count). The route is accepted as rendered if all≥3 and at least one of: text≥20 chars, visuals≥2, or interactive≥2. The any-of disjunction (rather than a stricter conjunction such as “text>10 ∧ all≥5”) admits legitimate splash/CTA pages with shallow DOM but meaningful content (e.g., a <div><button>Skip</button><p>...</p></div> Intro component). Routes that fail the probe are recorded as None and become MISS candidates for the matcher.
Route-alias deduplication. After each successful screenshot we hash the page’s innerText+child-count signature and consult a per-app seen_dom_hashes table. If the hash recurs we record the new route as an alias of the canonical route and reuse the canonical screenshot rather than writing a duplicate. This collapses redirects (<Route path="/" element=<Navigate to="/foo">>) and HashRouter mirror routes that would otherwise inflate ntot and double-count gen content.
16
DOM-alignment matching (default path). We extract DOM text-blocks for every rendered gen route (cached as Br̂gen ) and, on the reference dev server, for every input route (cached as Bsrefi ). When the reference DOM at ri comes back empty (route auth-locked, dependency missing, build/dev-server failure on R at this route), we substitute OCR-derived blocks: Tesseract on the input PNG produces line-level boxes which are normalized to the same { text, bbox, fg_color, tag } schema as the DOM extractor (Color is degraded for OCR-derived blocks since fg_color defaults to black). Let M be the number of inputs and N the number of rendered gen routes. We build the similarity matrix Σij = compute_l3_page(Bsrefi , Br̂gen ) / 100, Σ′ij = min 1, Σij + β · Jacc(τ (ri ), τ (r̂j )) , j where compute_l3_page is the four-sub-metric scorer (§Sub-metrics below), τ (·) tokenizes a path on [-_/]+, and Jacc is set-Jaccard. The token bonus β=0.08 acts as a tie-breaker when DOM-alignment scores are within ∼0.05 of each other (e.g., gen /leads beats gen /settings/team for the input labeled leads, both candidates at Σ≈0.55), without overriding strong DOM evidence. We solve the assignment globally with the Hungarian algorithm (scipy.optimize.linear_sum_assignment) on cost C=1−Σ′ padded to the square size max(M, N ) with cost 1, then drop pairs whose underlying Σij falls below the absolute floor θDOM =0.20 (the input becomes MISS and contributes 0 to nmatched ). Each gen route is used at most once across the assignment. This hard cap is equivalent to setting Kmax =1 (the same parameter that the cascade fallback uses, see §F.1) and prevents a single dynamic route (e.g., a /post/:id render) from absorbing three to five unrelated inputs and over-counting matched routes.
DOM block extraction. We navigate Playwright to the route under evaluation, wait until domcontentloaded, dismiss any detected gate (only on non-/ routes; see auth-and-gate handling above), and idle for δref =4,000 ms (reference) or δgen =3,000 ms (generated). Both renderings use viewport 1440×900 to match the reference capture. We then run an in-page TreeWalker over text nodes and emit one block per visible parent element with: (i) lower-cased text trimmed to 200 characters; (ii) bounding box normalized to viewport so that (x, y, w, h) ∈ [0, 1]2 ×[0, 1]2 ; (iii) the foreground color from getComputedStyle().color; (iv) the parent’s tag name. Non-rendering elements (display:none, visibility:hidden, opacity 0), boxes smaller than 2×2 pixels, content more than three viewports below the fold (pixel-space ypx >3vh where vh =900 px is the viewport height), and text strings longer than 500 characters are filtered out before the bbox is normalized to [0, 1]2 . Adjacent same-row same-tag fragments are merged when within 0.02 normalized units, suppressing the spurious split of inline-styled text into multiple blocks. n gen m Block alignment within a pair. Let B ref ={bref ={bgen i }i=1 and B j }j=1 . We build the cost magen trix C ∈ Rn×m with Cij =1−ρ(tref i , tj ) where ρ= difflib.SequenceMatcher.ratio is the Ratcliff– Obershelp similarity (longest common contiguous matching) ρ(a, b)=2L/(|a|+|b|) with L the Pmatchedcharacter count. We pad C to the square size max(n, m) with cost 1, solve (σ ∗ , π ∗ )= arg min Cσπ via the Hungarian algorithm (scipy.optimize.linear_sum_assignment), and retain only assignments whose ρ≥θtext =0.3. The retained assignment set A is the basis for all four sub-metrics. gen Sub-metrics. For a pair p=(bref p , bp ) let a(b) denote area, c(b)=(x+w/2, y+h/2) the center, fg(b) the
foreground color, and ∆E00 the CIEDE2000 color difference computed in CIE Lab via skimage.color. The four sub-metrics are ! P ref gen 1 p∈A 2 a(bp )+a(bp ) , SIZE = min 1, P P gen 1 ref ) a(b )+ a(b i j j i 2 T EXT = P OSIT ION = COLOR =
1 X gen ρ tref , p , tp |A| p∈A 1 X max 0, 1− max |∆cx |p , |∆cy |p , |A| p∈A 1 X gen max 0, 1−∆E00 fg(bref p ), fg(bp ) /100 . |A| p∈A
We use Chebyshev (max-norm) displacement for Position rather than Euclidean because the four-metric average should treat horizontal and vertical drift symmetrically while remaining bounded in [0, 1] without an additional normalization constant. CIEDE2000 is divided by 100 since saturated complementary colors produce ∆E00 ≈100 in the Lab gamut. Each sub-metric defaults to 0 if |A|=0 (no aligned blocks).
Aggregation. The page-, application-, and model-level scores are ℓpage = 25 (SIZE+T EXT +P OSIT ION +COLOR), p
Lmatched = 3,a
1 X page ℓp , |Pa | p∈P a
17
Algorithm 1 Computation of VFS for a single (application, model) pair (DOM-alignment default; cascade fallback in §F.1). tot
Require: Input set I={(si , ri )}n i=1 ; reference source R; generated source G. 1: T ← E XTRACT ROUTES(G/src/App.tsx) ▷ indent-aware nested-<Route> parser; concretize :id→1 2: g ← D ETECT G ATES(G); σg ← D ETECT L OCAL S TORAGE S EEDS(G); ι ← D ETECT I N M EMO RYAUTH (G) when σg =∅ 3: σr ← D ETECT L OCAL S TORAGE S EEDS(R) 4: Sgen ← S TART D EV S ERVER(G); apply seeds σg via add_init_script 5: Hgen , B gen , aliases, hashes ← ∅, ∅, ∅, ∅ 6: if ι then capture pre-auth Login at __login__; F ORM F ILL L OGIN(Sgen ); switch to SPA pushState 7: end if 8: for r̂ ∈ sort(T ) ▷ / first, then by |r̂.url| do 9: navigate Sgen →r̂.url; if r̂.url̸=/ and g.has_gate then T RY D ISMISS(Sgen , g.selectors) 10: probe ← {all, text, visuals, interactive} on #root 11: if not I S R ENDERED(probe) then Hgen [r̂] ←None; continue ▷ all≥3 ∧ (text≥20 ∨ visuals≥2 ∨ interactive≥2) 12: end if 13: h ← md5(innerText | all); if h ∈ hashes then aliases[r̂] ← canon(h); Hgen [r̂] ← Hgen [canon(h)]; continue 14: Hgen [r̂] ←S CREENSHOT(Sgen ); Br̂gen ←E XTRACT B LOCKS(Sgen ); hashes[h] ← r̂ 15: end for 16: if σg ̸=∅ and __login__∈H / gen then render / in a fresh non-seeded context, save as __login__ 17: end if 18: S TOP D EV S ERVER(Sgen ) 19: Sref ←S TART D EV S ERVER(R); apply seeds σr ; if fail then goto cascade fallback (§F.1) 20: B ref ← ∅ 21: for (si , ri ) ∈ I do 22: b ←E XTRACT B LOCKS(Sref , ri with g-aware dismissal) 23: if b=∅ then b ←OCRB LOCKS(si ) ▷ Tesseract on input PNG; COLOR degraded to placeholder 24: Bsrefi ← b 25: end for 26: S TOP D EV S ERVER(Sref ) gen 27: Σ ∈ RM ×N with Σij ← compute_l3_page(Bsrefi , Br̂j )/100 (rendered routes only) ′ 28: Σij ← min(1, Σij + β · Jacc(τ (ri ), τ (r̂j ))) with β=0.08 29: (σ∗, π∗ ) ←H UNGARIAN(1−Σ′ ) on the max(M, N )-square padded matrix (pad cost 1) 30: P ← {(si , r̂j ∗ ) : Σij ∗ ≥ θDOM =0.20}; nmatched ← |P | ▷ each r̂ used at most once 31: L ← [ ] 32: for (si , r̂ ∗ ) ∈ P do 33: A ←H UNGARIAN(Bsrefi , Br̂gen ▷ within-pair text-dissimilarity alignment ∗ ; θtext =0.3) 34: ℓpage ← 25 (SIZE+T EXT +P OSIT ION +COLOR); push ℓpage to L ▷ Eq. (3) 35: end for 36: Lmatched ← mean(L); V F Sa,m ← Lmatched · nmatched /ntot 3 3 37: return V F Sa,m
V F Sa,m = Lmatched 3,a,m ·
nmatched a,m , ntot a
V F Sm =
Napp 1 X V F Sa,m , Napp a=1
(3)
with Napp =45 (the curated app count; the per-app local N in Algorithm 1 denotes the number of rendered gen routes and is unrelated). The coverage factor nmatched /ntot penalizes models that achieve a high per-page score on a small subset of routes. For example, 1/12 matched at Lmatched =31.7 degrades to V F S=2.6 rather than 3 retaining the per-page score. Consistent with the scoring policy of §3.3, applications with EXEC@3=F are zero-imputed at the application level. The conditional variant VFS⋆ (Appendix D) drops the zero-imputation, averaging only over EXEC@3=P applications.
F.1
Cascade fallback (T1 –T5b , used only when Sref fails to start)
When the reference dev server Sref cannot be started for an application (missing node_modules, install timeout, port collision under heavy parallelism), the DOM-alignment matrix in Algorithm 1 is undefined and we fall back to a path-name + visual cascade. Each input is matched to at most one rendered gen route (Kmax =1, including dynamic routes) and a hit at tier Tt skips Tt+1 , . . . , T5b .
18
T1 Exact route match. The reference route ri (lower-cased, slashes/query stripped) equals the generated route’s literal definition or its concretized URL. T2 Dynamic-route regex match. Generated routes containing path parameters (:slug, :id) are converted to [ˆ/]+ regex. Catches /product/1 ↔ /product/:id. T3 Path-suffix match. Largest common trailing-segment overlap k≥1. Admits /dashboard ↔ /app/dashboard. T4 Synonym structural match. Tokenize on [-_/] and expand by a fixed table of 32 synonym groups (e.g. {home, landing, index, overview}; {contact, support, etc, misc, links, elsewhere}; {signin, signup, forgot, otp, verify}); a hit requires non-empty intersection with at least one cross-direction unexpanded-vs-expanded overlap. T5a Visual-first phash Hungarian. Build the M ×N phash-similarity matrix on 16×16 perceptual hashes of input PNG vs gen screenshot, solve Hungarian on 1 − sim, and accept pairs with sim≥θphash =0.45. T5b Visual fallback (legacy SSIM+hist). For inputs unmatched after T1 –T5a , compute 0.7 SSIM + 0.3 HistCorr on 256×160 resized RGB; accept the best candidate only if it satisfies both the relative abs margin best≥1.2µ̄ and the absolute floor θvis =0.55. The absolute floor blocks “least bad” assignments from a uniformly poor candidate set. For instance, it rejects a zen-keyboard screenshot pairing to /player at SSIM+hist≈0.45.
Implementation notes and known limitations. The DOM-alignment default is robust to route renaming (gen / that actually serves Blog content correctly pairs with the reference’s /blog input via DOM evidence rather than path string) and rejects path-name false positives (gen / that serves Blog cannot pair with a reference / that serves a music landing). The token-Jaccard bonus β=0.08 is small enough that DOM evidence dominates whenever it is decisive, and large enough to break ties in the noisy 0.5–0.65 similarity band typical of darktheme/icon-heavy apps. The OCR fallback for Bref degrades Color (no real foreground color information) but preserves Size, Text, and Position. Because it is applied reference-side—identical for all six models on a given application—it cannot bias cross-model Color comparisons. This is the maximal signal recoverable for routes the reference dev server cannot reach. The strict Kmax =1 cap occasionally drops the second of two near-identical inputs (e.g., two distinct product-detail screenshots both matching the same /product/:id). This is a deliberate trade-off that removes the much larger error mode of dynamic routes absorbing semantically unrelated inputs. Because input screenshots in UI2App are URL-reachable static routes (§3.2), the pipeline does not replay click sequences before ExtractBlocks. Interaction-state fidelity is captured by IIS (§3.5) rather than VFS. We report VFS as observed under this protocol. An interaction-aware variant is left to future work. Table 8: Application-category composition by subcategory. The four application categories resolve into the subcategories listed below, spanning the realized dataset diversity. Subcategories are descriptive and not used as analytical strata.
G
Category
Share
Subcategories
Content Admin
35.6% 35.6%
Transaction Specialty
13.3% 15.6%
Portfolio, Blog, Documentation, SaaS Landing shadcn Admin, Ant Design Admin, MUI Admin, Tailwind / daisyUI Admin, Mantine / Other Admin, CRM, Vertical Admin (food) E-commerce, Crypto Exchange, NFT Marketplace, Booking Analytics Dashboard, Music Player, Todo, Learning App, Intranet
Failure taxonomy details
This appendix expands the failure analysis of §4.4 into a breakdown across the benchmark’s four scoring axes. We first decompose the 52 unrecovered EXEC@3 build/runtime failures into a nine-mode taxonomy (Table 9) with per-model incidence, prompt-clause attribution, and representative code anchors, then trace how the dominant mode shifts along the Qwen2.5-VL size ladder. We next specify the fixed self-debug feedback protocol under which EXEC@3 is measured, and the recurring patterns that reduce NRS, VFS, and IIS. A closing set of case studies illustrates the latent-affordance inference that IIS targets. Counts are over the 270 nominal (app, model) generation pairs spanning 45 apps and 6 baselines; one additional (GPT-5.4, app) pair rejected by the vendor’s content-safety filter is treated separately (Appendix C) and is excluded from these 52 failures.
G.1
Generation-level failures
The 52 unrecovered EXEC@3 failures decompose into nine fine-grained modes (Table 9), which collapse into five high-level causes: hallucinated imports (C1, C6); scaffold-file overwrites (C2, C8); undeclared / wrong-name source dependencies (C3); broken import paths (C4, C5); and parse-level or runtime errors (C7, C9).
19
Table 9: Build-failure modes across 52 unrecovered EXEC@3 failures. The Share column gives the mode’s percentage of all 52 failures; per-model columns give the mode’s incidence as a percentage of that model’s 45 apps. “IFV” marks instruction-following violations against the four explicit plan-step prompt constraints (Appendix G.1). Models: C=Claude Sonnet 4.6, Ge=Gemini 3.1 Pro Preview, Gp=GPT-5.4, K=Kimi K2.5, Q=Qwen3.5-397B-A17B, GL=GLM-4.6V. Percentages are rounded to the nearest integer, so the Share column need not sum to exactly 100% and a model’s per-mode entries need not sum exactly to its Failure rate. ID
Failure mode
Share IFV
C
Ge
Gp
K
C1 C2 C3 C4 C5 C6 C7 C8 C9
Hallucinated icon export (lucide-react) Scaffold package.json overwritten Pkg used but not in extra_dependencies @/* path alias used Wrong relative path Cross-file import/export mismatch JSX / TypeScript parse error Scaffold src/index.css overwritten Runtime API misuse
37% 35% 10% 4% 2% 6% 4% 2% 2%
0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0
11% 0 0 0 0 0 2% 0 2%
9% 16% 7% 2% 7% 31% 2% 0 9% 0 0 4% 0 0 2% 0 0 7% 0 0 2% 0 0 2% 0 0 0
Failure rate (failures / 45 apps) ICS = 1 − nIFV /napps
✓ ✓ ✓
✓ — —
Q
GL
0 0 16% 13% 22% 64% 100% 100% 100% 96% 93% 53%
C1: hallucinated lucide-react icon export (37% of failures). The model imports a symbol not exported by the scaffold-pinned lucide-react@ˆ0.460.0: typically a brand icon (Tiktok, Spotify, Weibo, Alipay, etc.), a future-renamed name (Funnel, the post-0.460 canonical name introduced when the icon previously exported as Filter was renamed), or a foreign-library identifier (DashboardOutlined from Ant Design). Anchor: GPT-5.4 on 55_shadcn-nextjs-dashboard writes import { Funnel } from "lucide-react", but in 0.460.0 the icon is still exported as Filter.
C2: scaffold package.json overwritten (35% of failures). The model emits a package.json in violation of the “Do NOT include scaffold files” clause, pinning unpublished versions or inventing package names (e.g., mantine as a top-level package; registry name is @mantine/core). pnpm install aborts with ERR_PNPM_NO_MATCHING_VERSION / ERR_PNPM_NO_VERSIONS. Anchor: GLM-4.6V on 60_mantine-dashboard.
C3: source uses pkg not in extra_dependencies (10%). The plan-step prompt requires a structured extra_dependencies field for non-scaffold packages, but the source imports a package neither pinned in the scaffold nor declared. Declaring non-scaffold dependencies is the model’s responsibility under the plan-step contract, and the Self-Debug stage does not backfill the missing declaration by scanning generated source. Anchor: GLM-4.6V on 54_slash-admin declares react-calendar but imports react-big-calendar.
@/* path alias used (4%). The model writes shadcn-style imports (import X from "@/components/X") although the prompt requires relative imports and the scaffold’s tsconfig.json declares no such alias. C4:
C6: cross-file export name mismatch (6%). The importer references an identifier whose source declaration uses a different name (default vs. named, casing mismatch, or omitted altogether). Anchor: GLM-4.6V on 38_dexkit-nft-marketplace uses <CollectionCard> at Home.tsx:31 without a matching import. vite build passes (esbuild does not type-check undefined identifiers) but the home route throws ReferenceError on first render, failing EXEC’s home-route render gate (§3.4).
Long-tail modes C5, C7, C8, C9 (10% combined). C5 (wrong relative path): file-root-relative specifier instead of ../. C7 (JSX/TS parse error): unbalanced closing tag, stray >. C8 (scaffold src/index.css overwritten): @layer base without @tailwind base. C9 (runtime API misuse): GPT-5.4 on 18_myPortfolio destructures a non-existent location field from the <NavLink className={...}> render-prop, which in react-router-dom@6 provides { isActive, isPending, isTransitioning }.
Per-model signature and prompt-clause attribution. Cases are assigned to the earliest-firing root cause: install-time (C2) > build-time > runtime (C9); module-resolution (C3, C4, C5) > symbol-export (C1, C6) within build. Per-model totals (Table 9) reveal distinct signatures: Claude and Gemini produce zero failures; GPT-5.4 (16% of its 45 apps) is dominated by C1 (71% of its failures, all factuality); GLM-4.6V (64% of its apps) is dominated by scaffold-respect violations (C2+C3+C4+C8 ≈ 72%). The plan-step prompt (Appendix H)
20
states four mechanically-checkable constraints: (i) source files under src/; (ii) relative imports; (iii) every non-scaffold import declared in extra_dependencies; (iv) no regeneration of scaffold files. Half of all failures are instruction-following violations (IFV column of Table 9): clause (iv) accounts for 73% of IFVs via C2 + C8, clause (iii) for ∼20% (C3), clause (ii) for 8% (C4), and clause (i) has zero violations. The other half (C1, C5, C6, C7, C9) are coding-quality or factuality errors not pre-empted by the prompt. The Instruction-Compliance Score (ICS) row shows the resulting model-level split. Self-debug retries a failing app up to three rounds with the same model and a fixed feedback window. An app failing after round 3 is scored EXEC@3=F under this fixed, model-identical feedback protocol (Appendix G.3).
G.2
Per-size failure-mode shifts on the Qwen2.5-VL ladder
Figure 5b composes the build-failure category at each size, supporting the dominant-shift narrative in §4.5: the dominant category shifts from syntax (3B) to inconsistency (7B) to a mixed syntax + hallucination regime (32B). The post-transition 72B residual is dominated by syntax at deeper build stages plus the figure’s other bucket, here runtime errors (e.g., importing TypeScript-only types as values). These four buckets correspond to the C1–C9 taxonomy of §G.1 as: syntax⊇C7, inconsistency⊇{C2,C3,C4,C5,C8}, hallucination⊇{C1,C6}, other⊇C9. Each successive size escapes the previous bottleneck only to surface a more semantically loaded one. We caution that the breakdown at ≤ 32B is over a near-fully-failing population (≥ 44/45 failures per size), so the precise within-size category proportions are noisy; the identity of the dominant (mode) category, taken over 44–45 failures, is well determined and is what the cross-size dominant-shift claim relies on.
G.3
Self-debug feedback protocol
EXEC@3 is measured under a Self-Debug stage configured identically for every model: each failing app is retried for up to three rounds with the same model, and each round forwards the toolchain’s own diagnostics back to the model for repair. EXEC@3 therefore reflects iterate-and-fix ability under this fixed feedback protocol rather than the model in isolation; how reliably a model localizes and repairs the root cause from standard build output is part of what it measures, with weaker self-debuggers returning NO_FILE.
G.4
Navigation and visual-fidelity failures
Among EXEC@3-passing apps, NRS is reduced by two recurring patterns: (i) mismatched navigation, where a <Link to="/foo"> resolves to no <Route> in the application’s router, and (ii) hallucinated routes, where the application registers and renders a route that corresponds to no input screenshot. Mismatched navigation dominates: models default to plausible web-app conventions such as sign-in, sign-up, forgot-password, or dashboard sub-routes when the input screenshot set underspecifies the route closure. Among EXEC@3-pass apps, the sub-skill view is best read off the conditional decomposition Table 7 (the zero-imputation columns of Table 2 are dominated by build-success differences, not sub-skill). Conditional spreads are tight: Color 68.9–80.8 (11.9), Position 73.9–83.3 (9.4), Size 70.6–75.6 (5.0), Text 87.0–91.7 (4.7). Position drops are dominated by reference-subset rendering rather than horizontal grid collapse: the generated DOM contains a strict subset of the reference’s blocks, so unpaired reference blocks dominate the centroid mismatch (see the GLM-4.6V case study in Fig. 13). Color drops by mis-derived theme palettes when screenshots use a non-default brand color. GLM-4.6V is the salient outlier: last on Size and Position but first on Color among build-pass apps. Palette extraction on the matched subset is strong, but the generated DOM covers only part of the reference.
G.5
Interaction-inference failures
Figure 6 anchors a representative IIS failure on a shared input. Two models receive the same five e-commerce screenshots of 48_shopco-ecommerce; an annotator then performs the same Add-to-Cart action on the product page and navigates to /cart. Claude Sonnet 4.6 (left) threads the action through a CartContext and the cart route renders the added product (result=W). Gemini 3.1 Pro Preview (right) emits a visually-faithful product page with an Add to Cart button whose handler does not propagate to any cross-route store, so the cart route still renders its empty-state placeholder (result=F). The cart route’s static screenshot at protocol time is therefore indistinguishable from the reference’s empty-cart capture.
21
(a) Claude Sonnet 4.6, /cart: product persists across the route boundary.
(b) Gemini 3.1 Pro Preview, /cart: empty-state placeholder; handler did not reach a cross-route store.
Figure 6: Interaction-inference failure on 48_shopco-ecommerce. Identical input screenshots and identical click sequence; the divergence is the Broken shared state pattern. VFS on the static cart-route capture cannot distinguish the two outputs (both render an empty cart at protocol time), whereas IIS flags the missing S3 affordance. G.6
Case studies: latent-affordance inference
The eight case studies below illustrate four claims from the main text: bimodal interaction emergence, domainlevel behavioral inference, build-time failure modes, and VFS sub-metric divergence; they are chosen to span these four claim types, and the aggregate evidence for each lives in the corresponding section and table (§4.3, Tables 9 and 7).
Audio synthesis (31_genshin-music). The 31_genshin-music task supplies 9 screenshots of a stylized in-browser music synthesizer (Fig. 7): a hexagonal note pad, a key-binding overlay, a composer timeline, and a full-screen “Zen mode” player. None of the screenshots state explicitly that the application produces sound. The audio capability must be inferred from layout cues alone: musical symbols on each pad, MIDI-style key labels (Q–U / A–J / Z–M), an instrument-picker dropdown, and a decay-envelope visualization behind each note. Of the six baselines, four reproduce the interface as a runnable static visual replica: pads animate on hover, but onMouseDown handlers are absent or no-ops and the emitted package.json declares no audio dependencies; GLM-4.6V fails to build. Only Kimi K2.5 instantiates a complete Web-Audio signal chain (AudioContext → triangle-wave OscillatorNode → GainNode with a 500 ms exponential-decay envelope) and wires keyboard events together with pointer events on every pad through a MIDI-to-frequency converter f = 440 · 2(n−69)/12 . The artifact is a functionally playable synthesizer rather than a screenshot. The contrast illustrates the regime IIS (§3.5) is designed to expose. The rendered interfaces are visually comparable and VFS cannot separate them, yet only one artifact is functionally playable. Crucially, no input screenshot can be flagged as “wrong” for the four UI-only models, since their renderings reproduce the visual targets’ structure: grid, labels, and layout. The gap is one of behavior inferred from cues that the visual modality under-specifies, precisely the regime UI2App is built to expose for multi-modal code-generation agents beyond surface mimicry. Figure 8 contrasts the source for one pad cell across the two regimes (the UI-only listing is from Gemini 3.1 Pro Preview, representative of the four visually-faithful-but-silent baselines).
/
/composer
22
/zen-keyboard
/vsrg-player
/vsrg-composer
/keybinds
/theme
/blog
/blog/posts/...
Figure 7: Input set for 31_genshin-music (9 screenshots, one per route). The audio capability must be inferred from these static cues alone: note pads with musical glyphs and MIDI key labels (Q–U/A–J/Z–M) on /zen-keyboard and /keybinds, a composer timeline (/composer, /vsrg-composer), and a full-screen player (/vsrg-player).
23
src/hooks/useAudio.ts src/components/Keyboard.tsx const playNote = useCallback( (midiNote: number) => { initAudio(); const ctx = audioContextRef.current; if (!ctx) return; stopNote(midiNote); const osc = ctx.createOscillator(); const gain = ctx.createGain(); const freq = 440 * Math.pow( 2, (midiNote - 69) / 12); osc.type = ’triangle’; osc.frequency.setValueAtTime( freq, ctx.currentTime); gain.gain.setValueAtTime( 0.3, ctx.currentTime); gain.gain.exponentialRampToValueAtTime( 0.01, ctx.currentTime + 0.5); osc.connect(gain); gain.connect(ctx.destination); osc.start(); oscillatorsRef.current.set(midiNote, osc); setTimeout(() => stopNote(midiNote), 500); }, [initAudio]);
function KeyButton({ label, ... }) { return ( <button className="w-[72px] h-[72px] rounded-full bg-[# ebdcb8] active:scale-95 ..."> <div className="rounded-full bg-[#f8f5ea] border-2 flex flex-col ..."> <svg width="24" height="24" ...> <circle cx="12" cy="12" r="9" /> <line x1="4" y1="12" x2="20" y2="12" /> </svg> <span className="..."> {label} </span> </div> </button> ); // no onClick / onMouseDown }
Listing 1: (b) Gemini 3.1 Pro Preview: Listing 2: (c) Kimi K2.5: src/components/Keyboard.tsx (ex- src/hooks/useAudio.ts (extract). tract). Pad is a styled <button> with no playNote builds the Web-Audio chain onClick/onMouseDown; the SVG and key label (OscillatorNode → GainNode, 500 ms are decorative. decay) with a MIDI-to-frequency converter. Figure 8: Latent-affordance inference on 31_genshin-music. Four of six baselines emit (b)-style code: visually-faithful pads with no audio. Only Kimi K2.5 instantiates the (c)-style Web-Audio signal chain, AudioContext → OscillatorNode → GainNode with a 500 ms exponential decay, and a MIDI-to-frequency converter f = 440 · 2(n−69)/12 . VFS cannot read this gap from the rendered DOM; the missing audio runtime is a latent affordance that surfaces only under behavioral inspection, the inference regime IIS targets (§3.5). State-machine inference (01_fashion-ecommerce, 48_shopco-ecommerce). Two e-commerce tasks, 01_fashion-ecommerce (6 screenshots: home, shop, product detail, cart, checkout, search) and 48_shopco-ecommerce (5 screenshots: home, shop, cart, and two product details), require an analogous inference along an orthogonal direction. Rather than an audio runtime, the model must infer the underlying state machine that backs the interface: a global cart that survives route changes (S3), variant selection on the product page that propagates into add-to-cart payloads (S2), search and sort on the shop (S2), and controlled inputs on a checkout form. As with audio, the screenshots never declare these affordances explicitly: a “$0.00” subtotal hints at a reactive total, a size grid at selectable variants, a search bar at client-side filtering. Across interaction items spanning the two projects, Claude Sonnet 4.6 produces lots of interaction components that work end-to-end: a CartContext de-duplicating items by the composite key (id, size, color), auto-clearing entries when quantity drops to zero, a controlled CheckoutPage form, and a search page with submit-driven filtering on product name and category. The next-best on these two tasks, GPT-5.4, supplies a partially wired ShopContext on 48_shopco-ecommerce, but its addToCart mutator is never read by the cart page, which renders a static empty-state placeholder, dead code in the literal sense. The remaining four baselines (Gemini 3.1 Pro Preview, GLM-4.6V, Qwen3.5-397B-A17B, Kimi K2.5) systematically emit visually faithful but inert components: add-to-cart buttons without onClick, cart pages reduced to “Intentionally left blank” placeholders, and no cross-component state container at all. Figure 9 contrasts the /cart render across the three regimes.
(a1 ) Sonnet, before click
(a2 ) Sonnet, after wired: item, $145, badge 1
24
(b1 ) GPT-5.4, before click
(b2 ) GPT-5.4, after dead code: cart empty, badge 1
(c1 ) Gemini, before click
(c2 ) Gemini, after inert: no change, no badge
Figure 9: State-machine inference on 48_shopco-ecommerce: before/after live capture across an identical click sequence. Each row is one model, before/after an identical add-to-cart → cartnavigation sequence (SPA <Link>, no hard reload, so in-memory state must survive). Left column (before): all three carts are empty baselines. Right column (after): (a2 ) Sonnet’s CartContext propagates end-to-end: G RADIENT G RAPHIC T- SHIRT (Size Large, $145) appears, subtotal recomputes, header badge reads 1; (b2 ) GPT-5.4’s badge also reads 1 (addToCart did mutate cartCount), yet the cart route still renders “Your shopping cart is empty.”: CartPage never reads cartItems from ShopContext, a literal dead-code branch; (c2 ) Gemini’s render is visually identical to its pre-click state (no badge change, no list item), no state container at all. All six panels (three models × before/after) are comparable in VFS, yet IIS on the S3 cross-route scope splits them sharply. Persistent task state (39_maciekt07-todoapp). The todo task supplies 5 screenshots of a single-page task manager with categories, multi-select, completion toggles, and JSON import/export. Of the six models that generated this app, only Kimi K2.5 persists task state, lifting it into a single AppContext that mirrors itself into localStorage on every mutation (STORAGE_KEY = "taskapp_data") and re-hydrates on mount. None of the other five survives a reload: Claude implements working in-memory CRUD in an app-level Context, while GPT-5.4, Gemini, and Qwen render the task list as static mock data with no mutable state; GLM-4.6V’s generated source holds tasks in component-local useState, but the app fails to build (EXEC@3=F). Figure 10 verifies the re-hydration half directly: three tasks seeded into localStorage survive a hard reload and re-appear on mount. This cross-session (localStorage-backed) persistence is strictly stronger than the S3 cross-route criterion (§B), which several models satisfy only in memory (e.g., Sonnet’s cart in Fig. 9).
(a) Kimi /purge, fresh ctx “no tasks”
25
(b) Kimi /purge, after seeding localStorage + reload “3 total, 0 selected, 1 completed”
Figure 10: Cross-session state persistence on 39_maciekt07-todoapp. (a) the /purge route on a fresh context shows “no tasks to purge” and disabled action buttons; (b) after seeding three tasks (one completed) into localStorage['taskapp_data'] and triggering a hard reload, the same route reports “3 total, 0 selected, 1 completed” and re-enables the P URGE D ONE/P URGE A LL TASKS buttons, proving that Kimi’s AppContext re-hydrates from localStorage on mount. None of the remaining five baselines survives a reload: Claude holds tasks in an app-level Context (in memory only) while GPT-5.4, Gemini, and Qwen render the list as static mock data—a S3 miss invisible to VFS; GLM-4.6V’s source uses component-local useState but the app fails to build. Drag-and-drop file upload (39_maciekt07-todoapp). The same task’s /transfer screen shows a dashed-border “Drop JSON file here to import tasks” import region. The cue is purely visual: a dashed rectangle with an upload icon and a brief instruction label, with no API specification. Kimi K2.5 and Claude Sonnet 4.6 both extract a FileDropZone component bound to native HTML5 onDragOver/onDrop events that pre-fills the JSON-import field on drop. Gemini 3.1 Pro Preview, GPT-5.4, and Qwen3.5-397B-A17B render the dashed rectangle as decoration plus a click-only fallback button with no drop handlers; GLM-4.6V’s build of this app fails (EXEC@3=F).
(a) input /transfer
26
(b) Kimi K2.5 onDrop wired
(c) Sonnet 4.6 onDrop wired
(d) Gemini 3.1 decoration only
Figure 11: HTML5 DnD primitive reach on 39_maciekt07-todoapp. Kimi/Sonnet bind onDragOver+onDrop to a real file-import flow; Gemini produces the same rectangle as decoration. Identical visual targets, divergent event topology. Pairs with the audio case (Fig. 8) to illustrate that frontier models repeatedly reach for domain primitives invisible at the pixel level. Rich-text editing (67_shadboard). 67_shadboard ships an editor screen whose only behavioral cue is a formatting toolbar (bold, italic, list, link) above an empty document area. Claude Sonnet 4.6 and Qwen3.5397B-A17B mark the document area with contentEditable, allowing genuine cursor-driven typing on the rendered page. Kimi K2.5 does the same and additionally binds the toolbar buttons to document.execCommand. GPT-5.4 is the sharper negative: it renders the full editor UI — a formatting toolbar above a document pane — that looks like a working editor, yet the pane is a static <div> (the toolbar icons carry no handler and there is no <textarea> or contentEditable element), so typing produces no change. The failure is indistinguishable from a real editor at idle and surfaces only under the live-typing test (Fig. 12). The remaining two models land between these poles: GLM-4.6V wires a plain <textarea> that accepts text but leaves its formatting toolbar inert, and Gemini 3.1 Pro Preview emits no /editor route at all.
(a1 ) Sonnet /editor, idle
(a2 ) Sonnet, after typing: text appears
27
(b1 ) Qwen3.5 /editor, idle
(b2 ) Qwen3.5, after typing: text appears
(c1 ) GPT-5.4 /editor, idle: looks like an editor
(c2 ) GPT-5.4, after typing: no change — static <div>, no editable element
28
Figure 12: Editor primitive on 67_shadboard: before/after typing into the document area. Top of each pair (idle): each model immediately after navigating to its editor route (Sonnet: 3 [contenteditable] regions detected; Qwen: 1; GPT-5.4: 0 — the document panes are static <div>s under a decorative toolbar). Bottom (after typing): the same test string is typed into the first document pane. (a2 , b2 ) Sonnet and Qwen show the typed text inside the Basic Editor card — the toolbar+document affordance wired into a real editing surface. (c1 , c2 ) GPT-5.4 renders a visually faithful editor whose panes are inert: the after-typing capture is identical to idle (no text appears). The live-typing test exposes the divergence IIS captures and pixel fidelity cannot. Reference-subset rendering (62_tokyo-mui-dashboard). The Tokyo crypto-dashboard organizes content into two vertical sections: a top section (Welcome header, ACCOUNT BALANCE card, donut chart with per-asset breakdown) and a continuation (WALLETS row of three asset cards plus the ACCOUNT S ECURITY side card). Gemini 3.1 Pro Preview reproduces both sections: the WALLETS header is visible at the bottom of the captured viewport, with the first row of asset-card frames already laid out. GLM-4.6V’s render is a strict subset of the reference: the top section is reproduced faithfully (the donut, the ACCOUNT BALANCE card with the same numeric value and Send/Receive controls, the per-asset legend), while the WALLETS row and ACCOUNT S ECURITY card have no counterpart in the generated DOM. We verify the subset relation rather than infer it from a viewport screenshot: a full-page Playwright capture across seven candidate routes (/dashboard, /crypto, /cryptocurrency, /dashboards/crypto, /dashboard-crypto, /overview, /) returns document.documentElement.scrollHeight=900 px (exactly the viewport height, so no overflow exists where missing blocks could hide), and document.body.innerText on every probed route contains none of the WALLETS/S ECURITY section keywords. The Wallets/Security blocks are absent from the rendered DOM, not pushed past the viewport. This subset relation is exactly what the DOM-block matching procedure of VFS penalizes through the coverage factor nmatched /ntot in Eq. (3): each reference block with no counterpart in the generated DOM is a missed match, and the unmatched-block count dominates Position/Size for this app. The case is the per-app face of GLM’s signature VFS⋆ profile (Table 7): the matched subset (top row) keeps palettes close to the input, explaining GLM’s highest aggregate Color, while the unmatched complement (the below-fold sections) keeps GLM’s Position/Size the lowest on the build-pass subset.
(a) input /crypto: top row + WALLETS header + ACCOUNT S ECURITY card visible
(b) Gemini 3.1 below-fold preserved: top row + WALLETS row
(c) GLM-4.6V strict reference subset: top section only; Wallets/Security blocks have no DOM counterpart (scrollHeight=900, probed keywords absent)
29
Figure 13: Reference-subset rendering on 62_tokyo-mui-dashboard/crypto. Pictorial counterpart to §E. GLM-4.6V’s rendered DOM is a strict subset of the reference dashboard’s DOM: the top section is reproduced faithfully, while the reference’s subsequent WALLETS+S ECURITY blocks have no counterpart in the generated DOM. We verify the subset relation, not just a viewport screenshot: a full-page Playwright capture across seven candidate routes returns scrollHeight = 900 px (no overflow) and innerText contains none of the WALLETS/S ECURITY section keywords. Gemini preserves both sections. Per VFS’s coverage factor nmatched /ntot in Eq. (3), every unmatched reference block contributes a missed match, so the matched subset (top row) determines GLM’s Color (palette preserved ⇒ highest aggregate) while the unmatched complement (below-fold sections) determines Position/Size (centroid mismatch dominated by unpaired reference blocks ⇒ lowest aggregate). Build-time failures (55_shadcn-nextjs-dashboard, 60_mantine-dashboard). The two dominant unrecovered EXEC@3 modes (Table 9) have orthogonal etiologies. C1 (hallucinated icon) is a factuality error: GPT-5.4 imports Funnel from lucide-react, but in the scaffold-pinned 0.460.0 the icon is still exported as Filter (Funnel is the post-0.460 canonical name); vite build aborts with “"Funnel" is not exported by .../lucide-react.js”. C2 (package.json overwrite) is an instruction-compliance error: GLM-4.6V regenerates the scaffold’s package.json despite the explicit “Do NOT include scaffold files” clause (Appendix H, clause iv), pinning the unpublished version "mantine":"ˆ1.0.0"; pnpm install aborts with ERR_PNPM_NO_VERSIONS. package.json (regenerated) AnalyticsPage.tsx
{ "dependencies": { "mantine": "^1.0.0"
import { Download, CalendarDays, Funnel // <-- not exported } from "lucide-react";
// pkg name is // @mantine/core
} }
Listing 3: (a) C1, GPT-5.4 on 55_shadcn-nextjs-dashboard: hallucinated lucide-react export. Factuality error.
Listing 4: (b) C2, GLM-4.6V on 60_mantine-dashboard: regenerated package.json (clause-iv violation). Instruction-compliance error.
Figure 14: Factuality vs instruction-compliance failures. C1 (left) is a symbol-existence hallucination that the prompt does not pre-empt. C2 (right) violates clause (iv) of the plan-step prompt. The orthogonality of the two error modes motivates reporting an instruction-compliance rate alongside EXEC@3 (§G.1). Functional quiz engine (61_react-duolingo). The gamified learning app’s 10-screenshot input set (Fig. 15) spans a lesson map, a lesson exercise, a leaderboard, a shop, a profile, and settings. The lesson exercise — a prompt, three answer tiles, and a C HECK button — has behavior the screenshots never state: selecting a tile, judging it against the correct answer, and awarding XP on success must all be inferred. Across the six baselines the rendered exercise is near-identical, yet only Claude Sonnet 4.6 wires a working quiz engine — handleCheck compares the selection to question.correctId, sets correct/incorrect feedback, and on a correct answer calls incrementLessons (advancing lesson progress and adding 10 XP). Kimi K2.5 only partially wires it (its C HECK handler navigates without validating or scoring), and the remaining four (Gemini 3.1 Pro Preview, GPT-5.4, Qwen3.5-397B-A17B, GLM-4.6V) render a static mockup whose C HECK button carries no handler. The 1-functional / 1-partial / 4-static split is invisible to VFS — every render shows the same tiles and button — and is the answer-validation analogue of the audio case (Fig. 8).
30
/
/register
/learn
/lesson
/leaderboard
/shop
/profile
/settings/account
/settings/coach
/settings/sound
31
Figure 15: Full input set for 61_react-duolingo (10 screenshots, one per route) given to every model. The lesson exercise (/lesson) implies an answer-validation + XP-scoring engine; whether each model actually builds it is shown in Fig. 16.
(a) Claude lesson — before
(b) Claude — after: Correct! + C ONTINUE
(d) Claude /learn — before: Daily Quest 0/10 XP
(c) Gemini — after: inert, no feedback
(e) Claude /learn — after: Daily Quest 10/10 XP
LessonPage.tsx const handleCheck = () => { if (!selected) return; const ok = selected === question.correctId; setCorrect(ok); setChecked(true); if (ok) incrementLessons(); // +1 lesson, +10 XP };
Listing 5: Claude — LessonPage.tsx (extract): the answer-validation + XP engine the other five omit. Figure 16: Functional quiz engine on 61_react-duolingo (before/after the same selectanswer→C HECK sequence). (a,b) Claude validates the answer and shows Correct! feedback, advancing to C ONTINUE; (c) Gemini’s C HECK has no handler, so the same clicks leave the question unchanged. (d,e) the correct answer also awards XP, filling Claude’s daily quest 0/10 → 10/10. Only Claude wires the full engine (Kimi partial; the other four inert) — an answer-validation + scoring affordance VFS cannot see, since every render is near-identical. Synthesis across cases. The eight cases together draw three lines that no single metric exposes. First, latent-affordance inference is neither monotonic in model scale nor in visual fidelity: Kimi K2.5 is the only baseline that recognizes the audio runtime in 31_genshin-music (Fig. 8) and the only baseline that lifts task state into localStorage on 39_maciekt07-todoapp (Fig. 10), yet it ships inert e-commerce UIs (Fig. 9). Claude Sonnet 4.6 dominates state-machine inference on both shops and reaches for contentEditable on the editor task (Fig. 12) and alone wires the answer-validation + XP engine behind 61_react-duolingo’s lesson exercise (Fig. 16), yet ships a silent music application. Second, the failure surface is itself bimodal (Fig. 14, quantified in Table 9): GPT-5.4’s unrecovered EXEC@3 losses are dominated by factuality errors (hallucinated icons), whereas GLM-4.6V’s are dominated by instruction-compliance errors (scaffold overwrites), two orthogonal error modes that EXEC@3 alone conflates. Third, the visual-fidelity dimension is internally split (Fig. 13; aggregate decomposition in Table 7): GLM-4.6V’s Color sub-metric is the highest of any model on the build-pass subset while its Position sub-metric is the lowest, an inversion that motivates layout-targeted training rather than further color supervision. Across all eight cases, the disagreement among baselines on which
32
behavioral type or structural primitive is recognized is precisely the regime that pixel-fidelity scores cannot expose and that IIS, NRS, and the failure taxonomy are jointly constructed to surface.
H
Prompt templates
Below are the prompt templates used in the pipeline. Placeholders rendered as {name} are populated at run time; line breaks and angle-bracket tags are exactly what the model receives.
Stage 1 (Plan). The Plan prompt is issued in the second user turn after the assistant has emitted the scaffold listing in Turn 1. The same turn carries all M screenshots as image content. You are an expert frontend developer. You are given {M} screenshots of a web application. Your task is to reproduce the UI as a complete, runnable Vite + React + TypeScript + Tailwind CSS project. The project scaffold has already been set up (you saw it above). src/App.tsx is the root component mounted by main. All source files you generate must be under src/. tsconfig uses "jsx": "react-jsx" so explicit React imports are unnecessary. react-router-dom and lucide-react are available in dependencies. <code_guidelines> - Use coding best practices. - Use Tailwind CSS utility classes for styling. Match the screenshots as closely as possible. - Use relative imports. Do not add file extensions in imports for .ts/.tsx files. - Files containing JSX must use .tsx extension. </code_guidelines> Analyze the {M} screenshots carefully. Then output a JSON file plan –- a list of all source files you will create, with a one-line description of each file’s purpose. Output ONLY valid JSON (no markdown fences, no explanation): {"extra_dependencies": {}, "plan": [{"path": "Root component with routing setup"}, ...]}
"src/App.tsx", "description":
Rules: - Include src/App.tsx and all component/page files. All paths start with "src/". - "extra_dependencies" lists any npm packages beyond the scaffold. - Do NOT include scaffold files. Do NOT generate any code yet –- only the plan. The four constraints we treat as explicit instruction clauses for the failure-mode attribution in §4.4 are: (i) all source files under src/; (ii) relative imports; (iii) every non-scaffold package declared in extra_dependencies; (iv) no regeneration of scaffold files.
Stage 2 (Generate). The Generate prompt is issued in the next user turn after the model has returned the plan. Screenshots are not re-sent. Now generate the COMPLETE source code for ALL files in the plan below. <project_plan> {plan_summary} </project_plan> Output each file using this exact format (no markdown fences, no explanation):
33
–- src/App.tsx –[complete source code for App.tsx] –- src/components/Navbar.tsx –[complete source code for Navbar.tsx] ...and so on for every file in the plan. CRITICAL RULES: - Write COMPLETE file content for every file. Never truncate, abbreviate, or use comments like "// rest of code". - Files containing JSX/TSX syntax (<div>, <Component/>) MUST use .tsx extension. - Use 2 spaces for indentation. - Every file must start with "–- filepath –-" on its own line. - Do NOT output anything other than the file blocks. If the model’s response is truncated (finish_reason ̸= stop), we send up to three Continue turns: “Continue writing from exactly where you left off. Do not repeat any content. Continue with the next file or finish the current file.”
Stage 3 (Self-Debug). On EXEC failure the same model is re-invoked for up to three rounds. Each round runs three sub-prompts as needed.
(a) File-localization fallback. When the regex extractor cannot identify which source files to repair from the build error, the model is asked to localize: The following build error occurred.
Which source file(s) need to be fixed?
<error> {errors} </error> <project_files> {file_list} </project_files> Reply with ONLY the file path(s) that need fixing, one per line. explanation.
No
(b) Build-error repair. For each identified file the model receives the error, the project file listing, and the file’s current contents: Fix the following build error in {filepath}: <error> {errors} </error> <project_files> {file_listing} </project_files> <current_code file="{filepath}"> {current_code} </current_code> Output ONLY the fixed complete file content –- no explanation, no markdown fences.
(c) Runtime-error repair. When the build passes but the home route fails to render, the model receives all candidate files (App.tsx, main.tsx, plus any flagged by error parsing) together for a holistic fix:
34
The React app builds successfully but renders a blank page at runtime. <error> {errors} </error> {file_contents} For EACH file that needs changes, output: –- filepath –<complete fixed file content> Output ONLY the fixed files –- no explanation, no markdown fences.
35