LiveWorkspace: Externalizing Visual Understanding as Persistent Shared State for Screenshot-to-Code Agents

LiveWorkspace: Externalizing Visual Understanding as Persistent Shared State for Screenshot-to-Code Agents Shijun Xiao1,2 , Zonghui Yang3 , Xinrong Chen3 , Hao Liu2 , Wensheng Zhang4 , Yongqiang Tang2,∗ 1

Nankai University, Tianjin, China

Institute of Automation, Chinese Academy of Sciences, Beijing, China 3 4

Guangzhou University, Guangzhou, China

Corresponding author: yongqiang.tang@ia.ac.cn

Coding Agent

Screenshot-to-Code requires coding agents to sustain visual understanding across code generation, execution, and revision. We introduce LiveWorkspace, a visual coding harness that externalizes this understanding as Shared Visual State (Shared VS), an implementationindependent representation of visual hierarchy, node attributes, and layout relations. LiveWorkspace constructs and retains Target from screenshots and recovers Current from browser execution in the same schema. Structured Diff and implementation bindings guide code modifications; user edits update Target and reuse the alignment loop. Using GPT-5.2 for both methods, LiveWorkspace improves end-to-end Avg by 1.8 points on Design2Code and 2.0 on ScreenBench over a control with matched model-call counts and stagewise budget limits. Further experiments demonstrate reuse of constructed Shared VS across coders and improvements in iterative repair. LiveWorkspace also achieves 93.0% edit success across 100 controlled editing cases. These findings support treating visual understanding as a persistent shared working state throughout code generation, execution, and modification.

1

Coding Agent

Coding Agent

1

Generate

Round 2~n

Round 2~n

Code

Target-side Structure

Image-level Feedback

Code 4 Diff

Shared Visual State

Recover

arXiv:2611.12112v1 [cs.SE] 9 Nov 2026

Abstract

Render

∗

Peking University, Beijing, China

Compare

2

3

Target Regions

Page Decomposition Round 1

Target Screenshot

Current Screenshot

Coding Agent

Render

2

Code

Round 1

Target Screenshot

(a) Structure-guided (b) Screenshot-driven

Page

Page

Target

Current

Visual State Constructor Round 1

Target Screenshot

(c) LiveWorkspace (Ours)

Figure 1: Visual interfaces for Screenshot-to-Code. (a) Target-side structure guides generation; (b) rendered screenshots provide image-level feedback; (c) LiveWorkspace maintains Target and Current in Shared VS and exposes their differences for code revision.

deviations become apparent only after the code is rendered, requiring the coding agent to verify and revise the implementation. The central challenge of Screenshot-to-Code is therefore to sustain visual understanding that guides code generation and modification after execution. Existing methods improve this process along two main directions. One line of work uses explicit page decomposition and structural modeling to provide intermediate representations of regions, layouts, or hierarchies for code generation, including DCGen, LaTCoder, LayoutCoder, and ScreenCoder (Wan et al., 2025; Gui et al., 2025b; Wu et al., 2025a; Jiang et al., 2025). Another brings browser execution results back into generation

Introduction

As coding agents evolve from code completion and program generation toward understanding, building, and modifying real software interfaces (Yang et al., 2024; Wang et al., 2025, 2024), Screenshotto-Code is becoming a fundamental capability for translating visual intent into executable software. Given a target webpage screenshot, a model must generate HTML/CSS/JavaScript whose browser rendering faithfully reproduces the target design. The model must identify visual elements, recover their structure and attributes, and translate these visual constraints into executable code. Code generation is also not the end of this process: many visual 1

generation Avg scores of 92.7 and 83.0 on Design2Code and ScreenBench. Controlled repair experiments show that Structured Diff improves the repair rate, with further gains from source binding; LiveWorkspace also achieves higher final fidelity and introduces fewer new defects than screenshot feedback. Across 100 controlled editing cases, LiveWorkspace achieves 93.0% edit success. Our main contributions are as follows:

and performs multiple rounds of refinement based on differences between the target screenshot and the current rendering, including UI2CodeN , VisRefiner, and other rendering-in-the-loop methods (Yang et al., 2025b; Deng et al., 2026). The former improves understanding of the target page structure; the latter enables continued correction of visual errors through execution feedback after initial generation. However, target-side structural representations and execution-side visual feedback still lack a common, persistent state interface. Structural representations primarily guide code generation, without normalizing execution results into the same representation. Screenshot-based feedback still relies on the model to infer visual correspondences between the target and current implementation from images. Based on this observation, we formulate Screenshot-to-Code as a continuous process of visual alignment and introduce Shared Visual State (Shared VS) to externalize page understanding into a working state independent of the specific HTML/CSS implementation. Shared VS describes the final visible structure and appearance of a page through visual hierarchy H, node attributes A, and layout relations L, without requiring recovery of the original DOM or CSS. The same schema supports recovery of Target Shared VS from the target screenshot and Current Shared VS from code execution, enabling explicit comparison between the target design and current implementation. Building on this representation, we introduce LiveWorkspace, a visual coding harness that uses Shared VS as its core working state and combines the coding agent’s structural reasoning with deterministic visual measurement. LiveWorkspace retains Target Shared VS across iterations and recovers Current Shared VS after each execution, using their differences to guide subsequent modifications; user edits update Target and reuse the same alignment process. Figure 1 illustrates the distinction between target-side structure, image-level feedback, and persistent shared visual state. We evaluate the complete LiveWorkspace harness on Design2Code and ScreenBench. Compared with a GPT-5.2 control with matched model-call counts and stagewise budget limits, LiveWorkspace improves final reconstruction Avg from 91.8 to 93.6 on Design2Code and from 81.2 to 83.2 on ScreenBench, gains of 1.8 and 2.0 points, respectively. With only Recovered Target Shared VS as visual input, DeepSeek-V4 achieves initial-

• We introduce Shared VS as an implementation-independent interface that represents target designs and executed implementations in a common visual schema for explicit comparison and alignment. • We build a visual coding harness around Shared VS, separating agent-based structural reasoning from deterministic visual measurement. It retains Target, recovers Current after execution, and unifies generation, repair, and user editing through the same state interface. • Our experiments show that Shared VS supports reuse across coding models, Structured Diff turns state discrepancies into actionable repair feedback, and source binding provides implementation-location cues that further improve repair.

2

Related Work

2.1

Structured Visual Understanding for Screenshot-to-Code

Screenshot-to-Code has progressed from early image-to-code generation to multimodal webpage generation and evaluation (Beltramelli, 2018; Laurençon et al., 2024; Si et al., 2025). Larger training datasets (Gui et al., 2025a; Yun et al., 2024), broader benchmarks (Guo et al., 2025; Xiao et al., 2025), data synthesis (Ge et al., 2025), and visualreward training (Zhao et al., 2025) support this development. UI understanding integrates visual, textual, and structural information (Deka et al., 2017; Bai et al., 2021; Baechler et al., 2024; Li et al., 2025b), while screen parsing recovers elements and relations (Xie et al., 2020; Wu et al., 2021; Lu et al., 2024; Gurbuz et al., 2026). DCGen decomposes screenshots into regions, and UICopilot separates coarse hierarchy generation from finegrained code synthesis (Wan et al., 2025; Gui et al., 2025c). LaTCoder uses Layout-as-Thought for block generation and assembly, LayoutCoder constructs element relations and layout trees (Gui et al., 2

2025b; Wu et al., 2025a), and ScreenCoder separates grounding, planning, and generation (Jiang et al., 2025). LiveWorkspace extends this structural perspective with a common, implementationindependent schema for target designs and executed implementations, enabling explicit visualstate comparison.

2.2

The overall process is S t = Construct(I t ), Skc = Recover(Ck , Render(Ck )), ∆k = Diff(S t , Skc ), Ck+1 = Agent(S t , Skc , ∆k , Bk , Ck ).

(2) (3) (4)

Initial generation uses only Target Shared VS, which is retained across refinement and updated for user edits. After each execution, the harness recovers Current Shared VS in the same schema, recomputes the structured difference, and refreshes the implementation binding.

Visual Feedback and Iterative UI Refinement

Self-Refine establishes a general feedback-andrefinement loop (Madaan et al., 2023), while Sketch2Code studies interactive webpage prototyping with user feedback (Li et al., 2025a). For Screenshot-to-Code, UI2CodeN uses rendered feedback for iterative visual optimization (Yang et al., 2025b), and VisRefiner learns from visual differences and corresponding code edits (Deng et al., 2026). ComUICoder uses element-level matching to produce targeted instructions for adding or removing elements and correcting visual attributes (Xiao et al., 2026a). Visual software repair benchmarks screenshot-bearing issues (Yang et al., 2025a) and explores cross-modal reasoning, region-focused perception, and rendered feedback (Huang et al., 2025; Xiao et al., 2026b; Liu et al., 2026). Reflexion and FailureMem retain feedback or repair experience across attempts (Shinn et al., 2023; Ma et al., 2026). LiveWorkspace retains the desired page state as Target and recovers Current after each execution. Their shared schema makes visual discrepancies explicit, while implementation bindings connect them to code locations. User edits update Target and reuse this alignment loop.

3

(1)

3.1

Shared Visual State

Shared VS describes the visible state of a page rather than the DOM or CSS that produces it. Since different implementations can yield identical or similar renderings (Lai et al., 2025), we canonicalize a page as S = (H, A, L),

(5)

where H denotes visual hierarchy, A denotes node attributes, and L denotes layout relations between nodes. Visual hierarchy H. H describes visual grouping and containment, with internal nodes for meaningful regions or groups and leaves for visible elements such as text and images. Synthetic groups can express visual groupings without corresponding source-code elements. Node attributes A. A describes observable visual facts about each node, including position and size, text, foreground color, fill, and border. Layout relations L. L describes how elements within the same visual group are arranged, including sequence, grid, and overlay, as well as recoverable direction, spacing, and padding. H describes which elements belong to the same group, while L describes how elements within that group are arranged. During execution, LiveWorkspace maintains two instances of the same Shared VS schema: Target Shared VS describes how the page should appear, and Current Shared VS describes the visual state actually produced by the current code. DOM nodes, selectors, and source locations are stored in an independent binding and are excluded from Shared VS itself.

Method

As shown in Figure 2, LiveWorkspace is a Screenshot-to-Code harness whose core working state is Shared Visual State (Shared VS). Given a target screenshot I t and code Ck at iteration k, the system maintains the Target state S t , Current state Skc , structured difference ∆k , and an independent implementation binding Bk . The same coding agent performs page-structure reasoning and code decisions, while the harness handles visual observation, state recovery, comparison, and execution feedback. This agent–harness interaction follows the reasoning–action paradigm (Yao et al., 2023). 3

Target construction

Shared Visual State

Target screenshot Card A

Card B

Details

Details

Persistent target

Cards

Visual hierarchy

Card A

Text

Agent updates Target

Target Shared VS Card B

Visual observations Region

Generation and execution

User instruction Move the cards closer

T Card B

Paint

row

Layout relations

T Details

gap 16 px

Harness

Visual observations

Cards Card A

Card B

Visual hierarchy

+

Visual hierarchy

Text

Card A

Bind observations to hierarchy

Current

Target

Cards / gap Card B / fill

28 px

16 px

.card-a .card-b

Card B Look up

Read / update

Recover

Card B

row

Code reference

Card A Card B

<div class="card-b"> … </div> .cards { gap: 28px; } .card-b { background: #D9DDE3; }

Recovered after each execution

T Card B

Layout relations

Visual node

Current code

Node / property

Cards

Paint

Coding Agent

Implementation bindings

Structured Diff

Current

Current Shared VS

Region

States + Diff

Card B

edited

Alignment Target

Coding Agent

Node attributes text "Card B" size 180 × 112 px fill

Initial generation

T Details

Node attributes text "Card B" size 180 × 112 px fill

gap 28 px

Current-state recovery

Card B

Cards Recover

Card A

Card B

Details

Details 28 px

112 px 180 px

Figure 2: Overview of LiveWorkspace. The agent uses the screenshot and harness-provided observations to infer visual hierarchy. The harness binds observations to this hierarchy and recovers attributes and layout, forming Target Shared VS S = (H, A, L). Initial generation uses only Target. Current is recovered from browser execution in the same schema. Structured Diff and implementation bindings guide code repair. Target persists across iterations; user edits update it and reuse the alignment loop.

3.2

Target Shared VS Construction

Current and Target use the same canonical visual schema but rely on different recovery evidence. Target uses the screenshot and visual observations, whereas Current uses the runtime DOM, computed styles, and actual painted geometry. Through Visual-Tree Canonicalization, the harness collapses wrappers without visual meaning, prunes invisible elements, and infers layout relations from painted geometry. The DOM serves only as recovery evidence; implementation identifiers such as tags, classes, and selectors are excluded from Skc . Since Target and Current are recovered independently and do not share node IDs, the harness first establishes node correspondences. Nodes with visible text are matched exactly using normalized text; multiple candidates with identical text are disambiguated by geometric center distance. Remaining nodes are matched greedily, one-to-one, using the IoU of absolute page bounding boxes (IoU ≥ 0.5), prioritizing pairs with the largest overlap. Text matching does not require box overlap, so displacement is reported as an attribute difference for textmatched nodes. The harness compares visual attributes and within-group layout for corresponding nodes, and reports unmatched reportable visual nodes as structural differences.

Target Shared VS construction separates agentbased structural reasoning from deterministic visual measurement. For the target screenshot I t , the harness first extracts three types of low-level visual observations: O = Oregion ∪ Otext ∪ Opaint ,

(6)

which provide region candidates, text and its geometry, and paint evidence such as color, fill, and border, respectively. The coding agent uses the target screenshot and observations O to infer visual hierarchy H t , identifying visual units and their grouping and containment relations. Given H t , the harness associates observations with canonical nodes: node attributes are recovered from the associated text, geometry, and paint evidence, while layout relations are inferred from the painted geometry of children within each group. The agent thus supplies page organization, and the harness fills the representation with visual facts supported by the observations, yielding the complete state S t = (H t , At , Lt ): S t = Recover(I t , O, H t ). 3.3

(7)

Current Shared VS and Alignment

After code execution, LiveWorkspace recovers Current Shared VS from the current rendering: Skc = (Hkc , Ack , Lck ).

∆(v) = {∆geometry , ∆text , ∆color , ∆layout , . . .}.

(9)

Attribute differences record current and target values; numerical attributes use tolerances to filter

(8) 4

minor measurement deviations. Alongside Current Shared VS, the harness maintains an implementation binding separate from Shared VS:

image features (Radford et al., 2021). We also use Design2Code for representation and repair analyses. User editing is evaluated separately on 100 synthetic controlled cases. ScreenBench is evaluated after freezing the visual tools, recovery rules, and thresholds to assess cross-benchmark generalization. Baselines. Baselines include direct multimodal models such as Qwen2.5-VL (Bai et al., 2025), training-free UI-to-Code methods (Wan et al., 2025; Gui et al., 2025b; Wu et al., 2025a; Jiang et al., 2025), and models trained for UI-to-Code (Yang et al., 2025b; Laurençon et al., 2024; Jiang et al., 2025). Existing methods use their public implementations and recommended configurations. Controlled setting. The end-to-end LiveWorkspace evaluation uses GPT-5.2 as its sole coding agent. GPT-5.2 Direct is a single-call reference. All methods use the same browser environment, and invalid or unrenderable outputs remain in the evaluation. End-to-end protocol. LiveWorkspace constructs Recovered Target Shared VS, while GPT5.2 (Matched Budget) uses free-text visual analysis; both then generate an initial implementation. Both use two initialization calls and up to three repair rounds, with Shared VS feedback for LiveWorkspace and screenshot feedback for the control. We match actual model-call counts and stagewise budget limits, including browser-execution budgets. Both allow early termination and are evaluated on their last generated implementation. Appendix B reports initial-generation model calls, token usage, and visual-tool latency.

Bk : v c ↔ {DOM node, selector, source span}. (10)

For generated single-file HTML, the harness establishes implementation bindings using element types, text containment, and document order. Structured Diff identifies visual attributes to change and their current and target values; binding provides implementation-location cues for the corresponding Current nodes. The agent combines this information with the current code to decide how to modify the implementation. 3.4

Generation, Refinement, and Editing

Initial code generation uses Target Shared VS as visual input and produces implementation C0 . After execution, the harness recovers Current and computes Diff and implementation bindings; the coding agent uses them to choose a local patch or a broader modification. The modified code is executed again to produce the next Current, while Target remains unchanged throughout refinement. Given a user instruction u, the agent identifies the relevant visual nodes and updates the affected hierarchy, attributes, or layout relations in Target Shared VS, yielding S t′ : S t′ = Edit(S t , u).

(11)

The updated state is saved and becomes the new alignment target. The harness compares it with Current, and the coding agent modifies the code using the new differences and implementation bindings, reusing the execution and feedback loop of refinement.

4

4.2

Table 1 reports final reconstruction performance of the complete LiveWorkspace harness under the end-to-end protocol in §4.1. With matched model-call counts and budget limits, LiveWorkspace improves Avg from 91.8 to 93.6 on Design2Code and from 81.2 to 83.2 on ScreenBench, gains of 1.8 and 2.0 points. These gains support the shared-state approach under matched call counts and budget limits. On ScreenBench, the largest gains over the matched control are in Color (85.2 to 90.6) and Block (67.7 to 70.7). LiveWorkspace also achieves competitive reconstruction scores relative to the direct multimodal models and existing UI-to-Code systems in Table 1.

Experiments

We evaluate LiveWorkspace from four perspectives: end-to-end Screenshot-to-Code performance (§4.2), the representational capabilities of Shared VS (§4.3), the role of visual state in iterative repair (§4.4), and controlled user editing (§4.5). 4.1

Overall End-to-End Performance

Experimental Setup

Benchmarks and metrics. We evaluate on the complete test sets of Design2Code and ScreenBench (Si et al., 2025; Jiang et al., 2025). We report Block, Text, Position, Color, CLIP, and their mean, Avg (Si et al., 2025), using pretrained CLIP 5

Model

Design2Code ScreenBench Block Text Position Color CLIP Avg Block Text Position Color CLIP Avg Direct Multimodal Models

GPT-4o GPT-4V GPT-5.2 Gemini-2.5-Pro Claude Sonnet 4.6 Claude Opus 4.6 Claude Opus 4.8 Qwen2.5-VL-72B

84.3 83.5 86.8 88.8 94.5 95.1 94.1 86.6

96.3 95.9 96.1 93.1 96.1 97.3 98.7 91.6

89.9 89.1 91.2 85.6 92.2 92.5 92.8 76.8

88.0 86.8 91.4 71.2 84.4 86.1 88.1 67.8

92.2 90.8 90.5 81.0 91.5 92.2 92.4 77.8

90.1 89.2 91.2 83.9 91.7 92.6 93.2 80.1

75.0 71.6 67.1 71.2 76.7 77.2 76.9 77.5

84.0 81.3 92.1 91.1 93.0 90.7 93.7 78.9

72.1 69.7 73.6 73.6 74.2 75.3 73.9 61.4

70.1 68.6 84.1 82.4 81.5 82.4 84.1 51.8

77.2 75.5 86.7 88.1 86.6 90.1 88.4 63.9

75.7 73.3 80.7 81.3 82.4 83.1 83.4 66.7

73.5 73.1 73.7 77.2 67.7

82.6 83.3 83.8 85.8 92.3

70.8 71.3 71.4 75.0 74.1

69.5 70.4 70.4 73.7 85.2

76.2 76.7 76.6 81.1 86.5

74.5 75.0 75.2 78.6 81.2

74.6 67.9 76.2

78.9 76.9 84.3

58.1 55.2 73.7

58.1 60.9 71.3

76.0 69.1 75.1 67.2 78.8 76.9

70.7

93.7

74.5

90.6

86.6 83.2

Training-free Baselines DCGen LaTCoder LayoutCoder ScreenCoder (Agentic) GPT-5.2 (Matched Budget)

83.7 83.4 83.6 86.9 87.7

95.7 95.5 96.8 97.7 96.3

88.9 89.4 88.9 92.2 91.8

86.0 86.4 88.0 90.4 92.4

89.9 90.7 91.1 92.6 90.8

88.8 89.1 89.7 92.0 91.8

Post-Trained Baselines N

UI2Code Websight-8B ScreenCoder (Finetuned)

85.0 75.6 84.7

90.5 90.6 96.7

75.8 77.2 91.5

76.5 78.0 88.8

LiveWorkspace

89.0

96.8

93.5

96.2

87.3 83.0 86.1 81.5 91.1 90.6

Ours 92.3 93.6

Table 1: End-to-end Screenshot-to-Code performance on Design2Code and ScreenBench.

Two-call generation control. On a 50-page Design2Code subset, we compare free-text analysis with Recovered Target Shared VS for initial generation. Both use two GPT-5.2 calls with matched stagewise token limits and no refinement. Shared VS achieves 94.53 Avg, compared with 91.13 for free-text analysis, a gain of 3.40 points. Appendix D details this control.

Block

DOM Recovered VS

Text

Reference VS

Position Color CLIP Avg 75

4.3

Shared VS: Sufficiency, Recoverability, and Transferability

85

95

100

Figure 3: Screenshot-free initial generation on Design2Code from different visual representations using GPT-5.2.

We compare two construction sources for Target: Recovered Target Shared VS is constructed from the target screenshot, whereas Reference Target Shared VS is derived from the reference implementation. Representation sufficiency. We fix GPT-5.2 as the coder and compare initial generation from different representations without giving the coder the target screenshot. Reference Target Shared VS is an oracle representation deterministically derived from the reference rendered page into the same canonical schema; the coder receives only the resulting state. In Figure 3, screenshotinferred DOM, Recovered Shared VS, and Reference Shared VS achieve 87.3, 92.3, and 97.1 Avg, respectively. Recovered Shared VS improves over DOM by 5.0 points, supporting its ability to retain visual information for code generation.

Recoverability. Table 2 shows strong text and color recovery, while hierarchy attachment, size estimation, and layout-primitive recognition remain the main bottlenecks. Visual grouping and layout direction are recovered more consistently. Agreement with a reference decomposition does not fully measure the completeness of recovered visual constraints: the same arrangement may be described as a grid or as a vertical sequence of horizontal sequences. Figure 3 therefore provides complementary downstream evidence. Its 4.8-point Recovered–Reference gap captures the aggregate effect of state recovery on reconstruction, rather than layout-primitive mismatch alone. Cross-model transferability. We construct Re6

Component Hierarchy Geometry Layout Text Color

Metric

Result

Co-group F1 ↑ Hierarchy Acc. ↑ Size within tol. ↑ Primitive F1 ↑ Direction Acc. ↑ Coverage ↑ Exact Match ↑ Coverage ↑ Exact Match ↑

0.79 50.6 41.2 0.44 89.8 96.8 93.3 98.2 86.7

solved, while Final Fidelity uses the independent Design2Code evaluator described in Appendix F. Adding Structured Diff to Target and Current feedback raises Final Fidelity from 90.0 to 91.9 and Repair Rate from 11.8% to 46.3%, turning state differences into explicit repair targets. Implementation binding. Adding source binding raises Final Fidelity from 91.9 to 93.1 and Repair Rate from 46.3% to 53.8%. Diff identifies visual discrepancies, while binding provides implementation-location cues. Collateral rises from 4.9 to 5.1 newly defective nodes per page, but remains below Screenshot Feedback (14.6) and Raw Observations (13.7).

Table 2: Screenshot-to-Target Shared VS recovery by component. F1 scores are on a 0–1 scale; other results are percentages.

covered Target Shared VS using GPT-5.2 and the harness, then reuse the state across coders for initial generation without refinement. As shown in Table 3, Shared VS improves DeepSeek-V4’s Avg from 89.6 to 92.7 on Design2Code and from 79.7 to 83.0 on ScreenBench compared with Raw Visual Observations. These results support reuse of the constructed state across coding models. Coder

Input

Condition

✓ ✗ ✗ ✗

91.2 92.3 89.6 92.7

Initial C0 Screenshot Raw O Static VS ✓ Target + Current ✓ ✓ + Diff ✓✓✓ LiveWorkspace ✓ ✓ ✓ ✓

80.7 83.1 79.7 83.0

Component ablation. Table 4 removes hierarchy, attributes, or layout from Reference Target Shared VS. Removing attributes produces the largest drop, from 97.1 to 62.5 Avg; removing hierarchy or layout reduces Avg to 83.1 or 91.2, respectively. Components

Avg ↑

− Hierarchy − Attributes − Layout Full

A+L H +L H +A H +A+L

83.1 62.5 91.2 97.1

91.2 0.0 91.6+0.4 89.8 -1.4 90.9 -0.3 90.0 -1.2 91.9+0.7 93.1+1.9

– 14.6 13.7 1.3 2.4 4.9 5.1

On 50 pages containing 1,392 visual nodes, binding coverage is 100%, with 78.9% independent position agreement. The manual audit in Appendix C covers 250 nodes and finds 73.2% actionable source localization. Figure 4 illustrates how Shared VS differences and implementation bindings guide an agentgenerated patch to correct layout and typography errors. Target persistence. Figure 5 compares five repair rounds on a 50-page Design2Code subset using GPT-5.2, with all four conditions sharing an initial C0 of 90.7 Avg. Rebuild uses the same LiveWorkspace feedback interface and initial Target, but reconstructs Target before each round under matched repair budgets and token limits. LiveWorkspace maintains fidelity around 92.2 after the first-round improvement, while Rebuild finishes at 91.2; Screenshot Feedback and Static Shared VS fluctuate in later rounds. Across the five rounds,

Table 4: Initial-generation ablation of Reference Target Shared VS on Design2Code with GPT-5.2. Full denotes H + A + L.

4.4

– 34.0 41.2 9.9 11.8 46.3 53.8

Collateral ↓

Table 5: Single round repair with GPT-5.2 from a shared C0 . T , C, ∆, and B denote Target, Current, Structured Diff, and binding; O denotes Raw Observations. Repair Rate (%) measures resolved initial Shared VS discrepancies; Final Fidelity is Design2Code Avg; Collateral counts newly defective Target nodes per page. Subscripts indicate fidelity changes from C0 .

Table 3: Cross-model transfer of Recovered Target Shared VS for initial generation. Shared VS and Raw Visual O are text-only inputs.

Representation

Results

T C ∆ B Repair ↑ Fidelity ↑

Img D2C SB

GPT-5.2 Screenshot GPT-5.2 Recovered Shared VS DeepSeek-V4 Raw Visual O DeepSeek-V4 Recovered Shared VS

Feedback

Iterative Repair in LiveWorkspace

Structured feedback. Table 5 compares feedback interfaces for a single round of repair from the same C0 using GPT-5.2. Repair Rate measures the proportion of initial Shared VS discrepancies re7

Target

(a) Initial C₀

(c) Corrected C₁

(b) Repair Shared VS Current

1

vs.

L.direction row A.font_size 48 px

2

Target

Diff Layout column Typography 20.8 px

Binding

.featured-posts-ul .featured-posts-headings .heading-2

3 Agent generates patch Target Shared VS Layout L.direction column Text A.font_size 20.8 px

flex-direction: column !important; font-size: 20.8px !important;

Current Shared VS: C₀ Layout L.direction row Text A.font_size 48 px

Current Shared VS: C₁ Layout L.direction column Text A.font_size 20.8 px

Figure 4: Automated repair with LiveWorkspace. Target–Current Shared VS differences expose layout-direction and font-size errors. Implementation bindings connect these errors to code locations, enabling the agent to generate a patch that restores the target layout and typography.

ing other repeated instances. Instance Preservation measures unchanged non-target repeated instances; Global Preservation measures unchanged non-target node presence, text, color, and geometry/layout state. Both preservation metrics are computed per case and then averaged. All reordering cases undergo independent rendered-page verification of the final visual order, as detailed in Appendix E. In Table 6, LiveWorkspace improves Edit Success from 41.0% to 93.0%, Instance Preservation from 76.5% to 97.5%, and Global Preservation from 93.9% to 99.2%. These results support reusing the harness’s state-alignment interface for localized editing while preserving non-target content and layout.

Screenshot Feedback Static Shared VS LiveWorkspace Rebuild

Figure 5: Five-round repair on a shared 50-page Design2Code subset with GPT-5.2. All conditions start from the same C0 ; LiveWorkspace retains Target Shared VS, while Rebuild reconstructs it before each round.

retaining Target uses 42.7% fewer input tokens and 15.7% fewer output tokens in mean per-page totals, as reported in Appendix D. 4.5

Method

Edit Success ↑ Inst. Pres. ↑ Global Pres. ↑

Direct Editing LiveWorkspace

41.0 93.0

76.5 97.5

93.9 99.2

Controlled Editing in LiveWorkspace Table 6: Controlled editing across 100 Appearance, Layout, and Structure cases. All results are percentages. Global Preservation covers non-target node presence, text, color, and geometry/layout.

LiveWorkspace updates and saves Target Shared VS according to the user instruction, then reuses the alignment-and-repair loop from §4.4. The harness recovers Current after execution to verify the edit. Visual GUI agents study instruction-to-element grounding (Cheng et al., 2024; Gou et al., 2025; Wu et al., 2025b; Lin et al., 2025). Our controlled cases test this grounding for localized code edits. We construct 100 controlled cases covering Appearance, Layout, and Structure. Each page contains three or four repeated instances whose source and visual orders differ, and instructions identify the target by visual position. Both methods receive the same inputs and use matched budget limits. Edit Success requires completing the requested change on the intended instance without modify-

5

Conclusion

We introduce LiveWorkspace, a Screenshot-toCode harness that externalizes visual understanding as Shared Visual State (Shared VS). LiveWorkspace retains Target and recovers Current after execution in a common, implementationindependent schema, with Structured Diff and implementation bindings guiding code modifications. With GPT-5.2 and matched model-call counts and budget limits, LiveWorkspace improves end-to-end reconstruction. Further experiments demonstrate 8

cross-model state reuse and improvements in iterative repair and controlled editing. Controlled comparisons highlight complementary benefits of structuring visual evidence under matched budgets and retaining target intent instead of reconstructing it across iterations. These findings support treating visual understanding as a persistent shared working state throughout code generation, execution, and modification.

CAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 1705–1712. ijcai.org. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, MingHsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025. Qwen2.5-vl technical report. CoRR, abs/2502.13923. Tony Beltramelli. 2018. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, EICS 2018, Paris, France, June 19-22, 2018, pages 3:1–3:6. ACM.

Limitations LiveWorkspace has several limitations. First, recovering Target Shared VS from screenshots remains imperfect, particularly in visual hierarchy, geometric estimation, and layout relations. These errors can propagate to subsequent generation and repair. Second, Target–Current alignment relies on normalized text matching and greedy spatial correspondence. While efficient, these rules may produce incorrect matches when text changes, elements move substantially, or page structures differ considerably. Our evaluation focuses on visual reconstruction of webpages and controlled user edits involving appearance, layout, and structure. It does not directly assess interactive behavior or multi-page/fullwebsite functionality (Xiao et al., 2024; Wan et al., 2024; Lu et al., 2025), responsive adaptation, or longer-term development workflows. In addition, recovering Current Shared VS relies on browser runtime information, including DOM, computed styles, and rendered geometry. Extending this approach to interfaces without comparable runtime access would require alternative state-recovery mechanisms.

Kanzhi Cheng, Qiushi Sun, Yougang Chu, Fangzhi Xu, Yantao Li, Jianbing Zhang, and Zhiyong Wu. 2024. Seeclick: Harnessing GUI grounding for advanced visual GUI agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 9313– 9332. Association for Computational Linguistics. Cheng Cui, Ting Sun, Suyin Liang, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Xueqing Wang, Changda Zhou, Hongen Liu, Manhui Lin, Yue Zhang, Yubo Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. 2026. Paddleocr-vl-1.5: Towards a multi-task 0.9b VLM for robust in-the-wild document parsing. CoRR, abs/2601.21957. Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, Yue Zhang, Wenyu Lv, Kui Huang, Yichao Zhang, Jing Zhang, Jun Zhang, Yi Liu, Dianhai Yu, and Yanjun Ma. 2025. Paddleocr 3.0 technical report. CoRR, abs/2507.05595. Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST 2017, Quebec City, QC, Canada, October 22 - 25, 2017, pages 845–854. ACM.

References Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Carbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. 2024. Screenai: A vision-language model for UI and infographics understanding. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pages 3058–3068. ijcai.org.

Jie Deng, Kaichun Yao, and Libo Zhang. 2026. Visrefiner: Learning from visual differences for screenshot-to-code generation. CoRR, abs/2602.05998. Tong Ge, Yashu Liu, Jieping Ye, Tianyi Li, and Chao Wang. 2025. Advancing vision-language models in front-end development via data synthesis. CoRR, abs/2503.01619.

Chongyang Bai, Xiaoxue Zang, Ying Xu, Srinivas Sunkara, Abhinav Rastogi, Jindong Chen, and Blaise Agüera y Arcas. 2021. Uibert: Learning generic multimodal representations for UI understanding. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJ-

Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. 2025. Navigating the digital world as humans do: Universal visual grounding for GUI agents. In The Thirteenth International Conference on Learning

9

Hugo Laurençon, Léo Tronchon, and Victor Sanh. 2024. Unlocking the conversion of web screenshots into HTML code with the websight dataset. CoRR, abs/2403.09029.

Representations, ICLR 2025, Singapore, April 2428, 2025. OpenReview.net. Yi Gui, Zhen Li, Yao Wan, Yemin Shi, Hongyu Zhang, Bohua Chen, Yi Su, Dongping Chen, Siyuan Wu, Xing Zhou, Wenbin Jiang, Hai Jin, and Xiangliang Zhang. 2025a. Webcode2m: A real-world dataset for code generation from webpage designs. In Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025- 2 May 2025, pages 1834–1845. ACM.

Ryan Li, Yanzhe Zhang, and Diyi Yang. 2025a. Sketch2code: Evaluating vision-language models for interactive web design prototyping. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 3921– 3955. Association for Computational Linguistics.

Yi Gui, Zhen Li, Zhongyi Zhang, Guohao Wang, Tianpeng Lv, Gaoyang Jiang, Yi Liu, Dongping Chen, Yao Wan, Hongyu Zhang, Wenbin Jiang, Xuanhua Shi, and Hai Jin. 2025b. Latcoder: Converting webpage design to code with layout-as-thought. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V.2, KDD 2025, Toronto ON, Canada, August 3-7, 2025, pages 721–732. ACM.

Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. 2025b. Ferret-ui 2: Mastering universal user interface understanding across platforms. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Yi Gui, Yao Wan, Zhen Li, Zhongyi Zhang, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, and Xiangliang Zhang. 2025c. Uicopilot: Automating UI synthesis via hierarchical code generation from webpage designs. In Proceedings of the ACM on Web Conference 2025, WWW 2025, Sydney, NSW, Australia, 28 April 2025- 2 May 2025, pages 1846–1855. ACM.

Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Stan Weixian Lei, Lijuan Wang, and Mike Zheng Shou. 2025. Showui: One vision-language-action model for GUI visual agent. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2025, Nashville, TN, USA, June 11-15, 2025, pages 19498–19508. Computer Vision Foundation / IEEE.

Hongcheng Guo, Wei Zhang, Junhao Chen, Yaonan Gu, Jian Yang, Junjia Du, Shaosheng Cao, Binyuan Hui, Tianyu Liu, Jianxin Ma, Chang Zhou, and Zhoujun Li. 2025. Iw-bench: Evaluating large multimodal models for converting image-to-web. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Findings of ACL, pages 6449– 6466. Association for Computational Linguistics.

Zhilin Liu, Ye Huang, Ting Xie, Ruizhi Zhang, Wen Li, and Lixin Duan. 2026. Coding with eyes: Visual feedback unlocks reliable GUI code generating and debugging. CoRR, abs/2604.19750. Yadong Lu, Jianwei Yang, Yelong Shen, and Ahmed Awadallah. 2024. Omniparser for pure vision based GUI agent. CoRR, abs/2408.00203.

A. Said Gurbuz, Sunghwan Hong, Ahmed S. Nassar, Marc Pollefeys, and Peter W. J. Staar. 2026. Moving beyond sparse grounding with complete screen parsing supervision. CoRR, abs/2602.14276.

Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. 2025. Webgenbench: Evaluating llms on generating interactive and functional websites from scratch. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2025, NeurIPS 2025, San Diego, CA, USA, December 2-7, 2025 / Mexico City, Mexico, November 30 - December 5, 2025.

Kai Huang, Jian Zhang, Xiaofei Xie, and Chunyang Chen. 2025. Seeing is fixing: Cross-modal reasoning with multimodal llms for visual software issue repair. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025, pages 1156–1168. IEEE.

Ruize Ma, Yilei Jiang, Shilin Zhang, Zheng Ma, Yi Feng, Vincent Ng, Zhi Wang, Xiangyu Yue, Chuanyi Li, and Lewei Lu. 2026. Failuremem: A failure-aware multimodal framework for autonomous software repair. CoRR, abs/2603.17826.

Yilei Jiang, Yaozhi Zheng, Yuxuan Wan, Jiaming Han, Qunzhong Wang, Michael R. Lyu, and Xiangyu Yue. 2025. Screencoder: Advancing visual-to-code generation for front-end automation via modular multimodal agents. CoRR, abs/2507.22827.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In Advances in

Peichao Lai, Jinhui Zhuang, Kexuan Zhang, Ningchang Xiong, Shengjie Wang, Yanwei Xu, Chong Chen, Yilei Wang, and Bin Cui. 2025. Webrenderbench: Enhancing web interface generation through layoutstyle consistency and reinforcement learning. CoRR, abs/2510.04097.

10

Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Fan Wu, Cuiyun Gao, Shuqing Li, Xin-Cheng Wen, and Qing Liao. 2025a. Mllm-based ui2code automation guided by UI layout information. Proc. ACM Softw. Eng., 2(ISSTA):1123–1145.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR.

Jason Wu, Xiaoyi Zhang, Jeffrey Nichols, and Jeffrey P. Bigham. 2021. Screen parsing: Towards reverse engineering of UI models from screenshots. In UIST ’21: The 34th Annual ACM Symposium on User Interface Software and Technology, Virtual Event, USA, October 10-14, 2021, pages 470–483. ACM. Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, and Yu Qiao. 2025b. OS-ATLAS: foundation action model for generalist GUI agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Jingyu Xiao, Jiantong Qin, Shuoqi Li, Man Ho Lam, Yuxuan Wan, Jen-tse Huang, Yintong Huo, and Michael R. Lyu. 2026a. Comuicoder: Componentbased reusable UI code generation for complex websites via semantic segmentation and element-wise feedback. CoRR, abs/2602.19276.

Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2025. Design2code: Benchmarking multimodal code generation for automated front-end engineering. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1: Long Papers, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 3956– 3974. Association for Computational Linguistics.

Jingyu Xiao, Yuxuan Wan, Yintong Huo, Zhiyao Xu, and Michael R. Lyu. 2024. Interaction2code: How far are we from automatic interactive webpage generation? CoRR, abs/2411.03292. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R. Lyu. 2025. Designbench: A comprehensive benchmark for mllm-based front-end code generation. CoRR, abs/2506.06251.

Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, and Michael R. Lyu. 2024. Mrweb: An exploration of generating multi-page resource-aware web code from UI designs. CoRR, abs/2412.15310.

Jingyu Xiao, Zhongyi Zhang, Haoran Hou, Yuxuan Wan, Yuan Jiang, Yintong Huo, and Michael R. Lyu. 2026b. Visualrepair: Dynamic tool calling and region focusing for visual software issue repair. CoRR, abs/2607.14075.

Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, and Michael R. Lyu. 2025. Divide-and-conquer: Generating UI code from screenshots. Proc. ACM Softw. Eng., 2(FSE):2099– 2122.

Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. In ESEC/FSE ’20: 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Virtual Event, USA, November 8-13, 2020, pages 1655–1659. ACM.

Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better LLM agents. In Fortyfirst International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, volume 235 of Proceedings of Machine Learning Research, pages 50208–50232. PMLR / OpenReview.net.

John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024.

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, and 2 others. 2025. Openhands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.

John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida Wang, and Ofir Press.

11

2025a. Swe-bench multimodal: Do AI systems generalize to visual software domains? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Zhen Yang, Wenyi Hong, Mingde Xu, Xinyue Fan, Weihan Wang, Jiele Cheng, Xiaotao Gu, and Jie Tang. 2025b. Ui2coden: A visual language model for test-time scalable interactive ui-to-code generation. CoRR, abs/2511.08195. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinhong Wang, Tianhua Tao, Junbo Li, Haonan Li, Preslav Nakov, Timothy Baldwin, Zhengzhong Liu, Eric P. Xing, Xiaodan Liang, and Zhiqiang Shen. 2024. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multimodal llms. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Yubo Zhang, Xueqing Wang, Manhui Lin, Yue Zhang, Penglongyi Deng, Ting Sun, Tingquan Gao, Zelun Zhang, Jiaxuan Liu, Changda Zhou, Hongen Liu, Suyin Liang, Cheng Cui, Yi Liu, Dianhai Yu, and Yanjun Ma. 2026. Pp-ocrv6: From 1.5m to 34.5m parameters, surpassing billion-scale vlms on OCR tasks. CoRR, abs/2606.13108. Xuanle Zhao, Deyang Jiang, Zhixiong Zeng, Lei Chen, Haibo Qiu, Jing Huang, Yufeng Zhong, Liming Zheng, Yilin Cao, and Lin Ma. 2025. Vincicoder: Unifying multimodal code generation via coarse-to-fine visual reinforcement learning. CoRR, abs/2511.00391.

12

A

State and Agent Interfaces

Target and Current denote runtime roles; Recovered and Reference distinguish Target construction sources. All instances use the same S = (H, A, L) schema.

Target is retained after construction. Each execution recovers Skc , recomputes ∆k , and refreshes Bk . Refinement reuses Target; editing updates Target before reusing alignment. Persistence concerns the external target state, while Current is recovered from each execution.

A.1

A.3

Visual State and Recovery Evidence

Visual state. H stores grouping and containment; A stores observable geometry, text, and paint; L stores within-group arrangements, direction, spacing, and padding. Synthetic groups need not correspond to single source elements, so the visual hierarchy need not be isomorphic to the DOM. Target recovery evidence. Recovered Target is constructed entirely from the screenshot, screenshot-derived region, text, and paint observations O, and the inferred hierarchy. GPT-5.2 infers hierarchy; the harness recovers attributes and layout from this evidence, without accessing reference DOM, computed styles, or reference-code rendering. Reference Target is deterministically recovered from the reference rendered page or implementation, using runtime DOM, computed styles, and painted geometry as evidence. The reconstruction coder receives only the resulting state. Current recovery evidence. Current uses runtime DOM, computed styles, and painted geometry. Canonicalization collapses non-visual wrappers, prunes invisible elements, and infers layout from painted geometry. Tags, classes, selectors, and source locations remain implementation evidence or independent binding, outside Shared VS. Implementation binding. For self-generated, single-file HTML, the binder parses source elements and greedily assigns visual nodes one-toone in document order, using tag compatibility and containment of the node text in the candidate element’s full get_text() output. Binding metadata remains separate from Shared VS. The position audit in Appendix C evaluates these assignments independently. A.2

Agent Inputs and Experimental Conditions

Initial generation. Construction uses the screenshot and O; code generation uses Target. Appendix B checks screenshot and reference-HTML access in the pilot C0 requests. Full repair interface. The agent receives Ck , S t , Skc , ∆k , Bk and decides on local patches or broader changes. The harness provides recovered states and execution feedback. Table 5 progressively adds states, Diff, and binding to compare interfaces. User editing. The agent locates affected Target ′ nodes and attributes, updates and saves S t , and reuses the full repair interface. Each execution recovers Current and refreshes Diff and binding against the updated Target. Representation conditions. Figure 3 and Table 3 evaluate initial-generation outputs without subsequent repair. They use the same benchmark pages as Table 1, which instead reports the final outputs of the complete LiveWorkspace runs. Their results are therefore reported separately by evaluation stage. Full in Table 4 uses Reference Target Shared VS and corresponds to the Reference condition in Figure 3, rather than the Recovered condition in the main table. A.4

Implementation Details

Visual observations. Region candidates are obtained with PP-DocLayoutV3 (Cui et al., 2026). Text detection and recognition use PPOCRv6_medium_det and PP-OCRv6_medium_rec (Zhang et al., 2026), respectively, from the PaddleOCR toolkit (Cui et al., 2025). Paint observations are extracted by h2_m1c_paint.py, which uses screenshot color connected components to recover painted non-text geometry, colors, and borders. These local visual tools run on CPU. GPT-5.2 separately infers hierarchy from the screenshot and the resulting observations. Canonicalization. Non-painting wrappers with no text of their own and exactly one child are collapsed; wrapper candidates include div, span, section, article, main, header, and footer.

Correspondence and State Lifecycle

Node IDs are independent. Text-bearing nodes match by normalized exact text, with center distance resolving ties. Remaining nodes match greedily and one-to-one by absolute bounding-box IoU (IoU ≥ 0.5), prioritizing larger overlaps. Attributes are compared only for matched pairs; unmatched reportable Target and Current nodes are MISSING and EXTRA, respectively. 13

Elements such as picture are treated as nonpainting wrappers. Invisible and off-canvas elements are pruned, with a 0.5 px collapsed-element threshold. Sequence, grid, overlay, and freeform layout relations, together with direction, gap, and padding, are inferred from child geometry; layout primitives are not assigned from CSS display labels.

Text-leaf node { "id": "t0", "type": "element", "role": "text", "attributes": { "geometry": {"width": 334.0, "height": 18.0}, "typography": {"color": "rgb(255, 255, 255)"}, "content": {"text": "SOUTH ASIA INSTITUTE | College of Liberal Arts"} }, "children": [] }

Diff tolerances. Width and height discrepancies are reported beyond max(2 px, 0.05 × target size). Position discrepancies are reported for center differences greater than 12 px, and gap uses a 6 px tolerance. Colors are parsed into integer RGB triples and compared exactly. Font family is not included in Diff.

A.5

Structured Diff · case 341 (excerpt) # Visual-State Diff (target 29 nodes, current 30, aligned 29) ## PROPERTY differences (22 nodes) node visual_container at (0.0, 0.0, 1280.0, 720.0): typography.color: target=’rgb(255, 255, 255)’ current=None node visual_container at (0.0, 0.0, 1280.0, 36.0): typography.color: target=’rgb(255, 255, 255)’ current=None node visual_container at (51.0, 213.0, 1106.0, 306.0): position.x: target=51.0 current=87 typography.color: target=’rgb(255, 255, 255)’ current=None node visual_container at (51.0, 213.0, 240.0, 135.0): position.x: target=51.0 current=87

Worked State and Diff Examples

The cards below reproduce a root-container node, a text-leaf node, and a Structured Diff excerpt. The Diff compares C0 with Recovered Target Shared VS for Design2Code case 341. It reports 29 Target nodes, 30 Current nodes, 29 aligned pairs, and property differences on 22 nodes; only the displayed portion is reproduced. None denotes an absent value in this output.

B

Initial-Generation Budget Pilot

We measured initial-generation budgets on 50 Design2Code pages. Direct makes one GPT-5.2 call per page. LiveWorkspace makes two calls: Target hierarchy inference and generation of C0 from the resulting Target Shared VS. All stages use GPT5.2. Table 7 illustrates pages 341, 1002, and 10032 from this study, reporting actual input/output usage returned by the API, configured token caps, and call latency. This pilot measures Target construction and initial code generation only; it does not include the automatic repair rounds used in the main-table evaluation. Accounting. Each row corresponds to one model call; Time is in seconds. Target denotes hierarchy inference, with subsequent attribute and layout recovery performed by the harness. Direct and C0 use token caps of 32,000 and 96,000, respectively. The Target-stage cap was not recorded in the pilot summary and is shown as a dash. The

Root-container node { "id": "G", "type": "region", "role": "visual_container", "attributes": { "geometry": {"width": 1280.0, "height": 720.0}, "appearance": {"fill": "rgb(51, 63, 72)"}, "typography": {"color": "rgb(255, 255, 255)"} }, "layout": { "type": "sequence", "direction": "column", "padding": [0.0, null, null, null], "gap": 17.0, "wrap": false } }

14

Case

Stage

Input Output

Cap Time

341

Direct 1,263 Target 2,219 C0 5,518

1,127 32,000 25.2 349 – 9.9 2,457 96,000 21.5

1002

Direct 1,263 Target 1,944 C0 3,727

856 32,000 23.6 420 – 10.7 1,658 96,000 15.2

10032 Direct 1,263 Target 2,286 C0 6,521

1,731 32,000 49.8 527 – 13.1 2,846 96,000 22.2

C

Implementation-Binding Audit

C.1

Coverage, Consistency, and Position Agreement

The binding audit covers 50 generated pages with 1,392 rendered visual nodes, comprising 761 textbearing and 631 non-text nodes. As summarized in Table 8, we distinguish binding availability, text/tag consistency, independent position agreement, and manually assessed actionable source localization. Availability and consistency. All 1,392 nodes receive bindings. For all 761 text-bearing nodes, the bound element’s full text contains the corresponding normalized node text; all 631 non-text nodes pass tag-consistency checks. These checks establish availability and text/tag consistency but do not independently identify the intended rendered element. Independent position audit. Each bound element receives a unique data-bchk marker before re-rendering. Its getBoundingClientRect() box is compared with the corresponding Current Shared VS node’s painted box. Both use documentabsolute coordinates at a viewport width of 1280 px and zero scroll offset. Position agreement requires the L1 distance between box centers to be at most 24 px. This signal is independent of the binder’s tag, text, and document-order checks. Agreement is 1,098/1,392 (78.9%), comprising 678/761 textbearing nodes (89.1%) and 420/631 non-text nodes (66.6%). This criterion measures spatial agreement rather than exact element identity or source-span correctness.

Table 7: API usage and model-call latency for three illustrative pages from the 50-page initial-generation budget study. All calls use GPT-5.2.

logs recorded no retries and no rendering of initial code C0 on these three pages. Per-page means. Across the 50 pages, Direct uses one call per page, averaging approximately 1.26K input tokens and 1.24K output tokens. LiveWorkspace’s two stages together use two calls, averaging approximately 7.51K input tokens and 2.81K output tokens. Target hierarchy inference alone averages approximately 2.19K input and 0.45K output tokens; C0 generation averages approximately 5.32K input and 2.36K output tokens. This study compares the initial-generation costs of GPT-5.2 Direct and LiveWorkspace, which use the same model but different call counts and token budgets. The main-table GPT-5.2 (Matched Budget) control instead matches actual model-call counts and stagewise budget limits.

Check

Count Rate (%)

Binding coverage Text consistency Tag consistency (non-text) Position agreement Text nodes Non-text nodes Source-line availability

Context checks. The metadata for all three C0 requests records no image attachment and no mention of reference HTML. These checks concern the separate code-generation call; hierarchy construction still uses the target screenshot.

Actionable source localization Text nodes Non-text nodes

Model and tool latency. Across the 50 pages, Direct’s mean model-call latency is 33.1 seconds. Target hierarchy inference and C0 generation average 11.6 and 20.3 seconds, respectively; their sum is 31.9 seconds for LiveWorkspace. These records use cached low-level visual observations. Separately measured M1 processing averages 10.2 seconds. Adding it gives a serial initial-generation estimate of 31.9 + 10.2 = 42.1 seconds. This is a sum of stage means, rather than a directly timed end-to-end latency.

1392/1392 761/761 631/631 1098/1392 678/761 420/631 1392/1392

100.0 100.0 100.0 78.9 89.1 66.6 100.0

183/250 114/135 69/115

73.2 84.4 60.0

Table 8: Implementation-binding audit on 50 pages. Position agreement is evaluated independently of text/tag consistency; actionable source localization is assessed by manual review.

C.2

Failure Modes and Source Locations

In the three-page illustrative audit, mismatches mainly involve containers and text–container cor15

respondences. Wrapper collapse or text promotion during canonicalization can shift visual-node order relative to source-element order, binding a node to a neighboring element or its parent container. Of 18 flagged bindings, 11 have center distances exceeding 100 px; smaller deviations also include tolerance-boundary cases around 28 px. One audited text node has painted box (0, 545, 195, 19), while its bound element has box (0, 545, 195, 76), expressed as (x, y, w, h) in pixels. The boxes share an origin, but the latter covers a taller container region. Text containment and nearby centers can therefore still select a parent container rather than the intended text element. All 1,392 bound elements have source-line metadata. A separate manual audit of 250 nodes reports actionable source localization of 183/250 (73.2%), comprising 114/135 text nodes (84.4%) and 69/115 non-text nodes (60.0%). A binding is actionable if it identifies a specific source element or style rule where the intended visual change can be made without further global source search. Source-line availability, position agreement, and actionable source localization are reported separately. Table 5 measures repair gains from adding binding, while this audit characterizes location cues in the 50-page audit. Together they support using binding as implementation-location cues for the agent. The full repair interface also retains visual nodes, Current state, and Structured Diff, rather than treating binding as a guarantee of exact localization.

D

the value of Shared VS as a generation interface with matched call counts and token limits. Avg ↑

Gain

Free-text analysis Recovered Shared VS

91.13 94.53

– +3.40

Table 9: Initial generation from free-text analysis or Recovered Target Shared VS, without refinement. Both conditions use two GPT-5.2 calls per page with matched stagewise token limits.

D.2

Retained versus Reconstructed Target

We compare two Target lifecycles: LiveWorkspace retains the initial Target, whereas Rebuild reconstructs Target before each repair round. Both conditions share the same C0 and initial Target, with matched repair budgets and token limits; actual usage remains below the limits. Figure 5 reports the shared initial Avg and results after repair rounds 1–5; Table 10 reports the recorded mean per-page total token usage across the five rounds. Target lifecycle

Input ↓

Output ↓

Rebuild Persist

27,505 15,774

15,472 13,048

Table 10: Mean per-page cumulative token usage over the five repair rounds in Figure 5. Persist corresponds to LiveWorkspace with retained Target.

Rebuild ends at 91.2 Avg after round five, below LiveWorkspace with retained Target. Retaining Target reduces recorded input and output token usage by 42.7% and 15.7%, respectively. The shared-subset comparison supports Target reuse across repair iterations.

Additional Controlled Experiments

We report controlled Design2Code experiments, using GPT-5.2 and Design2Code Avg to measure reconstruction quality. The free-text versus Shared VS generation control uses 50 Design2Code pages. The four conditions in Figure 5 share a 50-page Design2Code subset and its initial implementations. D.1

Intermediate input

E

Controlled Editing Protocol

E.1

Repeated Instances and Visual Targets

Each synthetic base page contains K ∈ {3, 4} repeated text elements or cards, with hidden unique ItemID markers used to identify instances during evaluation. CSS decouples source order from visual order using flex-direction: column-reverse, reversed order values, or absolute positioning with reversed vertical offsets. We render the base page and extract Current Shared VS, recording each instance’s rendered bounding box, marker, and source index. The target is determined from rendered geometry, for example the bottommost instance or the second instance from the top.

Free-Text Analysis versus Shared VS

We compare free-text analysis with Recovered Target Shared VS as intermediate interfaces. Both conditions use two model calls per page: intermediaterepresentation construction and code generation. Stagewise token limits and budgets are matched, and actual usage remains below the limits. In Table 9, Recovered Target Shared VS achieves 94.53 Avg versus 91.13 for free-text analysis, a 3.40-point advantage. This comparison supports 16

Instructions refer to these visual positions rather than source indices. Source order alone therefore does not identify the requested instance. E.2

state. After before–after node alignment, a node in Ui is changed if it is unmatched, its normalized text or RGB text color differs, or the geometry/layout checks identify a change. With Qi ⊆ Ui denoting these changed nodes,

Edits and Task-Specific Verification

LiveWorkspace updates and saves Target Shared VS according to the user instruction, then passes the updated Target to the same alignment and repair loop used for refinement. After execution, Current Shared VS is recovered for verification. Task-specific checks assess the requested change, while preservation scores assess non-target content. The cases cover text and color changes, layout changes, and card deletion or reordering. Color edits are verified after rendering the edited page: the instance nearest the recorded target box must have the requested color, and the other repeated instances must retain their colors. Geometric layout checks use rendered Current Shared VS geometry. For deletion and order checks, the scorer parses the edited HTML and compares marker presence and sequence with the recorded ground truth. Deletion requires removal of the target marker while retaining the others; order checks concern marker sequence in HTML. The marker-based checks themselves do not require rendering. All reordering cases additionally undergo independent renderedpage verification to confirm that the final visual order satisfies the instruction. Other editing cases requiring visual inspection are confirmed by manual review. E.3

GlobalPresi = 1 −

|Qi | . |Ui |

This metric measures preservation of non-target node presence, text, color, and geometry/layout state. Both preservation metrics are computed separately for each case and macro-averaged over the 100 cases. Direct Editing and LiveWorkspace receive the same inputs and matched budget limits; actual usage does not reach the limits.

F

Repair Metrics

For Table 5, Repair Rate and Collateral use a fixed Target for each page. Repair Rate counts initial discrepancy keys. Each key identifies a Target node and a differing attribute; a missing Target node is represented by a MISSING key. Let D0,p be the initial discrepancy keys on page p, and let Rp,m ⊆ D0,p be those resolved by condition m. The pagelevel rate is RepairRatep,m = 100

|Rp,m | . |D0,p |

Reported rates are macro-averaged over pages. Collateral counts newly defective Target nodes, once per node. Let Fp,0 and Fp,m denote the initial and post-repair defective Target-node sets. A Target node is defective if it has at least one attribute difference beyond tolerance or is missing. Then

Localized Success and Preservation

Let naivei indicate whether the target instance completes the requested type of edit, and let overi count modified non-target repeated instances. Edit Success is the mean localized-success indicator,

Collateralp,m = |Fp,m \ Fp,0 |.

locali = 1[naivei = 1 ∧ overi = 0].

The reported value is the mean count per page. This excludes extra Current nodes and additional attribute errors on nodes already defective initially. It is an absolute node count, not a normalized regression rate. Final Fidelity is evaluated independently of Shared VS Diff using Design2Code’s block matching and rendered-page evaluation. Generated and reference implementations are evaluated for Block, Text, Position, Color, and CLIP; their five-score mean is scaled to 0–100. Reference implementations are used by this evaluator, not provided to the reconstruction coder.

Thus, modifying the correct instance is insufficient if any other repeated instance is also modified. Instance Preservation is overi InstPresi = 1 − , Ki − 1 where Ki − 1 is the number of non-target repeated instances on the base page. Global Preservation uses rendered Shared VS nodes as its counting unit. The target node is selected among nodes whose text matches the instruction’s original label by proximity to the recorded target box. Let Ui contain all other nodes in the pre-edit rendered 17

Related documents

• SWE-Shift: Evaluating Coding Agents in Evolving Workspaces#1447563
• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks#1128610
• CodeMeepo: A Multi-Agent Workflow for Repository-Level Code Migration and Optimization#1457787
• CodeMeepo: A Multi-Agent Workflow for Repository-Level Code Migration and Optimization#1450870
• GenClaw: Code-Driven Agentic Image Generation#240391