LiveEvalBench: Toward Open-World Evaluation for Web Generation
Analyze & explore related research in Conceptio
Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.
LiveEvalBench: Toward Open-World Evaluation for Web Generation Yiyao Wang1∗ , Zhen Wen1 , Yinghao Tang1 , Yixiao Fu1 , Lin Yuan2† , Xiaolu Zhang2 , Jun Zhou2 , Wei Chen3‡ 1
State Key Lab of CAD&CG, Zhejiang University 2 Ant Group 3 State Key Lab of CAD&CG, Zhejiang University; Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China
arXiv:2608.03689v1 [cs.AI] 4 Aug 2026
Abstract Large language models are increasingly capable of synthesizing executable frontend projects, yet existing benchmarks still treat web generation as a static evaluation problem. We argue that frontend artifacts demand a different paradigm: they are interactive rather than static, admit diverse yet equally valid implementations, and evolve faster than rigid pipelines can accommodate. To address these gaps, we present LiveEvalBench, an automated framework that reformulates web-generation evaluation as an agentic, adaptive, and extensible process. LiveEvalBench instantiates evaluation as a collaborative review workflow, in which a Build Engineer, a Code Engineer, and a UI Tester collectively gather evidence across the full lifecycle of a frontend project, from deployment and code inspection to browser-based interaction. To handle implementation diversity, an adaptive protocol couples shared rubrics for cross-model comparability with implementation-grounded criteria tailored to each artifact. The framework further supports incremental integration of new evaluator roles and assessment dimensions without pipeline redesign. Experiments across diverse real-world web-generation scenarios show that LiveEvalBench aligns closely with human expert judgment and provides fine-grained insights into frontier models’ web generation capabilities. Code is available at https://github.com/wyysteelhead/LiveEvalBench.
Introduction Large language models (LLMs) are rapidly evolving from generating isolated code snippets to synthesizing complete, executable frontend projects (Si et al. 2025; Zhu et al. 2025; Wu et al. 2025; Tran et al. 2026). As project-level web generation becomes a practical capability of modern LLMs, a fundamental question emerges: how should we evaluate what models actually build? Despite this progress, evaluation has not kept pace. Existing benchmarks for web generation inherit assumptions from traditional code evaluation: they assess outputs through predefined criteria, fixed execution scripts, or static judgments over source code and screenshots (Si et al. 2025; Zhu et al. 2025; Zhang et al. 2025; He et al. 2026). However, frontend applications are not static outputs. They are interactive, admit diverse yet equally valid implementations, and continuously evolve ∗
Yiyao Wang, Zhen Wen, Yinghao Tang, and Yixiao Fu are with the State Key Lab of CAD&CG, Zhejiang University. E-mail: {wangyiyao, wenzhen, yinghaotang, 3210101100}@zju.edu.cn. † Lin Yuan, Xiaolu Zhang, and Jun Zhou are with Ant Group. ‡ Wei Chen is with the State Key Lab of CAD&CG, Zhejiang University, and also with the Laboratory of Art and Archaeology Image (Zhejiang University), Ministry of Education, China. E-mail: chenvis@zju.edu.cn.
!"#$
%&
Prior Works
"' #&! ()
LiveEvalBench
Figure 1: LiveEvalBench versus prior web coding benchmarks. While prior works treat evaluation as a one-shot scoring procedure, LiveEvalBench models it as a collaborative expert review workflow with complementary agents, an adaptive protocol, and an extensible framework.
over time. This creates a fundamental paradigm mismatch: (1) static evaluation cannot judge artifacts whose quality depends on runtime interaction; (2) fixed rubrics cannot fairly assess open-ended generation where the same query admits many valid solutions; and (3) rigid pipelines cannot keep up with the rapid evolution of web generation capabilities. In this paper, we present LiveEvalBench, an automated evaluation framework designed for open-world web generation. Rather than treating evaluation as a one-shot scoring procedure, LiveEvalBench models it as a collaborative expert review workflow that is agentic, adaptive, and extensible (Fig. 1). First, to evaluate interactive artifacts through interaction rather than static inspection, we instantiate evaluation as a multi-agent workflow. A Build Engineer deploys the project and produces runtime artifacts. A Code Engineer inspects implementation quality and instruction following from the source side. A UI Tester actively explores the running applica-
tion through browser-based interaction. Each agent operates with distinct tools and evidence channels, collectively covering the full lifecycle of a frontend project from deployment to code to live behavior. Second, to fairly evaluate open-ended generation where the same user request admits diverse implementations, we propose an adaptive evaluation protocol. Shared rubric items are fixed across all models answering the same query, preserving cross-model comparability. On top of these, implementationgrounded checks are synthesized for each generated project based on what the model actually built, so that scoring reflects the artifact’s true capability rather than penalizing valid but unexpected design choices. Third, to support an evaluation framework that evolves alongside web generation itself, we design an extensible infrastructure. Each evaluator is declared as a configuration of three components: a persona specifying who is judging, a set of criteria specifying what is judged, and a set of tools specifying what evidence is consulted. Adding a new evaluation perspective (e.g., a mobile-only user, a colorblind user) requires filling in this configuration rather than redesigning the pipeline. We instantiate LiveEvalBench on a benchmark of 100 real-world web generation queries evaluated across 11 frontier models. The resulting evaluation shows that current frontier models fail most often on interactive runtime behavior. Across multiple validation experiments, LiveEvalBench shows strong alignment with human judgment across evaluation dimensions. Our contributions are summarized as follows: 1. We identify a fundamental evaluation paradigm mismatch in web generation and propose LiveEvalBench, an agentic evaluation framework for multi-perspective assessment using build, code, and browser evidence. 2. We introduce an adaptive evaluation protocol that balances cross-model comparability with implementationaware scoring for open-ended web generation. 3. We develop an extensible evaluation infrastructure where new evaluator roles and assessment dimensions can be added through configuration without redesign. 4. We construct a benchmark for web generation and uncover strengths and weaknesses of frontier models overlooked by existing evaluations.
Related Work Benchmark for Web Generation We organize prior work by the kind of evidence the evaluator inspects: source code, runtime functionality, and rendered visuals. Code evaluation Pure code-generation benchmarks score programs by executing them against fixed unit tests, spanning function-level suites (Chen et al. 2021; Austin et al. 2021; Hendrycks et al. 2021; Liu et al. 2023a), contaminationresistant and library-rich variants (Jain et al. 2025; Zhuo et al. 2025), and repository-level engineering settings (Jimenez et al. 2024). Web-generation benchmarks that adopt codelevel checks face two complications: the artifact is a frontend
project rather than a single program, so the “code” channel must also cover build configuration and asset organization; and ground-truth tests written against a fixed reference no longer apply when many UI layouts can satisfy the same user request (Xu et al. 2025; Lei et al. 2026). Functional evaluation A second line evaluates whether the deployed webpage behaves correctly at runtime. Sandboxed automatic-test pipelines pair each task with predefined scripts executed in a controlled environment (Xu et al. 2025; Zhu et al. 2025), achieving reproducibility at the cost of binding tests, fixtures, and judging logic to specific task templates. A more recent category lets an evaluator agent explore the deployed application in a real browser instead of executing fixed scripts (Wu et al. 2025; He et al. 2026; Lu et al. 2025; Tran et al. 2026). WebCompass (Lei et al. 2026) is representative, using an agent-as-a-judge procedure that synthesizes targeted test cases in the browser during execution. Adjacent web-agent resources (Deng et al. 2023; Zhou et al. 2024; Koh et al. 2024) do not evaluate generated frontends but inform evaluator design. Visual evaluation A third line judges the rendered interface. One branch treats UI generation as visual translation and scores generations against a target design (Beltramelli 2018; Yun et al. 2024; Li, Zhang, and Yang 2025): Design2Code (Si et al. 2025) is representative, benchmarking multimodal models on reproducing real-world webpages from screenshots via visual similarity and human preference. A second branch renders the artifact and asks an MLLM judge to score it via a per-task checklist (Xiao et al. 2024): ArtifactsBench (Zhang et al. 2025) is representative, driving each artifact through scripted interactions and scoring visual and interactive integrity. Both branches use rubrics tied to a single target or task template. Limitations of Existing Benchmarks Across these efforts, existing benchmarks have made important progress in web generation evaluation, but two limitations remain. First, their adaptive capacity is often bounded by predefined evaluation artifacts (e.g., expert-defined checklists), which ties evaluation to the benchmark’s existing queries and makes it costly to evaluate newly emerging web generation scenarios. Second, their evaluation pipelines are usually rigid: adding a new evaluator role or assessment dimension requires redesigning the pipeline rather than extending the existing framework. LiveEvalBench addresses these limitations within a unified agentic evaluation framework, using an adaptive evaluation protocol that grounds shared rubric items into implementationspecific checks and an extensible evaluation infrastructure that supports new evaluator roles and assessment dimensions without redesigning the pipeline.
LLM-as-Judge and Multi-Evaluator Frameworks LLM-as-judge work spans single proprietary judges with structured rubrics (Zheng et al. 2023; Liu et al. 2023b, 2024), specialized open evaluator models (Kim et al. 2024a,b), studies of judge reliability, bias, and validity (Chen et al. 2025; Guerdan et al. 2025; Bean et al. 2025), and survey-level framings (Li et al. 2025). Two threads directly motivate our design:
Work
Perspectives
Method
Design
Build Src. Brwsr. Interact. Visual Exec. Adapt. Multi-Eval. Extens. HumanEval (Chen et al. 2021) SWE-bench (Jimenez et al. 2024) Design2Code (Si et al. 2025) Web-Bench (Xu et al. 2025) FrontendBench (Zhu et al. 2025) ArtifactsBench (Zhang et al. 2025) FronTalk (Wu et al. 2025) Vision2Web (He et al. 2026) Vibe Code Bench (Tran et al. 2026) WebCompass (Lei et al. 2026) LiveEvalBench
— △ — △ — — — △ △ △ ✓
✓ ✓ — ✓ ✓ △ — — — △ ✓
— — △ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
— — — ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
— — ✓ — △ ✓ ✓ ✓ △ ✓ ✓
✓ ✓ — ✓ ✓ △ ✓ ✓ ✓ ✓ ✓
— — — — — △ △ △ — ✓ ✓
— — — — — — △ ✓ ✓ ✓ ✓
— — — — — — — — — — ✓
Table 1: Comparison of representative code, web, and frontend evaluation benchmarks. ✓indicates explicit support, △indicates partial or indirect support, and —indicates that the capability is not a primary part of the evaluation protocol. Src., Brwsr., Interact., Adapt., Multi-Eval., and Extens. denote source, browser, interaction, adaptive, multi-evaluator, and extensible, respectively. multi-judge juries replace one large judge with heterogeneous smaller judges, matching or outperforming a single strong judge while reducing bias and cost (Verga et al. 2024; Dubois et al. 2023); and agent-as-a-judge equips evaluators with tools to inspect files, execute code, or browse pages, approaching human reliability on coding tasks and materially affecting outcomes in browser-based web evaluation (Zhuge et al. 2024; Tran et al. 2026). Existing panels operate over textual outputs rather than multi-channel evidence, and evaluator behaviors are rarely modularized as swappable components—gaps our agentic evaluation framework and extensible evaluation infrastructure address.
Build Engineer The Build Engineer owns the deployment perspective and operates within the project directory together with a shell. It is also responsible for producing the build artifacts in the input step: it extracts the project files, performs minor repairs to auxiliary configuration when necessary, and launches the application following the response’s setup instructions. During evaluation, the Build Engineer revisits its own build trajectory (i.e.), the commands it issued and the auxiliary repairs it made, and from this evidence judges how much extra effort the deployment demanded beyond what the model response itself prescribes.
Method
Code Engineer The Code Engineer takes the perspective of a developer inspecting the implementation, with access to the source tree and the ability to run commands in the terminal for verification. It evaluates the project on the code side along two aspects: implementation quality and instruction following. Implementation quality considers both code readability and robustness, reflecting whether the code is well organized and whether it handles edge cases gracefully. Instruction following checks whether hard requirements stated in the user query are faithfully realized in the implementation, such as the use of a designated framework (e.g., Vue or React).
Framework Overview The overview of LiveEvalBench is presented in Fig. 2. LiveEvalBench takes the user query and model response as input, and deploys the response into a runnable link and a deployed project on disk. These artifacts are then handed to three specialized evaluation agents, namely the Build Engineer, the Code Engineer, and the UI Tester, which examine complementary facets of the project in parallel (Fig. 2(a)). During examination, each agent follows an adaptive evaluation protocol that grounds shared rubric criteria into implementation-specific checks at runtime (Fig. 2(b)). The framework is also extensible, supporting new evaluator roles through a unified agent infrastructure (Fig. 2(c)). The resulting per-criterion judgments are aggregated into a final 90-point score: Build contributes 15 points, Code contributes 30 points, and UI contributes 45 points, with per-evaluator caps that prevent any single facet from dominating.
Evaluation Agents As shown in Fig. 2(a), LiveEvalBench employs three specialized evaluation agents that examine these complementary facets in parallel, each operating within the environment natural to its perspective.
UI Tester The UI Tester stands in for the end user and interacts with the running web link solely through the browser, without access to the underlying code. It assesses the project along three criteria: visual quality, interaction experience, and a set of query-specific criteria generated from the user query. Visual quality looks at whether the interface renders completely and presents itself in a polished and visually coherent manner. Interaction experience examines whether the interface responds clearly and flows intuitively under user actions. The query-specific criteria, tailored to what each query actually asks for, provide a thorough examination of functionality.
%
“I want a website...”
"I will launch the project, assess the effort..."
"I will evaluate code quality and instruction..."
$% !"# &'(#
"I will test the UI and functionality..." $#
Interaction Experience <div />
! •
Hover ▪ check visual effect • Click ▪ check zoom <button /> animation )(*&'(# ,# +# “You are modile user, ! ( you goal is to test...”
write w rite keyboard screenshot ( click click "# $ • Interaction • Visual Quality
Figure 2: Overview of the LiveEvalBench framework. Given a user query and a model response, the response is deployed into a runnable link and a project directory, which are then evaluated in parallel by three specialized agents (a). Each agent follows an adaptive evaluation protocol that grounds shared rubric criteria into implementation-specific checks at runtime (b), and the agent design is extensible, supporting new evaluators through configuration (c).
Adaptive Evaluation Protocol As sketched in Fig. 2(b), we decouple what each evaluator should check from how those checks work. The first concern adapts to the user query; the second adapts to what each model actually produced. What to check (per query) Each evaluator carries a set of fixed criteria along the dimensions described above. On top of these, the UI Tester additionally takes in query-specific criteria. To faithfully cover every requirement raised by the user, we extract the functional requirements expressed in the query and turn each of them into a corresponding criterion. These query-specific criteria are shared across all models answering the same query, ensuring comparability between models. Whether to take in query-specific criteria is itself a configuration choice; the Code Engineer opts out and relies solely on its fixed rubric. How to check (per implementation) A criterion such as interaction experience is too abstract to apply directly; it must be turned into concrete actions on this particular project. For each criterion, the evaluator first inspects the implementation through what it can observe. For example, the UI Tester can examine the rendered DOM and accessibility tree of the running project. For each applicable element it finds there, the evaluator then writes an individual executable check tailored to that element. As illustrated in Fig. 2(b), the criterion of interaction experience is grounded in the buttons and containers actually present in the project, and is then verified through hover and click actions tailored to those elements. The criteria themselves stay fixed across all models, while the concrete checks adapt to each implementation.
Extensible Evaluation Infrastructure For the framework to evolve alongside web generation, adding a new evaluation perspective must be cheap. This reduces
to a more basic question: what is the minimum needed to define an evaluator? We summarize it into three components, illustrated in Fig. 2(c) with a touch-only mobile user as a running example. The first is a persona that specifies who is judging, fixing the role the evaluator plays and the standards it brings to the interface. The second is a set of criteria that specifies what is judged; the mobile user in Fig. 2(c) still cares about interaction and visual quality, the same dimensions the UI tester would examine. What separates the two is the third component: a set of tools that specifies what evidence is consulted. The mobile user is barred from using the keyboard and can only interact with the interface by touch; sourcelevel access such as read and write is also denied. Once the triple is filled in, the orchestrator schedules the evaluator and aggregates its score under the same rules as the default jury, with no changes to framework code. This abstraction extends easily beyond the running example. A color-blind end-user, for instance, can be added by reusing the same persona and criteria slots while swapping in a new screenshot tool that applies a color-vision transform, so the evaluator sees the page as the user would. Other perspectives, such as a visually impaired user relying on the accessibility tree, follow the same recipe by filling in the three slots. The full schema for specifying an evaluator under this three-slot abstraction is provided in the supplementary material.
Benchmark Construction Query Construction LiveEvalBench consists of 100 frontend project specifications across three difficulty levels (28 Level-1 (L1), 43 Level-2 (L2), and 29 Level-3 (L3) queries) and six task categories adapted from ArtifactsBench (Zhang et al. 2025) and Design Arena (Design Arena 2025): Data Visualization (17), UI Component (19), Game (18), Web App (18), Website (14), and 3D Design (14). The queries also span three specification
To instantiate the benchmark with model-generated artifacts, we run each query through the 11 frontier code-capable LLMs listed in the Experiments section, yielding one generated frontend project per query–model pair. All models are queried
me Ga
Generated Frontend Projects
2 3 5
5
6
6
5
Ap
6
p
7
4
We b
5
10
7
5
e
Human Curation From the stratified pool we hand-pick the final 100 queries to cover all six categories and multiple specification granularities across the benchmark, without enforcing equal counts per cell; selection prioritizes clarity, feasibility, evaluability, and novelty as judged by 2 experts with frontend engineering experience. The full 100 specifications, with granularity, category, difficulty annotations, and per-query evaluation criteria, will be publicly released upon publication under a license permitting free research use.
5 7
sit
Difficulty Stratification Each candidate is scored by an LLM judge along three rubric dimensions (number of views, external dependency complexity, and interaction/state complexity) and mapped to L1/L2/L3 via a deterministic rule. L1 covers small, self-contained features confined to a single page, such as a single form or an isolated canvas effect; L2 covers moderately complex tasks that may depend on several external libraries, involve a larger codebase, or require nontrivial application logic; L3 covers multi-page or multi-view applications with cross-view navigation, or several intricate features composed together. Two authors verify the labels and resolve disagreements by discussion.
10
b We
Role-Conditioned Synthesis Real-world frontend specifications differ substantially in abstraction and technical detail. To capture this heterogeneity, we rewrite each seed under four roles with distinct granularity: “end user” (intent-driven, non-technical), “designer” (visual and interaction details), “project manager” (structured requirements and acceptance criteria), and “developer” (component decomposition, APIs, edge cases). Rewrites are produced by three frontier models, namely Gemini 3 Pro Preview, GPT-5.2, and Claude Sonnet 4.5, using role-specific prompts provided in the supplementary material. This yields a pool of ∼4,000 candidate specifications that vary along two orthogonal axes: topical diversity (from seeds) and stylistic diversity (from roles and rewriters).
2
es ign
Seed Collection We aggregate real-world frontend requests from three complementary sources: public tutorials and coding guides, social media platforms, and task samples from ArtifactsBench (Zhang et al. 2025). Unlike prior benchmarks sourced from sites like GitHub, we treat social media as a primary source, since it is where practitioners share the most up-to-date queries probing frontier code-generation models. We retain only items containing a self-contained, implementable specification, and deduplicate across sources. Seeds serve only as topical anchors and are never used directly.
UI Component
3D D
granularities: abstract intent-level requests (46), productrequirements-document (PRD)-style specifications (41), and functional descriptions (13). The joint distribution over categories and difficulty levels is shown in Fig. 3. We build the benchmark through a four-stage pipeline.
Data Visualization L1 (Beginner)
L2 (Intermediate)
L3 (Advanced)
Figure 3: Distribution of LiveEvalBench queries across six categories. Each petal is one category, and the three concentric layers break it down by difficulty (L1 inner / L2 middle / L3 outer).
in a single turn with provider-default decoding settings. The only modification to the raw user specification is a single prepended instruction that frames the model as a frontend engineer and asks it to return frontend project content. This minimal-intervention protocol isolates each model’s native web generation ability and keeps comparisons faithful to default behavior.
Per-Query Evaluation Criteria LiveEvalBench is released with a set of criteria attached to each query. These criteria are produced following the adaptive evaluation protocol and manually reviewed for faithfulness. The full schema is provided in the supplementary material.
Experiments We organize the empirical study around three questions: (RQ1) how state-of-the-art models compare on LiveEvalBench, (RQ2) whether LiveEvalBench agrees with human judgment on the same generated projects, and (RQ3) whether LiveEvalBench provides reliable evaluation signals. Main Benchmark Evaluation We evaluate 11 frontier models: Claude Sonnet 4.6, Claude Opus 4.7, GPT-5.5, GPT5.4 Mini, Gemini 3.1 Pro, DeepSeek V4 Pro, Kimi K2.6, GLM 5.1, MiniMax M2.7, Ling 2.6 1T, and Qwen3.5-27B. The main leaderboard evaluates these models on all 100 LiveEvalBench queries. Each query–model record is evaluated three times using the same evaluation procedure, and criterion-level judgments are aggregated by majority before computing the main leaderboard scores. Human Agreement We select 70 generated frontend projects from the main benchmark evaluation for humanagreement validation. Each item is rated independently by
Claude Opus 4.7
Claude Opus 4.7
70.7
3.4 2.9 4.3
23.7
32.7
GLM 5.1
14.0
22.6
31.5
68.1
GLM 5.1
GPT-5.5
13.6
23.8
30.7
68.0
GPT-5.5 1.4 3.6 2.6 4.2
Claude Sonnet 4.6
12.5
23.8
29.2
65.5
Gemini 3.1 Pro
13.8
23.1
27.5
64.4
GPT-5.4 Mini
13.1
23.2
26.7
63.1
Kimi K2.6
14.0
21.6
26.8
62.4
DeepSeek V4 Pro
13.1
22.0
25.9
60.9
MiniMax M2.7
12.0
22.7
24.3
59.0
Ling 2.6 1T
12.2
22.9
Qwen 3.5 27B
12.0
22.9
0
15
30
Gemini 3.1 Pro
19.8
54.6
45
60
Code (30)
4.8
24.5
5.0
12.5
25.6
6.4
11.9
26.9
6.5
11.7
27.6
3.6
UI (45)
Figure 4: Main leaderboard on LiveEvalBench, visualized as Build + Code + UI sub-scores stacked to the 90-point total. Each segment is annotated with its sub-score and the row total is appended on the right. Models are ordered by total descending.
three annotators on the same criteria as the UI Tester. We compare the UI Tester’s three-run majority judgment with the human majority judgment, and report percent agreement and Gwet’s AC1, a chance-corrected metric for imbalanced pass/fail labels.
29.1
7.8
11.3
4.7 2.6
7.6
13.1
31.0
Ling 2.6 1T 2.8 4.2 2.9
8.5
12.7
31.1
Qwen 3.5 27B 3.0 3.5 3.7
9.8
MiniMax M2.7 3.0
90
22.0
10.4
DeepSeek V4 Pro 1.9 4.0 4.0
75
21.9
5.4
GPT-5.4 Mini 1.9 3.6 3.1
Scores of 11 models Build (15)
3.1 3.8
Kimi K2.6
7.8 10.2
5.7
Claude Sonnet 4.6 2.5 3.6 2.6
58.9
23.8
3.6 3.8
Build Smoothness Impl. Quality Instr. Following Visual Quality Interaction
19.3
8.0
14.3
0
10
15.4
20
30
35.4 40
Lost points of 11 models
Figure 5: Loss attribution by scoring dimension, on the 90point scale. Each row is a model; each segment is the points lost on one dimension; the value at the right of the bar is the total loss. Models are ordered by total loss ascending (best on top). Dimension / Criteria
Agreement ↑ AC1 ↑
UI Tester majority vs. human majority Visual quality 88.7% Interaction experience 83.0% Query-specific criteria 85.6% Overall 85.7%
0.831 0.718 0.805 0.793
Evaluator-Role Ablation We compare the full roleseparated evaluator with a single-evaluator ablation on the human-agreement subset, where one evaluator assesses all dimensions covered by the role-separated design in a single evaluation pass.
Table 2: Agreement of the UI Tester with human majority judgments. AC1 denotes Gwet’s chance-corrected agreement coefficient.
Reliability Evaluation We conduct two reliability evaluations. First, we compare three-time evaluation with a single evaluation pass to test whether three-time evaluation improves score stability. Second, we randomly sample 20 web generation queries and evaluate them with Kimi K2.6 and Qwen3.7-Plus under the same evaluation protocol as the main benchmark, then measure agreement across evaluator models.
Build sub-scores are more compressed, while Code sub-scores show moderate variation. This pattern reflects a natural hierarchy of frontend coding competence: successful builds are the entry-level bar, source-level quality is a step further, and robust UI interaction is the highest bar.
Results and Analysis Main Benchmark Evaluation We now report model performance on LiveEvalBench, with the main results presented in Figs. 4–6. Fig. 4 gives the overall leaderboard with each model’s total score broken down into the Build / Code / UI sub-scores, Fig. 5 shows where each model loses points across the five scoring dimensions, and Fig. 6 reports how scores change across the L1/L2/L3 difficulty levels. We discuss the key takeaways below. UI behavior matters most for frontend coding ability As shown in Fig. 4, which reports each model’s total score decomposed into the three per-evaluator sub-scores (Build / Code / UI), UI exhibits the widest range, spanning from 19.79 (Qwen3.5-27B) to 32.74 (Claude Opus 4.7).
Performance varies across difficulty levels Fig. 6 stratifies per-model scores by L1/L2/L3 difficulty. Aggregate scores decrease from L1 (63.8) to L2 (59.8) and L3 (55.5), indicating that the difficulty taxonomy captures a meaningful score gradient. The same trend appears for most individual models, with L1 generally receiving the highest scores and L3 the lowest in most cases. The leaderboard separates model performance tiers The updated leaderboard shows a clear spread in total scores, from 70.71 for Claude Opus 4.7 to 54.61 for Qwen3.5-27B. The top group remains led by Claude Opus 4.7, GLM 5.1, and GPT-5.5, while the lower-scoring models continue to lose the most points on runtime-facing UI behavior.
Score by Difficulty × Model (0–90) 90
L1
76.1
73.1
74.4
70.0
66.5
69.5
62.7
70.2
71.1
65.3
60.5
80
n=24
n=25
n=23
n=24
n=23
n=24
n=24
n=24
n=24
n=23
n=26
70
L2
68.7
70.8
66.7
66.4
66.1
63.6
61.8
60.4
60.1
59.9
55.0
50
n=37
n=36
n=38
n=40
n=37
n=36
n=39
n=35
n=36
n=37
n=36
40
Score on 90
Difficulty
60
30
L3
69.4
62.2
65.9
61.7
61.4
58.6
62.8
56.4
50.0
54.4
50.2
20
n=39
n=39
n=38
n=36
n=39
n=40
n=37
n=41
n=40
n=40
n=38
10
.7
5.1
4 us
M
GL
p
eO
d lau
.5
-5 PT
G
de lau
C
4 et nn
.6
So
C
m Ge
i in
ro 1P
3.
ni Mi 5.4
TGP
0
K
2.6 iK im
.7
o
k ee
S ep De
r 4P
2 xM
V
Ma ni
Mi
ng
Li
2.6
1T
B
en Qw
7 52
3.
Figure 6: Score by difficulty × model on the 90-point scale. Cells annotated with the per-cell sample size n. Difficulty levels (L1/L2/L3) reflect the query taxonomy defined in Benchmark Construction. Variant
0
1–29
30–59
60–89
90
Evaluation protocol
Mean per-query SD ↓
Mean CV ↓
Full 1-agent
18 21
4 0
17 0
28 0
3 49
Single evaluation pass Repeated evaluation
23.81 17.62
0.591 0.395
Table 3: Score distribution for evaluator-role ablation on the 70-case subset. Scores are on the 90-point scale.
Table 4: Repeated-evaluation reliability check. Repeated evaluation reduces per-query score variability compared with a single evaluation pass.
Human Agreement Tab. 2 reports agreement between the UI Tester and human majority judgments. Overall agreement reaches 85.7% with AC1 = 0.793, with similarly strong agreement across visual quality, interaction experience, and query-specific criteria. These results indicate that the LiveEvalBench evaluation framework provides human-aligned diagnostic signals for UI assessment.
Evaluator-Role Ablation Tab. 3 shows that the single-evaluator ablation collapses to a bimodal score distribution: on the 70-case subset, it places 21 cases at 0 and 49 cases at 90, with no intermediate scores. By contrast, the full role-separated design distributes scores across the scale. This indicates that separating evaluator roles provides finer diagnostic granularity rather than merely shifting average scores.
Reliability Evaluation Repeated Evaluation Improves Stability As shown in Tab. 4, repeated evaluation yields lower per-query score variability than a single evaluation pass. The mean per-query SD decreases from 23.81 to 17.62, and the mean CV decreases from 0.591 to 0.395. Agreement Across Evaluator Models Across evaluatormodel configurations, Kimi K2.6 and Qwen3.7-Plus reach
90.1% agreement, with substantial chance-corrected agreement (Cohen’s κ = 0.641). This suggests that LiveEvalBench generalizes across evaluator models.
Conclusion We presented LiveEvalBench, a multi-agent framework for evaluating LLM-generated frontend projects. The framework integrates complementary agents that evaluate a generated project from build, source, and runtime perspectives, an adaptive evaluation protocol that combines predefined criteria with implementation-grounded checks synthesized per project, and an extensible orchestration design that abstracts each evaluator into a configurable role, allowing new evaluation perspectives to be added with minimal effort. Building on this framework, we constructed a diverse benchmark of 100 real-world web generation queries across difficulty tiers and categories, and evaluate 11 frontier LLMs on this benchmark. The benchmark results show that runtime interaction remains the largest source of lost points for current web generation models. Complementary validation studies further show that LiveEvalBench provides reliable and human-aligned evaluation signals. We hope LiveEvalBench offers a new perspective on evaluating generative coding systems and a foundation for the community to build richer, more faithful evaluations.
References Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Bean, A. M.; Kearns, R. O.; Romanou, A.; Hafner, F. S.; Mayne, H.; Batzner, J.; Foroutan Eghlidi, N.; Schmitz, C.; Korgul, K.; Batra, H.; Deb, O.; Beharry, E.; Emde, C.; Foster, T.; Gausen, A.; Grandury, M.; Han, S.; Hofmann, V.; Ibrahim, L.; Kim, H.; Kirk, H. R.; Lin, F.; Liu, G.; Luettgau, L.; Magomere, J.; Rystrø m, J.; Sotnikova, A.; Yang, Y.; Zhao, Y.; Bibi, A.; Bosselut, A.; Clark, R.; Cohan, A.; Foerster, J.; Gal, Y.; Hale, S.; Raji, D.; Summerfield, C.; Torr, P.; Ududec, C.; Rocher, L.; and Mahdi, A. 2025. Measuring what Matters: Construct Validity in Large Language Model Benchmarks. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38. Curran Associates, Inc. Beltramelli, T. 2018. pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM SIGCHI symposium on engineering interactive computing systems, 1–6. Chen, L.; Zhang, Z.; Tan, H.; Dai, Q.; Hao, Y.; Dong, Z.; and Chen, X. 2025. Distributional LLM-as-a-Judge. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38, 30038–30067. Curran Associates, Inc. Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Deng, X.; Gu, Y.; Zheng, B.; Chen, S.; Stevens, S.; Wang, B.; Sun, H.; and Su, Y. 2023. Mind2Web: Towards a Generalist Agent for the Web. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 28091– 28114. Curran Associates, Inc. Design Arena. 2025. Design Arena. https://www.designarena. ai/. Accessed: 2026-07-27. Dubois, Y.; Li, X.; Taori, R.; Zhang, T.; Gulrajani, I.; Ba, J.; Guestrin, C.; Liang, P.; and Hashimoto, T. B. 2023. AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. In Advances in Neural Information Processing Systems. Guerdan, L.; Barocas, S.; Holstein, K.; Wallach, H.; Wu, S.; and Chouldechova, A. 2025. Validating LLM-as-a-Judge Systems under Rating Indeterminacy. In Belgrave, D.; Zhang, C.; Lin, H.; Pascanu, R.; Koniusz, P.; Ghassemi, M.; and Chen, N., eds., Advances in Neural Information Processing Systems, volume 38, 112282–112350. Curran Associates, Inc. He, Z.; Hong, W.; Yang, Z.; Pan, Z.; Liu, M.; Gu, X.; and Tang, J. 2026. Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification. arXiv preprint arXiv:2603.26648.
Hendrycks, D.; Basart, S.; Kadavath, S.; Mazeika, M.; Arora, A.; Guo, E.; Burns, C.; Puranik, S.; He, H.; Song, D.; et al. 2021. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938. Jain, N.; Han; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds., International Conference on Learning Representations, volume 2025, 58791–58831. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues? In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.; and Sun, Y., eds., International Conference on Learning Representations, volume 2024, 54107–54157. Kim, S.; Shin, J.; Cho, Y.; Jang, J.; Longpre, S.; Lee, H.; Yun, S.; Shin, S.; Kim, S.; Thorne, J.; and Seo, M. 2024a. Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models. In International Conference on Learning Representations. Kim, S.; Suk, J.; Longpre, S.; Lin, B. Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M. 2024b. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 4334–4353. Miami, Florida, USA: Association for Computational Linguistics. Koh, J. Y.; Lo, R.; Jang, L.; Duvvur, V.; Lim, M.; Huang, P.-Y.; Neubig, G.; Zhou, S.; Salakhutdinov, R.; and Fried, D. 2024. VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 881–905. Bangkok, Thailand: Association for Computational Linguistics. Lei, X.; Che, X.; Xiong, J.; Zhang, C.; Huang, Y.; Zhou, C.; Huang, H.; Liu, M.; Zhu, L.; Ye, H.; et al. 2026. WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models. arXiv preprint arXiv:2604.18224. Li, D.; Jiang, B.; Huang, L.; Beigi, A.; Zhao, C.; Tan, Z.; Bhattacharjee, A.; Jiang, Y.; Chen, C.; Wu, T.; Shu, K.; Cheng, L.; and Liu, H. 2025. From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge. In Christodoulopoulos, C.; Chakraborty, T.; Rose, C.; and Peng, V., eds., Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2757–2791. Suzhou, China: Association for Computational Linguistics. ISBN 979-8-89176-332-6. Li, R.; Zhang, Y.; and Yang, D. 2025. Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping. In Chiruzzo, L.; Ritter, A.; and Wang, L., eds., Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 3921–3955. Albuquerque, New Mexico: Association for Computational Linguistics. ISBN 979-8-89176-189-6.
Liu, J.; Xia, C. S.; Wang, Y.; and Zhang, L. 2023a. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in neural information processing systems, 36: 21558–21572. Liu, X.; Lei, X.; Wang, S.; Huang, Y.; Feng, A.; Wen, B.; Cheng, J.; Ke, P.; Xu, Y.; Tam, W. L.; Zhang, X.; Sun, L.; Gu, X.; Wang, H.; Zhang, J.; Huang, M.; Dong, Y.; and Tang, J. 2024. AlignBench: Benchmarking Chinese Alignment of Large Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11621–11640. Bangkok, Thailand: Association for Computational Linguistics. Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023b. G-Eval: NLG Evaluation Using GPT-4 with Better Human Alignment. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2511–2522. Singapore: Association for Computational Linguistics. Lu, Z.; Yang, Y.; Ren, H.; Hou, H.; Xiao, H.; Wang, K.; Shi, W.; Zhou, A.; Zhan, M.; and Li, H. 2025. WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratch. arXiv:2505.03733. Si, C.; Zhang, Y.; Li, R.; Yang, Z.; Liu, R.; and Yang, D. 2025. Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering. In Chiruzzo, L.; Ritter, A.; and Wang, L., eds., Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 3956–3974. Albuquerque, New Mexico: Association for Computational Linguistics. ISBN 979-8-89176-189-6. Tran, H.; Nashold, L.; Krishnan, R.; Bigeard, A.; and Gu, A. 2026. Vibe Code Bench: Evaluating AI models on end-to-end web application development. arXiv preprint arXiv:2603.04601. Verga, P.; Hofstatter, S.; Althammer, S.; Su, Y.; Piktus, A.; Arkhangorodsky, A.; Xu, M.; White, N.; and Lewis, P. 2024. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796. Wu, X.; Xue, Z.; Yin, D.; Zhou, S.; Chang, K.-W.; Peng, N.; and Wen, Y. 2025. FronTalk: Benchmarking Front-End Development as Conversational Code Generation with MultiModal Feedback. arXiv preprint arXiv:2601.04203. Xiao, J.; Wan, Y.; Huo, Y.; Wang, Z.; Xu, X.; Wang, W.; Xu, Z.; Wang, Y.; and Lyu, M. R. 2024. Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototyping. arXiv preprint arXiv:2411.03292. Xu, K.; Mao, Y.; Guan, X.; and Feng, Z. 2025. Web-bench: A llm code benchmark based on web standards and frameworks. arXiv preprint arXiv:2505.07473. Yun, S.; Lin, H.; Thushara, R.; Bhat, M. Q.; Wang, Y.; Jiang, Z.; Deng, M.; Wang, J.; Tao, T.; Li, J.; Li, H.; Nakov, P.; Baldwin, T.; Liu, Z.; Xing, E. P.; Liang, X.; and Shen, Z. 2024. Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs. arXiv preprint arXiv:2406.20098.
Zhang, C.; Li, Y.; Xu, C.; Liu, J.; Liu, A.; Zhou, C.; Deng, K.; Wu, D.; Huang, G.; Li, K.; et al. 2025. Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation. arXiv preprint arXiv:2507.04952. Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, volume 36, 46595–46623. Curran Associates, Inc. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; Alon, U.; and Neubig, G. 2024. WebArena: A Realistic Web Environment for Building Autonomous Agents. In Kim, B.; Yue, Y.; Chaudhuri, S.; Fragkiadaki, K.; Khan, M.; and Sun, Y., eds., International Conference on Learning Representations, volume 2024, 15585–15606. Zhu, H.; Zhang, Y.; Zhao, B.; Ding, J.; Liu, S.; Liu, T.; Wang, D.; Liu, Y.; and Li, Z. 2025. Frontendbench: A benchmark for evaluating llms on front-end development via automatic evaluation. arXiv preprint arXiv:2506.13832. Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y.; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y.; Shi, Y.; Chandra, V.; and Schmidhuber, J. 2024. Agent-as-aJudge: Evaluate Agents with Agents. arXiv:2410.10934. Zhuo, T. Y.; Vu, M. C.; Chim, J.; Hu, H.; Yu, W.; Widyasari, R.; Yusuf, I. N. B.; Zhan, H.; He, J.; Paul, I.; Brunner, S.; GONG, C.; Hoang, J.; Zebaze, A.; Hong, X.; Li, W.-D.; Kaddour, J.; Xu, M.; Zhang, Z.; Yadav, P.; Jain, N.; Gu, A.; Cheng, Z.; Liu, J.; Liu, Q.; Wang, Z.; Hui, B.; Muennighoff, N.; Lo, D.; Fried, D.; Du, X.; de Vries, H.; and Von Werra, L. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In Yue, Y.; Garg, A.; Peng, N.; Sha, F.; and Yu, R., eds., International Conference on Learning Representations, volume 2025, 66602–66656.
Supplementary Material Overview. Section A provides implementation details for the LiveEvalBench evaluation framework, including evaluator roles, configuration, scoring, and execution environment. Section B documents benchmarkconstruction details, including the query/checklist schema and role-conditioned synthesis prompts. Section C reports supplementary result analyses for score patterns, evaluatormodel agreement, and evaluator-role ablation. Section D presents qualitative case analyses from real evaluation traces. Section E discusses current limitations and future directions.
A
Implementation Details
This section documents the implementation details behind the LiveEvalBench evaluation framework. It describes the default evaluator agents, the configuration schema used by the extensible evaluation infrastructure, the score aggregation procedure, and the compute environment used for the reported runs. The evaluation framework, evaluator configurations, benchmark construction scripts, and analysis scripts required to reproduce the reported experiments will be publicly released upon publication under a license permitting free research use.
A.1
Evaluator Specifications and Prompts
Agent Configurations Table 5 provides information on the evaluator agents, detailing their persona, evaluation criteria, and allowed tools. Prompts The prompt templates for the evaluation process are illustrated in two parts. The adaptive evaluation protocol is detailed in Figures 12–14; in practice, to optimize costs, we integrate the synthesis of the adaptive protocol with the synthesis of query-specific criteria, which allows the system to dynamically select evaluation actions based on the project structure. Furthermore, the system prompts for the Build Engineer, Code Engineer, and UI Tester during task execution are provided in Figures 15–17.
A.2
Evaluator Configuration Schema
Table 6 lists the JSON fields that constitute an evaluator configuration in the released framework. Adding a new evaluator amounts to writing one such JSON file and dropping it into the agent registry; the orchestrator reads these fields and routes the pipeline accordingly. If a new tool is required, additional code for the new tool needs to be written, but no other engineering work is necessary.
A.3
Scoring and Aggregation Details
The reported score is normalized to a 90-point scale over five scoring dimensions. The Build Engineer contributes 15 points for build smoothness. The Code Engineer contributes 30 points, split evenly between implementation quality and instruction following. The UI Tester contributes 45 points, split evenly between visual quality and interaction experience. Query-specific criteria are used by the UI Tester as implementation-grounded evidence for the structured report, but they are not reported as a separate top-level score dimension in the current leaderboard.
In the adaptive evaluation protocol, each criterion may be decomposed into one or more concrete tasks. The scoring rule from tasks to a criterion is that if any single task fails, the entire criterion is considered failed, and all points assigned to that criterion are forfeited.
A.4
Computational Resources and Evaluation Environment
For the reported three-run leaderboard, all evaluation agents (Build Engineer, Code Engineer, and UI Tester) use the same evaluator-model configuration across runs; the released evaluation artifacts record the concrete model settings for each run. Compute infrastructure All experiments were conducted on machines with 64 GB RAM and 32 CPU cores. Evaluation was parallelized at the granularity of a single query–model pair, with up to 8 such pairs processed concurrently. Evaluation environment The evaluation pipeline runs on Ubuntu 22.04 LTS with Node.js 20.x, using Playwright 1.40 for browser automation (Chromium 120.0), npm 10.x for package management, and standard build tools (Vite 5.x, webpack 5.x) as detected from each project’s configuration. To reduce setup latency, each project directory is pre-populated with commonly used packages (react, vue, tailwindcss, etc.); only missing dependencies trigger additional npm install calls during evaluation.
B
Benchmark Construction Details
This section provides additional benchmark-construction material. It specifies the released query and checklist schema and includes the role-conditioned synthesis prompts used to rewrite seed requests into benchmark queries.
B.1
Query Specification and Checklist Schema
To make the format of a LiveEvalBench query and its accompanying evaluation checklist concrete, we reproduce one representative pair below. The remaining 59 queries follow the same structure: a short title, a self-contained naturallanguage description of the desired frontend project, and an enumeration of the visible features expected from any conforming implementation. No framework, library, or file layout is prescribed. Each query is released together with a checklist of evaluation criteria. The checklist is stored as a structured record with the following fields: • query_id – Stable identifier of the query, matching the entry in the released query list. • fixed_criteria – The evaluator-scored criteria shared across all queries: build smoothness, implementation quality, instruction following, visual quality, and interaction experience. • query_specific_criteria – A list of records, each containing:
Evaluator
Persona
Criteria
Tools
Build Engineer
Deployment engineer who extracts, config- Build smoothness: whether the build file-system read/write, shell ures, and launches the project completed without manual repairs, exec, runtime probe, port manmissing dependencies, or configura- agement, build trace inspection tion errors
Code Engineer
Developer inspecting implementation qual- Implementation quality (readability file-system read/write, one-off ity and instruction adherence + robustness), instruction following shell exec
UI Tester
End user interacting with the running ap- Visual quality, interaction experi- browser navigation, DOM inplication through the browser ence, query-specific functional re- spection, element interaction quirements (click, type, hover, scroll), screenshot capture
Table 5: Default jury of evaluators: persona, evaluation criteria, and allowed tools. Each evaluator is configured through the extensible evaluation infrastructure. ∗ name — a short label for the criterion. ∗ description — a one- or two-sentence explanation of the feature being checked. The query-specific criteria are predefined per query and held fixed across all models, ensuring that every implementation of the same query is evaluated against a consistent set of criteria.
B.2
Role-Conditioned Synthesis Prompts
The role-specific prompt templates used for query construction are given in Figures 18–23.
C
Supplementary Result Analysis
This section reports supplementary analyses that support the main empirical findings. It includes category-level score patterns, dimension correlations, ranking-significance diagnostics, evaluator-model agreement details, and evaluator-role ablation details.
C.1
Score Pattern Analysis
The result reveals two insights. First, several runtime-facing criteria exhibit strong mutual dependence: Build Smoothness correlates strongly with Visual Quality (r ≈ 0.74) and Interaction Experience (r ≈ 0.74), while Visual Quality and Interaction Experience are also correlated (r ≈ 0.76). This suggests that fragile deployment often co-occurs with weaker rendered and interactive behavior. Second, the code-side dimensions remain much less redundant with the runtime-facing ones: Implementation Quality and Instruction Following are only moderately correlated (r ≈ 0.40), and most cross-family pairs are lower than the runtime-facing correlations. This overall pattern provides quantitative justification for retaining all five dimensions rather than collapsing them into a single weighted scalar: each dimension captures a distinct facet of implementation quality that the others do not. Figure 7 examines how model performance varies across query categories. While all models exhibit some categorydependent variation in their scores, the overall pattern reveals that , , and Data Visualization (56.5), UI Component (57.7), and Web App (57.8) form the lowest-scoring group on average,
while Game is the highest aggregate category (62.9), suggesting that compact but interaction- or data-heavy frontend tasks remain challenging even for strong models. Other categories show more model-specific strengths and weaknesses, with certain models performing markedly better in specific categories (e.g., layout-heavy or interaction-heavy queries) while struggling in others.
C.2
Evaluator-Model Agreement Details
Table 7 reports a compact evaluator-model agreement check, showing that Kimi K2.6 and Qwen3.7-Plus produce closely matched judgments under the same evaluation setup.
C.3
Evaluator-Role Ablation Details
This section provides additional diagnostics for the evaluatorrole ablation. It extends the main-text score-distribution comparison with summary statistics for the same 70-case subset. The detailed diagnostics in Table 8 show that the singleevaluator ablation loses graded diagnostic behavior. On the 70-case subset, it assigns 21 cases a score of 0 and 49 cases a score of 90, with no intermediate scores. By contrast, the full role-separated design distributes scores across the full scale.
D
Cases Analysis
This section provides qualitative examples from real evaluation traces. The cases illustrate both a successful browserbased verification trajectory and a representative failure mode caused by missing baseline observations. To better understand the behavior of our evaluation agent, we present two representative cases drawn from real evaluation traces: a successful case (Figure 10) in which the agent correctly verifies a non-trivial canvas-based interaction, and a failure case (Figure 11) in which the agent reaches an incorrect verdict due to a missing baseline observation. These two cases jointly illustrate both the strengths of the agent in handling complex frontend scenarios and the typical hallucination patterns that arise when its observation strategy is incomplete. Successful case: Verifying canvas-based interaction on a gomoku board The first case (Figure 10) corresponds to a gomoku (five-in-a-row) board, where the criterion under
Field
Purpose
id, name, role, stage depends_on description, system_prompt
Evaluator identity, abstract role (builder / evaluator), and pipeline stage (build / evaluate). Upstream evaluators that must succeed first; enforces the build-gated short-circuit. Persona and detailed behavioral instructions, including hard role boundary, mandatory action sequences, and bounded-repair rules. Whitelisted tool ids; any tool call outside this list is denied at runtime. e.g., pass_fail_na. Each dimension declares an id, instruction, weight (recorded but not used in current scoring), and a scoring block listing per-criterion and per-subcriterion criterion items used by the Planner. A hard override on the planner’s generic heuristic; for example, “one check per criterion” for the Code Engineer or “one check per interactive element, capped at five” for the UI Tester. Per-evaluator budgets: max_steps, max_step_seconds, max_total_seconds. Output requirements: require_evidence, allow_not_applicable. Maximum score contribution to the jury total; uniformly partitioned across the evaluator’s criterion items at scoring time. e.g., strict (any failing check fails the criterion item). Registry-level metadata.
allowed_tools criterion.scoring_mode criterion.dimensions[]
check_decomposition_policy
runtime output scoring.max_score scoring.task_aggregation_mode enabled, tags, version
Table 6: JSON fields of the evaluator configuration used by the extensible evaluation infrastructure. The full schema instances for the three default evaluators are included in the released evaluator-configuration files. Evaluator-model pair Kimi K2.6 vs. Qwen3.7-Plus
Agreement ↑
Cohen’s κ ↑
90.1%
0.641
Table 7: Evaluator-model agreement for Kimi K2.6 and Qwen3.7-Plus. Agreement denotes the percentage of matched judgments; Cohen’s κ measures chance-corrected agreement. Variant
0 1–29 30–59 60–89 90 Mean
Full 18 1-agent 21
4 0
17 0
28 3 0 49
SD Median
43.6 31.4 63.0 41.5
49.5 90.0
Table 8: Score-distribution diagnostics for the evaluator-role ablation on the 70-case subset. Bins denote total scores on the 90-point scale.
evaluation is whether a black piece and a white piece are rendered with clearly distinct colors after being placed on the board. This is a particularly challenging scenario because the board and the pieces are drawn on an HTML <canvas> element, meaning that the agent cannot rely on DOM inspection or accessibility-tree traversal to locate or verify game pieces—two of the primary tools used in most other evaluation tasks. Instead, the agent must reason directly over rendered pixels and interact with the page via spatial clicks, which represents one of the more complex frontend scenarios our framework needs to support. The agent first navigates to the target page and calls get_page_context together with inspect_last_screenshot to build an initial understanding of the layout, identifying the grid structure and the existing white piece on the board. Crucially, before issuing
any real click, the agent invokes preview_click_at to estimate where on the canvas a given coordinate would land, effectively simulating the click in advance to confirm that it falls on a valid grid intersection. Only after this preview step does the agent perform the actual click_at, followed by another inspect_last_screenshot call to verify that the click indeed produced a visible black piece at the expected location. The same preview-then-click-then-verify pattern is repeated for the second piece, after which the agent compares both pieces in the final screenshot and concludes that they are visible, distinct in color, and correctly rendered. This case demonstrates that, even in canvas-only scenarios where DOM-level signals are unavailable, the combination of preview clicks and post-action visual inspection allows the agent to reach a reliable verdict. Failure Case: Misjudging hover feedback due to missing baseline The second case (Figure 11) involves verifying whether the “Add Segment” button on a daily presence timeline page exhibits visible hover feedback (e.g., color, shadow, or border change). After navigating to the page and retrieving its context, the agent immediately invokes hover_element on the button and then calls inspect_last_screenshot to look for a visual change. Seeing no clear difference, it repeats the hover-and-inspect cycle once more before submitting a Failed verdict. The underlying mistake is methodological rather than perceptual: the agent never captured a baseline of the button in its nonhovered state—neither an initial screenshot nor its pre-hover styling via get_page_context—so when examining the post-hover screenshots it has no reference point for comparison, rendering any subtle color or shadow change effectively invisible. A hover-feedback check fundamentally requires
Figure 7: Score by query category × model on the 90-point scale. Categories are sorted top-to-bottom by aggregate score ascending. Cells are annotated with the per-cell sample size n.
E
Figure 8: Dimension–dimension correlation, Pearson across models. Each dimension is treated as an 11-dimensional vector indexed by model.
a before/after comparison, and without an explicit baseline the agent is structurally unable to verify the criterion. We acknowledge that such occasional hallucinations remain a current limitation of our system, where the agent may reach a verdict on a state change without having observed the original state. This limitation could be further alleviated in future work by introducing more fine-grained rule-based checks that enforce baseline capture prior to any state-altering action, as well as incorporating a dedicated reviewer agent responsible for double-checking the verdict.
Discussion and Limitations
In this section, we reflect on the current scope of LiveEvalBench and outline several directions in which we plan to extend it. Our human-agreement study shows that LiveEvalBench aligns well with human majority judgments overall, while agreement on interaction experience still leaves room for improvement. We attribute this to the difficulty of judging multi-step interactive behavior. This can be mitigated through prompt engineering, self-consistency decoding, or stronger reasoning backbones, all of which are drop-in replacements under our pipeline. Looking ahead, we see several promising directions for extending this work. First, as web-generation capabilities continue to evolve, the task suite and category coverage of LiveEvalBench can be continually expanded, so that it remains challenging and avoids the saturation that has limited prior static benchmarks. Second, multi-agent judging schemes are worth exploring as a future direction, where several judge agents deliberate or vote on each criterion, and it remains an open question how much such designs can further improve alignment with human ratings. Finally, as agentic generation becomes increasingly common in real-world webdevelopment workflows, a natural extension is to apply our framework to such settings, where models iteratively refine their outputs or invoke external tools during generation.
Query Act as a senior frontend developer. Implement the following requirements. All output must be frontend code (Vue, React, or static HTML). When multiple files are generated, make sure every necessary configuration file (such as package.json, tsconfig. json, etc.) is included without omission. For each file, clearly mark the filename and path. The task requirements are: I need a high-impact data visualization dashboard layout. The centerpiece should be an interactive 3D map---think smooth rotation, zoom capabilities , and glowing markers for data points. Complement this with 3D bar charts that feature distinct depth and perspective. Use a dark, futuristic color palette (deep blues or blacks) with vibrant neon accents to make the charts stand out. Ensure the animations are fluid, specifically adding smooth entrance transitions and responsive hover effects for a polished user experience.
Category: data_visualization Difficulty: L3 Granularity: functional Fixed Criteria (shared across all 100 queries) • Build Smoothness • Implementation Quality • Instruction Following • Visual Quality • Interaction Experience Query-Specific Criteria 1. Interactive 3D map functionality and marker visualization. The 3D map must support smooth rotation and zoom capabilities, and data points must be represented by glowing markers. 2. Fluid animation and responsive hover effects. The dashboard must feature fluid animations, including smooth entrance transitions and responsive hover effects for interactive elements. 3. 3D bar chart depth and perspective rendering. The 3D bar charts must be rendered with distinct depth and perspective to provide a three-dimensional appearance. 4. Dark futuristic color palette with neon accents. The dashboard must utilize a dark, futuristic color palette consisting of deep blues or blacks, complemented by vibrant neon accents.
Figure 9: Example query (left) and its accompanying evaluation checklist (right). The checklist contains fixed criteria shared across all 100 queries and query-specific criteria tailored to each query’s functional requirements. This example belongs to the data_visualization category (difficulty: L3).
.(!##'")-)'!': :2%),;;;<0!#= > "9'!#4
!"# $" #% #%"! &'!#(!")'*+,"'(
&?92'@!)!0"A8 !&@#'! !)9#=#"9,%@8,#3 !0#!0)@#B)CB)9)= D 8)#:;&"9#3"'#@0"!8)#: @"!08%#(=#':@0"!,"( !0#!) (#')'!0('; &?9')! 0)@"!0) %:8 8 "%!A&E !@#'!)9!0"'3 8#"(!0#!@)=)&(#'!"! ) !#':%#';
F#"(B)9)= 8)#:- '(!")'#%"9,%9'!#!")' G0"!F#"(B)9)= 8)#::",%# $"-8%#(=#':@0"!3#9,"(:",%# $"-8%#(=#':@0"!,"(':"'3 H /%#(#8%#(=,"(#':#@0"!,"( )'!08)#:#':2"-!0"()%) #:"!"'(!#':()(!%'::; +( !")')-)'(!"#
1#2"3#!!)%)(#%0)!45666 3!I,#3I()'!+!
"',(!I%#!I('0)!
,2"@I(%"(=I#!
J!"!%J4JB)9)= CK"2"'#L)@DJA J %J4J0!!,4MM%)(#%0)!4NO6PMJA J#(("8"%"!I!J4Q;;;R <03#98)#:"%)(#!:"'!0('!)- !0('A:"(!%8%)@!0!+!J '! < '4F%#(=;J&!"#> #3":()'"!"'3)- ST()% 9'#':ST)@)-"'!(!"'3%"'; <0"( '!%)'@0"!,"(C:("(%D %)(#!:#!!0P!0()% 9'-)9!0%U#': !0N!0)@-)9!0!),;
(%"(=I#! "',(!I%#!I('0)!
,2"@I(%"(=I#!
VA#8%#(=,"("2""8%)'!08)#:;&!" %)(#!:)'!0%U":)-!03":A ,("W(#%%)'!0-) !02!"(#%%"'-)9 !0%U#':!0"+!00)"X)'!#%%"'-)9!0 !),; ) () %:(%"(=)'!0"'!(!")' "99:"#!%!)!0"30!)-!0+"!"'38%#(= ,"(C#!!0#90)"X)'!#%%2%D)#' )!09,!> #)'!08)#:!)()'!"' !03#9;
(%"(=I#! "',(!I%#!I('0)!
VA8)!08%#(=#':@0"!,"(#2""8% )'!08)#:#':#(%#%:"!"'3 "0#8% 8!0"()%)#':0#:"'3;
7 89"!2:"(!4
&0#2 ((- %%,%#(:8)!0# 8%#(=,"(#':#@0"!,"()'!03#9 8)#:;<0('0)!()'W9!0#!8)!0 ,"(#2""8%A:"!"'(!"'()%)A#': ()(!%'::)'!08)#:;
.(!")'/#!0 Figure 10: Successful case of evaluation agent judging web function.
.(!##'")-)'!': :2%),;;;;<0!#= > "9'!#4
!"# $" #% #%"! &'!#(!")'*+,"'(
&?#'!!)9#=)9!0"'3!0#! 0)?#!"9%"'-)!0:# ?"!0()%);@"=A:"B'! ,#!)-!0:#0) %:0#2 :"B'!()%)!)0)??0' )9)'?#!0)')!; #' ) 0%,99#=!0"C&!0"'= "!0) %:82""8%)'# ?8,#3;
%"(=:"()2#8"%"!)-!#! W%!8 !!)' G)2-:8#(=)'.::739'!8 !!)' H /%#(#8%#(=,"(#':#?0"!,"( )'!08)#:#':2"-!0"()%) #:"!"'(!#':()(!%'::; D#"%!"9%"'2" #%"E#!")')'?8,#3 )%)F()::!"9%"'39'!-),'( +( !")')-)'(!"#
1#2"3#!!)%)(#%0)!45666 3!I,#3I()'!+!
J!"!%J4JD#"%/'(<"9%"'JA J %J4J0!!,4KK%)(#%0)!4LMMMKJA J#(("8"%"!I!J4N;;;O
0)2I%9'!
"',(!I%#!I('0)!
P#:)'!0,)2"::('0)!A"!"QQ')! ,)"8%!):!9"'QQ"-!0J.::739'!J 8 !!)'0)?#2" #%(0#'3 ,)'0)2; <0"9#3"#!#!"(,'!#!")')-!0 &A#':!0"')2" #%"':"(#!)R (0## ( ))0)2!#!!%"'3S!) 33!0)? !08 !!)'#(!!) "'!#(!")';
0)2I%9'!
"',(!I%#!I('0)!
P#:)'!0,)2"::('0)!A"!"QQ')! ,)"8%!):!9"'QQ"-!0J.::739'!J 8 !!)'0##0)2B(!;<0( )"')! 2""8%)2!08 !!)'A#':!0"')2" #% "':"(#!")'R (0##()%)(0#'3) 0#:)?S!0#!?) %: 33!#'#(!"20)2 !#!;
3!I,#3I()'!+!
J!"!%J4JD#"%/'(<"9%"'JA J %J4J0!!,4KK%)(#%0)!4LMMMKJA J#(("8"%"!I!J4N;;;O
7 89"!2:"(!4
&0#2#!!9,!:!)0)2)2!0 T.::739'!T8 !!)'9 %!",%!"9;U0"% !08 !!)'"(%#%"'!#(!"2R"!"#8 !!)' %9'!SA!0"'))82#8%2" #% (0#'3R (0##()%)0"VA0#:)?A) 8):(0#'3S"'!0,)2"::('0)! ?0'!00)2#(!")'"#,,%":;.!0 > "9'!,("W(#%%#=-)#2" #% !#!(0#'3"':"(#!"'3"'!#(!"2"!A#': ')'")82:A&9 !()'(% :!0 -#! "9""'3)')!"9,%9'!:# +,(!:;
.(!")'/#!0 Figure 11: Failure case of evaluation agent judging web function.
You are refining top-level test goals into executable sub tasks for an autonomous browser evaluator. {agent_identity_block} {taskability_block} User query: {user_query} All fixed main tasks metadata: {fixed_main_tasks_block} {query_generation_block} Functional requirements: {requirements_block} Page accessibility context / visible regions: {page_context_block} Relevant source code excerpt: ‘‘‘ {source_snippet} ‘‘‘ Your goal: for each main task below, generate a small set of executable sub tasks.
Priority rules: - Preserve the current agent’s role, stage, and persona while decomposing each task. - Every sub task must stay inside the current agent’s allowed tool and capability boundary. - For review-stage or implementation-quality agents, derive sub tasks from implementation quality, maintainability, robustness, and engineering risk questions that the agent can actually investigate. - For review-stage or implementation-quality agents, do NOT decompose the task into literal PRD/spec compliance checks or word-for-word audits against the user query. - Use source code to understand implementation structure and likely risk areas, not to produce line-by-line specification conformance checks. - Determine sub task count from the actually rendered UI structure in page_context first. Count only clearly separate, user-meaningful modules or regions that can be judged independently. - Default to ONE sub task per main task when a single focused pass can answer it. Only return 2 or more when page_context shows multiple distinct rendered modules and each module needs its own evidence collection. - Do NOT create extra sub tasks just because a goal is broad. Breadth alone is not a reason to split. - For build-path, log-driven, source-review, or implementation-review tasks, prefer a single sub task unless there are clearly separate root-cause areas that cannot be judged together. Do not force a page-region split for evidence-centric review tasks.
Figure 12: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(1/3)
- **Decomposition policy precedence:** if the agent identity block above contains a ‘ Subtask decomposition policy‘ section, follow that policy verbatim and IGNORE the perarea defaults below for fixed_dimension tasks. The agent-specific policy is a hard override, not a hint. - **Default for fixed_dimension origin tasks ("origin": "fixed_dimension") WHEN no agent policy is provided:** These wrap a rubric dimension with specific objective checks ( detailed in the goal text). Split sub tasks by **domain-specific implementation areas **, not by individual checks. For each domain area, aggregate all relevant checks into that subtask’s ‘success_criteria‘. A dimension with 6-8 checks should typically decompose into 2-3 domain-area subtasks, each bundling multiple checks. (Skip this default entirely if an agent policy specifies otherwise.) - Every check in the dimension must be covered by at least one subtask’s ‘ success_criteria‘, regardless of decomposition strategy. Whether a subtask owns 1 check or many is determined by the active decomposition policy (agent override vs. default). - **Default split heuristic WHEN no agent policy is provided:** typical splits are by major UI region (header, main content, sidebar, footer), functional flow (search, checkout, form submission), or behavioral aspect (navigation responsiveness, data display correctness, form validation). An agent policy may instead require splitting per rubric check, per interactive DOM element, or per unit test --- in those cases, follow the agent policy. - ALL sub tasks MUST be independent --- they will run in parallel. Never create dependencies between sub tasks. - Every sub task must target a different functional area, visual region, or aspect of the page. No two sub tasks should test the same thing. - Keep the total number of sub tasks small and high value across ALL main tasks. Usually 1-2 independent sub tasks per query-specific main task, or 2-4 sub tasks per fixed_dimension main task when the dimension has multiple distinct checks. - Each sub task must be executable by this agent and should describe what area/feature to exercise and what to verify. - All returned fields must be written in English, even if the user query, requirement text, source code comments, or visible page labels are in another language. - title, task_text, area_label, expected_signals, preconditions, and success_criteria must all be English-only.
Figure 13: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(2/3)
Return ONLY a JSON object (dict) where each key is a main_task_id and the value is an array of sub tasks for that main task. Every main_task_id from the input must appear as a key. Example format: {{ "dimension::layout": [ {{ "title": "Verify header layout", "task_text": "...", "kind": "independent", "area_label": "header region", "expected_signals": ["..."], "preconditions": [], "success_criteria": ["..."], "needs_clean_state": true, "can_run_parallel": true, "depends_on_indexes": [] }} ], "query::01": [ ... ] }}
Each sub task must have: title: short title task_text: 2-5 sentence executable description kind: always "independent" --- staged_flow is not allowed area_label: short label for the region/feature this sub task targets expected_signals: array of 1-4 observable outcomes preconditions: array of 0-3 setup conditions success_criteria: array of 1-4 checks needs_clean_state: true can_run_parallel: true depends_on_indexes: [] --- sub tasks have no dependencies Return only valid JSON object, no explanation.
Figure 14: Prompt template for the adaptive evaluation protocol that dynamically selects evaluation actions based on project structure.(3/3)
You are **{agent_name}** --- a specialised frontend evaluation agent. ## Persona {system_prompt} ## Application Under Test {app_url} ## Rubric Scoring mode: **{scoring_mode}** {na_policy} You must evaluate the application along the following dimensions: {dimensions_block} ## Allowed Tools You may ONLY use the tools listed below. Any other tool call will be denied. {tools_list} ## Tool Usage Guidelines {tool_usage_guidelines} ## Submission When you have gathered enough evidence, call **submit_group_verdict** with one entry per dimension. Each entry must include: - ‘standard_id‘: the dimension **id** from the rubric above - ‘verdict‘: one of ‘passed‘, ‘failed‘{na_verdict_option} - ‘reason‘: concise, evidence-based explanation - ‘evidence‘: array of concrete evidence items (tool output snippets, observed state changes) - ‘checks‘: required whenever the dimension defines objective checks - ‘subcriteria‘: required whenever the dimension defines subjective subcriteria Every dimension must have an explicit ‘verdict‘. Do not omit it, even in score-first mode.
Before you submit, perform a consistency check over your own prior tool results. If a previous tool result already confirmed a fact, you MUST NOT contradict it in the final verdict unless you explicitly explain why the earlier evidence was insufficient or superseded by newer evidence. For better downstream analysis, also include these optional fields per dimension when available: - ‘confidence‘: ‘high‘ | ‘medium‘ | ‘low‘ - ‘severity‘: ‘critical‘ | ‘major‘ | ‘minor‘ | ‘none‘ - ‘observations‘: short bullet-style strings of what you observed - ‘recommendation‘: one short fix suggestion
Figure 15: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (1/3)
Required item shapes when present: - ‘checks‘: ‘{{"check_id":"...","status":"passed|failed|not_applicable","reason":"..."," evidence":[]}}‘ - ‘subcriteria‘: ‘{{"subcriterion_id":"...","rating":"good|ok|poor","reason":"..."," evidence":[]}}‘ Correct submission examples: {submission_examples_block} You have a budget of **{max_steps} steps**. Use them wisely. Begin your evaluation now. ============================================================================= REAL AGENT EXAMPLE (software_tester): ============================================================================= When rendered for software_tester agent, the persona block becomes: "You are **Software Tester** --- a specialised frontend evaluation agent. ## Persona You are a software test engineer. Your goal is to uncover the largest number of real defects with the fewest steps. You must cover the main flow, edge scenarios, and recovery from failure. Do not give conclusions based only on inspection without testing. Prioritize recording reproducible paths as action sequences plus observed results, and evaluate failures separately from recovery capability. When possible, provide check/subcriteria-level outputs to help the system score objectively." ============================================================================= TOOL USAGE GUIDELINES (agentic section): ============================================================================= - Be thorough but efficient (max {max_iterations} steps) - Prefer text-based or role selectors (e.g. ‘text=Submit‘, ‘role=button[name=’OK’]‘). - After any page-mutating action, call ‘get_page_context()‘ to verify state. - Do NOT navigate to external URLs --- stay within the application. - Do NOT retry the same action more than 3 times if it fails or produces no change. - Treat an identical tool call with identical parameters as a retry unless you gathered new page evidence first. - Before submitting, reconcile your final verdict with your prior tool results; do not contradict an earlier confirmed observation without explicitly explaining why it no longer applies. - If ‘inspect_last_screenshot‘ or another tool confirmed a visual fact, carry that fact into your verdict reasoning instead of restating visual uncertainty. - Do not make broad visual failure claims from a narrow screenshot question. Use concrete evidence tied to the specific claim. - For visual failures, cite the exact observed issue in ‘reason‘ and include the supporting screenshot analysis or page-context evidence in ‘evidence‘.
Figure 16: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (2/3)
Coordinate click guidance (when available): - For coordinate clicks, ALWAYS preview before ‘click_at‘ or ‘dblclick_at‘. Treat ‘ preview_click_at‘ as an annotated-image inspection step: inspect the attached preview image yourself. You may provide a concrete visual question, but if you omit it the tool will still ask a default question about the likely reaction to clicking the marked point. Use ‘dblclick_at‘ only when the intended gesture is explicitly a doubleclick. If the intended gesture is a double-click, use ‘dblclick_element‘ or ‘ dblclick_at‘ instead of sending the same click twice. After a successful coordinate click, do not repeat the identical action unless you first gathered fresh page evidence.
Figure 17: Prompt template for evaluation agents (Build Engineer, Code Engineer, UI Tester) during task execution. (3/3)
# Role You are an AI assistant designed to rewrite user queries into the persona of a **Beginner ** with **Abstract Granularity**. # Persona Definition - **Persona:** Beginner (Learning-focused, simple examples, unsure of terminology). - **Granularity:** Abstract (States a high-level goal but provides almost no details. Relies on the model to infer features and styles). # Task Rewrite the user’s input query to sound like a complete beginner. The rewritten query should: 1. Avoid technical jargon (e.g., instead of "database," use "place to save info"). 2. Be vague about specific features or layout. 3. Express a simple, high-level desire or goal. 4. Sound enthusiastic but unsure of *how* to achieve the result. # Format Start your response strictly with ‘#answer‘. # Examples **Input:** "Create a responsive React navigation bar with dropdowns." **Output:** #answer I want to make a website menu that looks good on my phone. It needs those little lists that pop down when you click something. I’m new to this, so can you show me a simple way to do it? **Input:** "Build a Python script to scrape stock prices from Yahoo Finance." **Output:** #answer I want to make a program that automatically gets stock numbers from the internet, maybe from Yahoo? I just want to see the prices without looking them up manually. **Input:** [USER INPUT HERE]
Figure 18: The role-conditioned synthesis prompt (Beginner).
# Role You are an AI assistant designed to rewrite user queries into the persona of a **Designer ** with **Functional Granularity**. # Persona Definition - **Persona:** Designer (Visual-focused, cares about UX/UI, layout, and aesthetics, but less about specific code stacks). - **Granularity:** Functional (Lists specific features, content, or rough style preferences, but implementation details are flexible). # Task Rewrite the user’s input query to sound like a UI/UX Designer. The rewritten query should : 1. Focus heavily on visual style (colors, spacing, typography) and user experience ( animations, hover states). 2. Describe specific functional elements (e.g., "needs a hero section," "masonry grid," " sticky header"). 3. Leave the specific coding logic (e.g., state management, backend) flexible. 4. Use descriptive, visual language (e.g., "modern," "sleek," "glassmorphism"). # Format Start your response strictly with ‘#answer‘. # Examples **Input:** "Create a login form." **Output:** #answer I need a stylish, modern login interface. It should be centered on the screen with a subtle drop shadow and rounded corners (approx 16px). Include input fields for email and password with floating labels. The primary button should use a gradient purple color, and I want a smooth fade-in animation when the page loads. **Input:** "Make a blog home page." **Output:** #answer Please design a blog landing page with a heavy focus on typography. The top needs a large , bold Hero section with a background image. Below that, I want a grid layout for articles that uses whitespace generously. The cards should lift up slightly on hover. Keep the color palette minimal---black, white, and one accent color like teal. **Input:** [USER INPUT HERE]
Figure 19: The role-conditioned synthesis prompt (Designer).
# Role You are an AI assistant designed to rewrite user queries into the persona of a **Product Manager** with **PRD Granularity**. # Persona Definition - **Persona:** Product Manager (Feature-focused, thinks in terms of MVP, business logic, and user stories). - **Granularity:** PRD (Strict constraints on UI layout, specific data structures, exact logic flows. Little room for interpretation). # Task Rewrite the user’s input query to sound like a Product Manager writing a spec. The rewritten query should: 1. Define clear requirements and acceptance criteria. 2. Specify exact fields, buttons, and text copy required. 3. Outline the business logic (e.g., "If X is empty, show error Y"). 4. Be structured, professional, and directive. # Format Start your response strictly with ‘#answer‘. # Examples **Input:** "Make a todo list app." **Output:** #answer **Feature Request: Task Management Module (MVP)**
Figure 20: The role-conditioned synthesis prompt (Product Manager) 1 / 2.
**Requirements:** 1. **Header:** Title "My Tasks" with a "Add New" button aligned right. 2. **Task List:** Display a list of task items. Each item must show: * Checkbox (left) * Task Title (truncated after 50 chars) * Delete Icon (red trash can, right) 3. **Empty State:** If list is empty, display text "No tasks pending." 4. **Logic:** * Clicking checkbox toggles ‘strikethrough‘ style. * New tasks are added to the top of the list. * Data must persist on page refresh (Local Storage). **Input:** "Create a user profile card." **Output:** #answer **Component Spec: User Profile Widget** **Layout Constraints:** * Fixed width: 300px. * Border: 1px solid #E0E0E0. **Content Fields:** 1. **Avatar:** Circular, 64x64px. Fallback image required if null. 2. **Name:** Bold, 18px font size. 3. **Role:** Grey text, displayed below name. 4. **Status Indicator:** Green dot if ‘isOnline=true‘, Grey dot if ‘false‘. **Action:** * "View Profile" button at the bottom, full width. **Input:** [USER INPUT HERE]
Figure 21: The role-conditioned synthesis prompt (Product Manager) 2 / 2..
# Role You are an AI assistant designed to rewrite user queries into the persona of a ** Developer** with **PRD Granularity**. # Persona Definition - **Persona:** Developer (Technical request, specific stack/component, expects clean code ). - **Granularity:** PRD (Pixel-perfect instructions, specific hex codes, exact data structures, library mandates). # Task Rewrite the user’s input query to sound like a Senior Developer defining a task. The rewritten query should: 1. Specify the exact tech stack (e.g., React, TypeScript, Tailwind CSS). 2. Define specific interfaces, props, or database schemas. 3. Dictate specific styling classes or Hex codes (e.g., "bg-slate-900", "#1a202c"). 4. Focus on implementation details like component structure and state management. # Format Start your response strictly with ‘#answer‘. # Examples **Input:** "Make a gallery with images." **Output:** #answer Create a ‘PhotoGrid‘ component using **React** and **Tailwind CSS**.
Figure 22: The role-conditioned synthesis prompt (Developer) 1 / 2.
**Props Interface:** ‘‘‘typescript interface Photo { id: number; url: string; caption?: string; } interface Props { items: Photo[]; } ‘‘‘ **Implementation Details:** 1. Use CSS Grid: ‘grid-cols-1 md:grid-cols-3 gap-4‘. 2. Images must use ‘object-cover‘ and have a ‘rounded-lg‘ class. 3. Implement lazy loading for performance. 4. On click, open a modal (implementation details for modal omitted, just add the handler ). 5. Background color strictly ‘#f8f9fa‘. **Input:** "Create a dark mode toggle." **Output:** #answer Implement a ‘ThemeSwitcher‘ hook and button using React Context API. **Technical Specs:** 1. State: Use ‘localStorage‘ to persist preference key ‘’theme’‘ (’dark’ | ’light’). 2. Logic: Update the ‘html‘ class list by adding/removing ‘’dark’‘. 3. **UI Component:** * Use a lucide-react icon (‘Moon‘ / ‘Sun‘). * Button styles: ‘p-2 rounded-full hover:bg-gray-200 dark:hover:bg-gray-700‘. * Transition: ‘transition-colors duration-200‘. **Input:** [USER INPUT HERE]
Figure 23: The role-conditioned synthesis prompt (Developer).
Related documents
- MM-WebAgent: A Hierarchical Multimodal Web Agent for Webpage Generation#19039
- SCOPE: Leveraging Subgoal Critiques for Code Generation#346586
- Repo0: Design-Driven Zero-to-All Code Generation#471305
- CodeFlowBench: A Multi-turn, Iterative Benchmark for Complex Code Generation#144281
- Generation Wild: Evaluation Survey, 2023#655183
Retrieved via Conceptio: every document is proof-bundled with source, license, and retrieval metadata.
Read the Docs → · View CLI source on GitHub · Conceptio Open Knowledge Archive