JoyFrontBench: Evaluating Front-End Coding Agents through Goal-Oriented Acceptance Contracts

JoyFrontBench: Evaluating Front-End Coding Agents through Goal-Oriented Acceptance Contracts Xiaoyi Qiao∗ Yicheng Wang∗ Jianan Lin∗ Xiaoyang Ji Dongyi Lv Yidan Huang Shi Suo Shiying Fan Nan Duan† Jiaqi Wang† JD.com

arXiv:2611.14952v1 [cs.SE] 11 Nov 2026

Abstract Front-end coding agents are expected to produce applications that fulfill user goals, yet high task scores can coexist with failures in essential workflows. Evaluating these agents therefore requires an explicit distinction between partial implementation and application success. We introduce J OY F RONT B ENCH, a benchmark that makes this distinction through goal-oriented acceptance contracts. Each contract identifies core tasks and mandatory criteria before generation: a core task passes only when all mandatory criteria are satisfied and a shared completion threshold is met, while application success requires every core task to pass on the same operational artifact within a single run. Derived from practitioner requirements, J OY F RONT B ENCH comprises 100 front-end development problems, 764 tasks, and 2,596 atomic criteria, with reviewed annotations linking user goals to core tasks and evidence requirements. The evaluation framework combines evidence-grounded judgments with deterministic record auditing and fixed scoring rules, preserving graded task assessment alongside explicit core-task and application-level acceptance. We evaluate five coding models under two agent harnesses, OpenHands and OpenCode, on the 70-problem test set; the remaining 30 problems form a public set for validation and diagnostic analyses. Under OpenHands, the mean Average Task Score (ATS) across models is 87.69%, while the mean Application Success Rate (ASR) is 51.52%. Further analyses reveal changes in model rankings across harnesses and variation in application success across repeated runs. These findings highlight the limitations of aggregate task scores in characterizing application success and motivate explicit assessment of core-goal fulfillment alongside graded task performance.

1

Introduction

Front-end coding agents translate natural-language requirements into interactive applications. Existing benchmarks assess visual fidelity, functional behavior, and multi-step workflows (Si et al., 2025; Lu et al., 2025; Zhu et al., 2025; Liu et al., 2026; Meng et al., 2026; Tran et al., 2026). These assessments characterize individual aspects of an application; determining whether the application achieves its intended user goal also requires specifying which requirements must hold jointly. For example, an application may support search, filtering, and editing yet lose saved changes after a reload. Its aggregate task score can remain high even when the persistence failure prevents completion of the primary workflow. Goal-oriented requirements engineering links system requirements to the objectives they serve (van Lamsweerde, 2001), while requirements research addresses differences in priority and dependency (Mougouei and Powers, 2017; Berntsson Svensson and Torkar, 2024). These distinctions motivate an acceptance rule that preserves partial credit while requiring the conditions essential to the user goal. We introduce J OY F RONT B ENCH, a front-end application benchmark with goal-oriented acceptance contracts. It comprises 100 problems derived from interviews with more than 30 practitioners, with 764 rubric tasks and 2,596 atomic criteria. Each problem pairs a generation specification with an evaluation rubric linking its primary user workflow to core tasks, mandatory criteria, and validation protocols. Specifications and rubrics undergo three-person review, and reference implementations are ∗ †

Equal contribution. Corresponding authors.

checked along their primary workflows. The benchmark is partitioned into a 70-problem test set and a 30-problem public set. A core task passes when all mandatory criteria are satisfied and its completion score meets a shared threshold. Application success requires every core task to pass on the same operational artifact. The public set supports threshold selection and diagnostic analyses; the acceptance rule and reporting choices are fixed before test-set evaluation. Evidence-grounded judgments undergo record auditing and deterministic scoring. We rank systems by Application Success Rate (ASR), while retaining Average Task Score (ATS) and Core-Task Pass Rate (CTPR) to characterize partial completion and individual core-task acceptance. We evaluate five coding models under OpenHands and OpenCode. On the test set under OpenHands, mean ATS, CTPR, and ASR across models are 87.69%, 80.19%, and 51.52%, respectively. High task scores thus coexist with substantially lower application success. Averaged across models, 48.6% of the 70 test problems yield both successful and unsuccessful outcomes over three OpenHands runs. Model rankings also change across harnesses. These results characterize distinct aspects of the evaluated systems: graded task performance, joint application acceptance, and consistency across repeated runs. Our contributions are: • A benchmark grounded in practitioner requirements. We construct 100 front-end application problems with reviewed specifications and workflow-aligned rubrics that connect user goals to verifiable requirements. • Goal-oriented acceptance contracts. We combine mandatory criteria with completion thresholds to define application success on a single operational artifact, alongside graded scores for diagnosis. • An empirical characterization of application success. Across five coding models and two harnesses, we examine the relationship between task scores and application acceptance, repeated-run variability, and changes in model rankings across harnesses.

2

Related Work

Front-End Development Benchmarks. Design2Code evaluates screenshot-to-code generation by comparing rendered pages with reference designs (Si et al., 2025). WebUIBench studies interface reconstruction alongside webpage perception, HTML programming, and correspondence between interfaces and code (Lin et al., 2025). FullFront covers webpage design, perception, and code generation. Its generation tasks include producing code from images or text, adding interactions, and refining existing code (Sun et al., 2025). Functional benchmarks examine how generated applications respond to user actions. WebGen-Bench specifies actions and expected outcomes for browser-executed tests and assesses website appearance separately (Lu et al., 2025). FrontendBench spans tasks ranging from basic components to complex interactive features and tests their behavior with predefined interaction scripts (Zhu et al., 2025). Vibe Code Bench evaluates complete applications built from natural-language specifications. Its workflows span multiple user actions and include external-service integrations, allowing evaluation to follow a user request through a sequence of steps (Tran et al., 2026). J OY F RONT B ENCH shares this focus on application-level workflows and derives its problems from requirements elicited through practitioner interviews. Reviewed specifications and rubrics link the primary user goal to testable tasks and criteria, making explicit both what the application should achieve and how it will be assessed. Evaluation Methods for Front-End Applications. Methods for front-end evaluation draw on source code, rendered interfaces, and runtime interaction. ArtifactsBench combines source code and temporal screenshots with task-specific checklists to assess the appearance and dynamic behavior of visual artifacts (Zhang et al., 2025). WebDevJudge studies the reliability of automated judges by comparing their judgments with human preferences over pairs of web implementations. It evaluates judges under static and interactive settings, with structured rubric trees guiding human annotation (Li et al., 2025). LiveEvalBench distributes evaluation across build, code, and browser evaluators, gathering evidence from deployment, source inspection, and live interaction. Shared rubrics support comparison across implementations, while implementation-specific checks accommodate different ways of realizing a request (Wang et al., 2026). 2

How individual judgments are combined also determines what a benchmark score represents. WebCoderBench aggregates rule-based and model-based quality metrics using weights derived from human preferences (Liu et al., 2026). WebRISE maps each requirement to states, transitions, and assertions. A requirement counts as satisfied only if all mapped checks pass; the benchmark then reports requirement coverage (Meng et al., 2026). Vibe Code Bench applies a 90% substep-success threshold to each workflow and reports the fraction of passing workflows (Tran et al., 2026). J OY F RONT B ENCH complements measures of task quality and completion with an explicit check of whether the application’s core goals are met together. Core tasks and mandatory criteria are fixed before generation; an application is accepted only if every core task meets all its mandatory criteria and a shared completion threshold on the same operational artifact within one run. High scores on other tasks therefore cannot compensate for a failed core task, while graded task scores still capture partial completion and support analysis of unmet requirements.

3

Method

J OY F RONT B ENCH separates graded task performance from application acceptance through goaloriented acceptance contracts. Each contract specifies core tasks, mandatory criteria, and validation requirements. Evaluation collects evidence-linked judgments, audits their records, and applies deterministic scoring rules (Figure 1).

Figure 1: Overview of J OY F RONT B ENCH. A skill-guided judge records evidence-linked judgments, a deterministic auditor checks record completeness and evidence admissibility, and a model-free scorer computes Average Task Score (ATS), Core-Task Pass Rate (CTPR), and Application Success Rate (ASR). Audit findings trigger up to three targeted remediation rounds before scoring. 3.1

Benchmark Construction

Acceptance contracts. Annotation starts from the primary user workflow: the interactions and resulting states needed to achieve the stated goal, following the goal-oriented view of requirements (van Lamsweerde, 2001). Requirements are decomposed into tasks and atomic criteria, each with a validation protocol specifying preconditions, actions, expected observations, and admissible evidence. A criterion is mandatory if it verifies the operability of a required step or the correctness of its committed state, including persistence after reload when specified. Pointwise tasks containing mandatory criteria are designated as core tasks. Pairwise comparisons against a reference artifact assess subjective presentation quality and contribute to graded task scores; core-task acceptance uses pointwise criteria. 3

Rule Simulation

38 28 42 234 91

764 rubric tasks

105 118 108

234 (30.6%)

State Management

118 (15.4%)

Computational Rendering

108 (14.1%)

Component Completeness

105 (13.7%)

Visual Quality

91 (11.9%)

Representation Fidelity

42 (5.5%)

Interaction Responsiveness

38 (5.0%)

Data Exchange

28 (3.7%)

Figure 2: Distribution of the 764 tasks across eight capabilities. Each task has one capability label; definitions appear in Table 4. Sector labels show counts; the legend reports counts and percentages rounded to one decimal place. Collection and review. We elicit requirements through interviews with more than 30 practitioners in front-end, full-stack, and broader software development, as well as data analysis. Model-assisted rewriting removes organization-specific information and private-service dependencies. Two reviewers independently assess specification clarity, rubric consistency, criterion atomicity, and observability; a third resolves disagreements. Reference implementations are checked along their primary workflows. After resolving review findings, we fix the specifications, rubrics, core-task designations, validation protocols, and operational requirements before generation. Appendix A details construction and annotation. Coding agents generate their own front-end implementations and mock data subject to the requirements stated in each problem specification, including any specified constraints on the data. Evaluation rubrics and reference artifacts are withheld during generation. The benchmark contains 100 problems, 764 tasks, and 2,596 atomic criteria, with 11–52 criteria per problem and one capability label per task (Figure 2). The 70-problem test set supports the main evaluation. The disjoint 30-problem public set supports selection of the shared completion threshold and reporting rules, together with diagnostic and human-reference analyses. These choices are fixed before test-set evaluation. Specifications, source scenarios, and derived evaluation artifacts are disjoint across the two sets. 3.2

Evaluation Procedure

Controlled generation. Each run starts in a fresh Docker workspace under OpenHands (Wang et al., 2025) or OpenCode (OpenCode Contributors, 2026). Adapters standardize model access, inference settings, dependency and network policies, browser runtime, and outer time and step budgets. Each harness retains its native agent loop, context management, and stopping behavior; the evaluated system is therefore a model–harness pair (Kapoor et al., 2025; Vats and Golev, 2026). Evaluation uses a separate container with the submitted workspace, specification, and scoring policy mounted read-only. Evidence-grounded judging. A GPT-5.5 evaluator with high reasoning effort follows versioned instructions and problem-specific validation protocols. Build and runtime checks provide evidence of operability. Playwright supports interaction recording and collection of screenshots, application state, files, network records, and temporal traces. Behavioral judgments require execution evidence; if launch fails, only criteria predefined as source-verifiable may retain supported credit. Each pointwise criterion receives a Pass/Fail verdict. Each pairwise task receives one Good/Same/Bad verdict under matched viewports, initial states, interaction paths, and observation times. All verdicts include rationales and evidence links; pairwise judgments require evidence from both artifacts. Deterministic auditing. The auditor checks identifiers, rubric coverage, evidence availability, and evidence-type admissibility. Defects trigger up to three rounds of targeted evidence collection and re-evaluation on the unchanged artifact. Revisions require new evidence, and the full history is retained. Unresolved missing, malformed, or unsupported outcomes receive zero only for the affected criteria or pairwise tasks. The judge supplies criterion verdicts, the auditor checks their supporting

4

records, and a deterministic scorer aggregates the retained results. Appendix C gives the record contract and audit workflow. 3.3

Acceptance Rules and Metrics

Core-task and application acceptance. For problem q, let Tq denote its tasks and Cq ⊆ Tq its core tasks. A pointwise task t has predefined criteria Jqt ; a core task additionally has mandatory criteria Mqt ⊆ Jqt . In run r, let yqrtj = 1 if a criterion’s Pass verdict is retained after auditing and 0 otherwise. Task completion is 1 X Aqrt = yqrtj , |Jqt | j∈Jqt

with all predefined criteria included in the denominator. Using τ = 0.8, selected on the public set, acceptance is defined as Y Pqrt = I[Aqrt ≥ τ ] yqrtj , (1) j∈Mqt

Sqr = Bqr

Y

Pqrt ,

(2)

t∈Cq

where Bqr = 1 only when build and runtime checks pass with a rationale and admissible evidence retained after auditing; otherwise, Bqr = 0. A core task must satisfy every mandatory criterion and meet the completion threshold. Application acceptance requires operability and acceptance of every core task on the same artifact within a single run. Failed operability sets Sqr = 0, while supported source-verifiable criteria may still contribute to ATS and CTPR. Scores and aggregation. For pointwise tasks, Zqrt = Aqrt . For pairwise tasks, Good, Same, and Bad map to Zqrt ∈ {1, 0.5, 0}, respectively; unresolved invalid outcomes receive zero. For a fixed model–harness pair, let Q be the evaluated split and R = 3 the scheduled runs per problem. Summing over all q ∈ Q and r ∈ {1, . . . , R}, we report 1 X 1 X ATS = Zqrt , |Q|R q,r |Tq | t∈Tq

1 X 1 X CTPR = Pqrt , |Q|R q,r |Cq |

(3)

t∈Cq

ASR =

1 X Sqr . |Q|R q,r

Problems receive equal weight after averaging their scheduled runs. ATS captures pointwise completion and pairwise presentation quality; CTPR measures core-task acceptance; ASR measures application acceptance, including operability. We rank systems by ASR and retain ATS and CTPR for diagnosis. Capability scores average each task over runs, then weight tasks within each capability dimension equally. Missing and invalid evaluations. Infrastructure failures are rerun. Unresolved missing runs or records invalidated as a whole receive zero for all three metrics and remain in the scheduled denominator. Every problem must have a nonempty core-task set for CTPR and ASR to be defined. Appendix D provides per-run definitions, conditional failure analyses, and threshold sensitivity.

4

Experiments

4.1

Experimental Setup

We evaluate five coding models under two agent harnesses, OpenHands (Wang et al., 2025) and OpenCode (OpenCode Contributors, 2026). OpenHands is the reference harness for the main evaluation; OpenCode supports cross-harness comparisons. Appendix B provides model details and shared execution controls. 5

J OY F RONT B ENCH contains 100 problems: a 70-problem test set and a disjoint 30-problem public set. The test set supports the main leaderboard, cross-harness comparisons, repeated-run analysis, and capability-level evaluation. The public set supports validation and auxiliary analyses of application failures, representative cases, threshold sensitivity, and judge–human agreement. We schedule three independent generation-and-evaluation runs per model–harness–problem condition, with a five-hour generation budget and up to 500 harness-native steps or iterations per run. Evaluation allows up to three targeted remediation rounds. Scoring and aggregation are detailed in Appendix D. 4.2

Overall Benchmark Performance

Table 1 reports task scores, core-task acceptance, and application success, together with harness comparisons and repeated-run outcomes. Performance (OH)

Harness Comparison

Repeatability (OH)

Model

ATS

CTPR

ASR

OC ASR

∆ ASR

All pass

Mixed

All fail

Qwen3.8 Max (Aug. 2026) Kimi K3 GPT-5.6 Sol DeepSeek-V4-Pro-0813 GLM-5.3

89.56 88.98 90.53 85.77 83.63

83.02 82.62 82.34 80.16 72.82

57.62 56.67 55.71 47.62 40.00

56.67 45.71 46.67 35.71 49.05

+0.95 +10.95 +9.05 +11.90 −9.05

38.6 20.0 40.0 21.4 14.3

37.1 65.7 37.1 51.4 51.4

24.3 14.3 22.9 27.1 34.3

Mean

87.69

80.19

51.52

46.76

+4.76

26.9

48.6

24.6

Table 1: Main evaluation results on the 70-problem test set, ordered by OpenHands (OH) ASR. Performance and harness scores (%) average three scheduled runs per problem, then macro-average across problems (τ = 0.8). ∆ ASR is OH minus OpenCode (OC) in percentage points, computed before rounding. OH repeatability (%) uses 70 model–problem conditions per model, each with three scored runs (210 runs per model). The mean row averages five models. Bold marks the highest performance scores and all-pass share. Overall performance. On the test set, the five models achieve a mean ATS of 87.69% and a mean ASR of 51.52% under OpenHands. High task scores thus coexist with substantially lower application success. GPT-5.6 Sol achieves the highest ATS (90.53%) but ranks third in ASR (55.71%), while Qwen3.8 Max (Aug. 2026) has the highest ASR (57.62%). The top three ASR point estimates differ by less than two points. CTPR measures acceptance of individual core tasks, complementing the joint application acceptance measured by ASR (Section 3.3). Harness sensitivity. Test-set performance varies across agent harnesses. The OpenHands– OpenCode ASR difference ranges from −9.05 to +11.90 percentage points. GLM-5.3 scores 49.05% under OpenCode versus 40.00% under OpenHands, ranking second under OpenCode but fifth under OpenHands. Kimi K3 and DeepSeek-V4-Pro-0813 instead score 10.95 and 11.90 points higher under OpenHands, respectively. These results characterize the complete model–harness configuration. Run-to-run variability. Each model has three scored OpenHands runs for each of the 70 test problems (210 runs per model). The share of model–problem conditions with both successful and unsuccessful outcomes averages 48.6% across models. Kimi K3 has the highest mixed-outcome share (65.7%), while GPT-5.6 Sol has the highest all-success share (40.0%). Similar aggregate ASR values can thus accompany different repeated-run outcomes; these runs include both generation and evaluation (Appendix E.4). Capability-wise analysis. Figure 3 reports scores across eight capabilities on the test set under OpenHands. Visual Quality is the lowest-scoring dimension for every model (cross-model mean 70.46%), compared with 95.31% for Interaction Responsiveness. GPT-5.6 Sol achieves the highest Visual Quality score (83.08%). These task-level diagnostics complement application-level success; category-level ASR is in Appendix E.1.

6

en Qw

3.8

A x( Ma

6) 02 .2 g u iK K im

3 G

l So .6 5 PT

De

e Se ep

k-V

3 81 o -0 r P 4GL

5.3 M-

Component Completeness

88.44

90.67

88.89

86.22

87.56

Representation Fidelity

94.44

94.44

96.67

93.33

83.33

Visual Quality

73.85

72.31

83.08

59.49

63.59

Interaction Responsiveness

98.77

92.59

95.06

96.30

93.83

Computational Rendering

92.64

91.34

91.77

88.74

86.15

Rule Simulation

92.42

91.42

92.61

88.82

84.63

State Management

90.48

91.27

89.29

90.87

88.49

Data Exchange

98.33

95.00

86.67

91.67

93.33

50

60

70

80

90

100

Capability score (%)

Figure 3: Capability scores (%) on the 70-problem test set under OpenHands. Cell labels give mean task scores; colors encode absolute values on a 50–100 scale. Full scores and task counts are in Table 8 (Appendix E.2). 4.3

Understanding Application Failures

On the public set, 32.3% of unsuccessful OpenHands runs with valid criterion judgments achieve at least 90% ATS. These runs show how high task scores can coexist with failures in the core workflow. Failure mechanisms. Table 2 characterizes mandatory-criterion and completion failures among all unsuccessful OpenHands runs on the public set with valid criterion judgments. Mandatory-only failures account for 54.4% of valid failed runs, compared with 28.6% completion-only and 17.1% combined failures. The mandatory-only share ranges from 43.2% for Qwen3.8 Max (Aug. 2026) to 63.0% for GLM-5.3. Model

Mandatory only

Completion only

Both

Qwen3.8 Max (Aug. 2026) Kimi K3 GPT-5.6 Sol DeepSeek-V4-Pro-0813 GLM-5.3

43.2% 53.8% 52.5% 55.3% 63.0%

37.8% 33.3% 22.5% 29.8% 22.2%

18.9% 12.8% 25.0% 14.9% 14.8%

All models

54.4%

28.6%

17.1%

Table 2: Failure composition among unsuccessful OpenHands public-set runs with valid criterion judgments (%). “Both” indicates a mandatory-criterion failure in one core task and a completion failure in another core task whose mandatory criteria pass. The final row pools valid failed runs across models; rows may differ from 100% by rounding. The three categories cover the valid failed runs analyzed. Mandatory-criterion failures arise when a required workflow step or committed-state condition is not satisfied, for example, when an application accepts invalid input or fails to synchronize dependent views. A completion failure occurs when a core task satisfies its mandatory criteria but falls below the completion threshold. These mechanisms correspond to the two conditions in the core-task acceptance rule.

7

Representative cases. Two public-set artifacts illustrate these mechanisms. In Q20, a financial dashboard achieves 90.48% ATS but fails critical transfer validation and state consistency: a negative input is silently converted to a positive amount and reaches confirmation, while a valid transfer does not update the budget view. Failed mandatory criteria give Sqr = 0 despite high graded completion. In Q29, a Scalable Vector Graphics (SVG) animation satisfies all six mandatory criteria, but two core tasks score only 75% and 50%, below the 80% completion threshold. The arms lack natural taper, and struck drum surfaces do not respond in synchrony with the sticks. The application thus has Sqr = 0 despite 87.50% ATS. Q20 exposes critical business-rule and cross-view state failures, whereas Q29 exposes incomplete core-task behavior. Table 3 contrasts their per-run scores, and Figure 4 shows the artifacts. Appendix D.3 provides detailed evidence; Appendix E.5 gives additional cases. Table 3: Audited single-run results from the public set. ATS, core-task pass fraction, and mandatorycriterion pass rate are percentages; Sqr is the binary application-success indicator. Case

Artifact

ATS

Core pass (%)

Sqr

Mandatory pass (%)

A B

Q20 financial dashboard Q29 octopus drummer

90.48 87.50

50.00 33.33

0 0

75.00 100.00

(a) Q20: failed transfer validation

(b) Q29: unmet completion rule

Figure 4: Audited public-set cases with Sqr = 0. (a) Q20: an invalid negative input leads to a $5.00 confirmation (failed mandatory criterion). (b) Q29: ride-cymbal contact frame; lack of synchronized response is established by runtime samples in Appendix D.3 (incomplete core task). 4.4

Evaluation Reliability and Threshold Sensitivity

Given reported disagreement between model-based web evaluators and human preferences (Li et al., 2025), we compare automated judgments with an adjudicated human reference on the 30-problem public set. Two evaluators independently annotate the artifacts under a shared protocol, with a third adjudicating disagreements. The automated judge has no access to these human labels. The study compares 726 pointwise criteria and 15 pairwise tasks (Appendix F). The pointwise comparison yields 83.47% exact agreement, Cohen’s κ = 0.651, and Macro-F1 of 0.825. For problem-level scores aggregated from pointwise judgments, Pearson r = 0.807 and Spearman ρ = 0.766. The root-mean-square error normalized by the standard deviation of the human scores (RMSE/SD) is 0.631, and mean error is +0.0362. Label agreement measures consistency of criterion verdicts; correlations measure association between scores, while RMSE/SD and mean error quantify their differences. Appendix F gives the full results and metric definitions. Threshold sensitivity. On the public set under OpenHands, mean ASR falls from 58.22% at τ = 0.75 to 50.89% at τ = 0.80 and 44.89% at τ = 0.90. At τ = 0.80, the finite number of criteria means that 86.6% of core tasks require every criterion to pass. The threshold determines how many non-mandatory criteria may fail; every mandatory criterion must pass at all thresholds. We use τ = 0.80 for the primary acceptance rule; full sensitivity results are in Appendix D.4.

8

5

Conclusion

J OY F RONT B ENCH evaluates application success through goal-oriented acceptance contracts. Mandatory criteria and completion thresholds define core-task acceptance, and all core tasks must pass on the same operational artifact. The resulting metrics connect graded task performance, individual core-task acceptance, and joint application success. Experiments on the 70-problem test set show that high task scores coexist with substantially lower application success, while repeated runs and different harnesses yield variation in outcomes and model rankings. Analyses on the 30-problem public set identify mandatory-rule violations, inconsistent committed states, and incomplete core behaviors in unsuccessful applications. These findings support evaluating core-goal fulfillment alongside partial completion, with results interpreted under the reported execution conditions. Limitations. The benchmark draws on one organizational setting and uses Chinese-language specifications for self-contained front-end applications. Broader coverage of organizations, languages, and applications with external service dependencies would strengthen its generality. Acceptance is determined by the reviewed rubrics and the completion threshold selected on the public set. Humanreference evaluation covers atomic verdicts and pointwise scores; direct validation of application-level acceptance remains an open task. The three repeated runs capture variation in both generation and evaluation. The reported rankings describe the tested model–harness configurations and budgets. Use of AI tools. AI tools assisted with language editing and consistency checks during manuscript preparation. Their use in benchmark construction and evaluation is described in Section 3 and Appendices A and C. The authors retain responsibility for the manuscript’s content.

References Alibaba Cloud. qwen3.8-max model info. Alibaba Cloud Model Studio documentation, 2026. URL https://www.alibabacloud.com/help/en/model-studio/qwen3-8-max. Evaluated release: Qwen3.8 Max (Aug. 2026), the original general-availability release. Richard Berntsson Svensson and Richard Torkar. Not all requirements prioritization criteria are equal at all times: A quantitative analysis. Journal of Systems and Software, 209:111909, 2024. doi: 10.1016/j.jss.2023.111909. URL https://doi.org/10.1016/j.jss.2023.111909. DeepSeek. DeepSeek-V4-Pro GA release. Official release note, August 13, 2026. URL https: //api-docs.deepseek.com/news/news260813. Evaluated release: DeepSeek-V4-Pro-0813. Sayash Kapoor, Benedikt Stroebl, Peter Kirgis, Nitya Nadgir, Zachary S. Siegel, Boyi Wei, Tianci Xue, Ziru Chen, Felix Chen, Saiteja Utpala, Franck Ndzomga, Dheeraj Oruganty, Sophie Luskin, Kangheng Liu, Botao Yu, Amit Arora, Dongyoon Hahm, Harsh Trivedi, Huan Sun, Juyong Lee, Tengjun Jin, Yifan Mai, Yifei Zhou, Yuxuan Zhu, Rishi Bommasani, Daniel Kang, Dawn Song, Peter Henderson, Yu Su, Percy Liang, and Arvind Narayanan. Holistic agent leaderboard: The missing infrastructure for AI agent evaluation, 2025. URL https://arxiv.org/abs/2510. 11977. Chunyang Li, Yilun Zheng, Xinting Huang, Tianqing Fang, Jiahao Xu, Lihui Chen, Yangqiu Song, and Han Hu. WebDevJudge: Evaluating (M)LLMs as critiques for web development quality, 2025. URL https://arxiv.org/abs/2510.18560. Zhiyu Lin, Zhengda Zhou, Zhiyuan Zhao, Tianrui Wan, Yilun Ma, Junyu Gao, and Xuelong Li. WebUIBench: A comprehensive benchmark for evaluating multimodal large language models in WebUI-to-code. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics: ACL 2025, pages 15780–15797, 2025. doi: 10.18653/v1/2025.findings-acl.815. URL https://aclanthology. org/2025.findings-acl.815/. Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, and Tao Xie. WebCoderBench: Benchmarking web application generation with comprehensive and interpretable evaluation metrics, 2026. URL https://arxiv.org/abs/2601.02430. Zimu Lu, Yunqiao Yang, Houxing Ren, Haotian Hou, Han Xiao, Ke Wang, Weikang Shi, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. WebGen-Bench: Evaluating LLMs on generating interactive and functional websites from scratch, 2025. URL https://arxiv.org/abs/2505.03733. 9

Yuxin Meng, Yuhan Suo, Junjie Wang, Yuhan Sun, Yiyao Yu, Ruixu Zhang, Ruining Hu, Yubin Wang, Shouwei Ruan, Bin Wang, Yuxiang Zhang, and Yujiu Yang. WebRISE: Requirement-induced state evaluation for MLLM-generated web artifacts, 2026. URL https://arxiv.org/abs/2606. 03220. Moonshot AI. Kimi K3 Tech Blog: Open frontier intelligence. Official technical blog, 2026. URL https://www.kimi.com/en/blog/kimi-k3. Davoud Mougouei and David M. W. Powers. Modeling and selection of interdependent software requirements using fuzzy graphs. International Journal of Fuzzy Systems, 19(6):1812–1828, 2017. doi: 10.1007/s40815-017-0364-4. URL https://doi.org/10.1007/s40815-017-0364-4. OpenAI. GPT-5.6 Sol. OpenAI API documentation, 2026. URL https://developers.openai. com/api/docs/models/gpt-5.6-sol. OpenCode Contributors. OpenCode: The open source AI coding agent. Software repository, 2026. URL https://github.com/anomalyco/opencode. Accessed 2026-11-08. Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2Code: Benchmarking multimodal code generation for automated front-end engineering. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 3956–3974, 2025. doi: 10.18653/v1/2025.naacl-long.199. URL https://aclanthology.org/2025.naacl-long.199/. Haoyu Sun, Huichen Will Wang, Jiawei Gu, Linjie Li, and Yu Cheng. FullFront: Benchmarking MLLMs across the full front-end engineering workflow, 2025. URL https://arxiv.org/abs/ 2505.17399. Hung Tran, Langston Nashold, Rayan Krishnan, Antoine Bigeard, and Alex Gu. Vibe Code Bench: Evaluating AI models on end-to-end web application development. In Proceedings of the ACM Conference on AI and Agentic Systems, pages 514–536, 2026. doi: 10.1145/3786335.3813180. URL https://doi.org/10.1145/3786335.3813180. A. van Lamsweerde. Goal-oriented requirements engineering: A guided tour. In Proceedings Fifth IEEE International Symposium on Requirements Engineering, pages 249–262. IEEE Computer Society, 2001. doi: 10.1109/ISRE.2001.948567. URL https://doi.org/10.1109/ISRE.2001. 948567. Naman Vats and Oleg Golev. The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation, 2026. URL https://arxiv.org/abs/2607.22585. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=OJd3ayDDoF. Yiyao Wang, Zhen Wen, Yinghao Tang, Yixiao Fu, Lin Yuan, Xiaolau Zhang, Jun Zhou, and Wei Chen. LiveEvalBench: Toward open-world evaluation for web generation, 2026. URL https://arxiv.org/abs/2608.03689. Z.ai. GLM-5.3: Frontier coding with emergent cyber capabilities. Official technical blog, August 14, 2026a. URL https://z.ai/blog/glm-5.3. Z.ai. GLM-5.3 – overview. Z.AI Developer Documentation, 2026b. URL https://docs.z.ai/ guides/llm/glm-5.3. Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Changzhi Zhou, Ken Deng, Dengpeng Wu, Guanhua Huang, Kejiao Li, Qi Yi, Ruibin Xiong, Shihui Hu, Yue Zhang, Yuhao Jiang, Zenan Xu, Yuanxing Zhang, Wiggin Zhou, Chayse Zhou, and Fengzong Lian. ArtifactsBench: Bridging the visual-interactive gap in LLM code generation evaluation, 2025. URL https: //arxiv.org/abs/2507.04952. Hongda Zhu, Yiwen Zhang, Bing Zhao, Jingzhe Ding, Siyao Liu, Tong Liu, Dandan Wang, Yanan Liu, and Zhaojian Li. FrontendBench: A benchmark for evaluating LLMs on front-end development via automatic evaluation, 2025. URL https://arxiv.org/abs/2506.13832.

10

A

Benchmark Construction and Rubric

A.1

Construction and Human Review

The benchmark contains 100 distinct Chinese-language problem specifications, with 764 tasks and 2,596 atomic criteria (11–52 criteria per problem). Specifications and evaluation artifacts were constructed in four stages: 1. Needs elicitation. In-depth interviews with more than 30 front-end, full-stack, and software engineers and data analysts at JD.com elicited front-end generation cases, product requirements, and full-stack development requests as candidate scenarios. 2. De-identification and reformulation. A structured-output GPT-5.6 Sol pipeline removed proprietary identifiers and sensitive business information, extracted the front-end objective, and eliminated dependencies on databases, back-end services, internal APIs, and organization-specific infrastructure. The evaluated model is required to generate the front-end implementation and mock data according to the specification, including any stated constraints on the data. 3. Human review. Three reviewers with at least one year of practical front-end development experience followed a shared protocol. Two independently checked that every scored requirement followed unambiguously from the specification for a qualified engineer, terminology and state rules were consistent, no criterion combined independently falsifiable requirements, and every validation step could yield observable evidence. The third reviewer adjudicated disagreements. 4. Implementation-based refinement. Each reviewed specification was implemented by a frontier coding model and exercised by hand along its primary user workflow. Specification gaps were revised before regenerating the artifact; implementation errors were repaired. Iteration continued until reviewers accepted the core functions, interaction loops, state rules, and visual requirements. The reviewed specifications, tasks, and criteria were then frozen. The resulting reference artifact is the pairwise Good/Same/Bad comparison target. A.2

Rubric Annotation and Capability Taxonomy

The generation specification defines the artifact, functional and interaction requirements, visual and layout requirements, and engineering constraints. The evaluation rubric additionally contains capability labels, tasks, criteria, and validation protocols. A task is a semantically coherent and independently evaluable objective; its criteria are atomic checks with explicit preconditions, actions, observations, and expected outcomes. Mandatory criteria identify operability or committed-state requirements along the primary user workflow. Tasks containing mandatory criteria are designated as core tasks; pairwise presentation tasks have no mandatory criteria. The complete benchmark has five mutually exclusive application categories: Web and Business Interfaces (30 problems), Demonstrations and Concept Visualizations (23), Applications (21), SVG and Vector-Graphics Generation (13), and Data Visualization (13). At task level, exactly one capability label is assigned to each task, so capability counts do not overlap. Table 4 provides operational definitions and the annotation distribution. A.3

Benchmark Materials

The benchmark links each specification to its tasks, criteria, judging modes, capability labels, and mandatory-criterion designations. The scorer binds core-task policies to versioned specifications and rejects mismatched rubrics.

11

Table 4: Operational definitions and annotation distribution of the eight capability dimensions. Percentages are calculated from unrounded counts and displayed to one decimal place; consequently, they sum to 99.9%. Capability

Operational definition

Component Completeness Representation Fidelity Visual Quality Interaction Responsiveness Computational Rendering Rule Simulation State Management Data Exchange Total

B

Tasks

Share

Completeness of required pages, components, controls, information modules, and necessary technical elements Fidelity with which a static view represents specified entities, data, structures, or domain objects Coherence of composition, hierarchy, typography, color, and overall style for the target setting Whether direct actions such as clicks, drags, scrolling, and keyboard input produce the expected response Whether changes in inputs or parameters correctly and synchronously update derived values, content, and visual output Correctness of implemented business rules, physical mechanisms, state machines, or deterministic algorithms Consistency of state across multi-step operations, views, undo/redo actions, or page reloads Correctness of content, structure, and format during data import, export, or transfer across system boundaries

105

13.7%

42

5.5%

91

11.9%

38

5.0%

108

14.1%

234

30.6%

118

15.4%

28

3.7%

—

764

100%

Harness Controls and Execution Budgets

Evaluated systems. The five evaluated models are Qwen3.8 Max (Aug. 2026) (Alibaba Cloud, 2026), Kimi K3 (Moonshot AI, 2026), GPT-5.6 Sol (OpenAI, 2026), DeepSeek-V4-Pro0813 (DeepSeek, 2026), and GLM-5.3 (Z.ai, 2026a). Qwen3.8 Max (Aug. 2026) denotes the August 2026 general-availability release; DeepSeek-V4-Pro-0813 denotes the August 13, 2026 release. Provider documentation lists image input for the first three (Alibaba Cloud, 2026; Moonshot AI, 2026; OpenAI, 2026) and text-only input for GLM-5.3 (Z.ai, 2026b). Two benchmark-specific adapters connect the unchanged upstream harnesses to a shared model registry. They standardize model access, inference configuration, workspace initialization, and benchmark execution. Each model–harness–problem–run combination starts independently. OpenHands is the reference harness for the main leaderboard and failure analysis; OpenCode supports cross-harness comparisons. Controlled settings. Both harnesses receive the same benchmark specification, clean Docker workspace, network and dependency policies, Playwright 1.58.1 runtime, shared model registry, and five-hour wall-clock budget. We additionally impose a nominal cap of 500 iterations in OpenHands and 500 steps in OpenCode. Iteration and step counts use each harness’s native execution units. Each harness retains its native agent loop, context-management strategy, tool/action representation, execution behavior, and stopping logic. These components remain part of each evaluated model– harness system.

C

Evaluation Skills, Report Contract, and Evidence Audit

Each artifact is evaluated in an isolated Docker container with the submitted workspace, specification, and scoring policy mounted read-only. Validation first checks the artifact contract and attempts the required build and launch. A build or runtime failure sets Bqr = 0. If the artifact cannot launch, only criteria predefined as source-verifiable may receive supported credit. The judge is GPT-5.5 with high reasoning effort. The automated judge follows the benchmark-wide pipeline skill, the common web-evaluation skill, and the evaluation profile for the current problem. Each profile specifies the required workflows, atomic criteria, and admissible evidence modalities. For pointwise tasks, the evaluation record associates each atomic criterion with its own Pass/Fail verdict, rationale, and supporting evidence. For pairwise tasks, it retains one task-level Good, Same,

12

or Bad verdict, evidence from both the candidate and reference artifacts, and the viewport, initial state, interaction path, and observation time used for comparison. The deterministic report writer matches these results to the authoritative task and criterion definitions. Missing criteria remain in the scoring denominator and receive zero. The Evidence Auditor checks report structure, criterion coverage, evidence availability, and evidencetype admissibility. A behavioral claim requires execution-grounded evidence; source inspection suffices only for criteria designated source-verifiable. Semantic verdicts remain the judge’s responsibility. An audit finding identifies the affected task or criterion and the evidence defect. The judge may perform up to three targeted remediation rounds for affected criteria, retaining the initial output, audit findings, additional evidence, and revised verdicts. A verdict changes only when new evidence supports the revision. After the final round, a criterion or pairwise task without a valid verdict or admissible evidence is assigned zero; accepted results for unrelated criteria remain available. The scorer computes ATS, CTPR, and ASR from the audited results and fixed scoring policy.

Figure 5: Evidence audit workflow from structured verdicts through up to three targeted remediation rounds to scoring. The limit applies to audit remediation rounds, not to individual test actions.

D

Metric Definitions and Threshold Sensitivity

D.1

Metric Semantics and Formal Definitions Table 5: Evaluation scope, unit scoring, and task aggregation for ATS, CTPR, and ASR.

Metric

Evaluated scope

Unit score

Across-task aggregation

Interpretation

ATS

All tasks

Graded

Average

CTPR

Core pointwise tasks

Binary pass

Average

ASR

Operability and all core tasks

Binary pass

Conjunction

Local completion and quality Average core-task acceptance Application-level success

For a fixed model–harness condition, let Q be the reported problem set and let R = 3 be the number of scheduled runs per problem. Let Tqpt and Tqpw denote the pointwise and pairwise tasks of problem q, respectively, with Tq = Tqpt ∪ Tqpw . 13

For pointwise task t, let Jqt be its atomic criteria. Its completion score is 1 X yqrtj , Aqrt = |Jqt | j∈Jqt

where yqrtj = 1 if criterion j retains a Pass verdict after auditing and 0 otherwise. Failed, missing, malformed, errored, or inadmissible criterion outcomes are assigned zero. For a pairwise task, define  Good, 1, Gqrt = 0.5, Same,  0, Bad. Invalid or missing pairwise outcomes are likewise assigned zero. For either judgment mode, define the task score as  Aqrt , t ∈ Tqpt , Zqrt = Gqrt , t ∈ Tqpw . The problem-run ATS is 1 X Zqrt . |Tq |

Oqr =

t∈Tq

Let Cq ⊆ Tqpt be the predefined core tasks and let Mqt ⊆ Jqt contain the mandatory criteria of core task t. Its pass indicator at threshold τ is   Y (τ ) Pqrt =  yqrtj  I[Aqrt ≥ τ ], j∈Mqt

and the problem-run CTPR is (τ ) Cqr =

1 X (τ ) Pqrt . |Cq | t∈Cq

Let Bqr ∈ {0, 1} indicate whether the generated application satisfies the build-and-runtime contract, with a rationale and admissible evidence retained after auditing. The problem-run application-level success indicator is Y (τ ) (τ ) Sqr = Bqr Pqrt . t∈Cq

For any problem-run quantity xqr , both splits first average all R = 3 scheduled run slots within each problem and then macro-average problems: Avg(x) =

R 1 X 1 X xqr . |Q| R r=1 q∈Q

The reported metrics are ATS = Avg(O),

CTPRτ = Avg(C (τ ) ),

ASRτ = Avg(S (τ ) ).

Let D denote the eight capability dimensions, let dqt be the dimension assigned to task t of problem q, and define Ud = {(q, t) : q ∈ Q, t ∈ Tq , dqt = d}. The capability score first averages each task across its scheduled runs and then gives equal weight to all tasks assigned to dimension d: 1 CapScored = |Ud |

R

X (q,t)∈Ud

1 X Zqrt , R r=1

d ∈ D.

These eight task-weighted capability scores support diagnostic analysis; the main ranking uses application success. 14

We use τ = 0.8 for the primary evaluation and abbreviate CTPR0.8 and ASR0.8 as CTPR and ASR; per-run quantities also omit the threshold superscript at this setting. Infrastructure failures are rerun. A model-generated artifact that fails to build or run remains in the evaluation matrix with Bqr = 0. All scheduled slots remain in the denominator. Unresolved missing runs or records invalidated as a whole receive zero for ATS, CTPR, and ASR. CTPR and ASR require a nonempty core-task set for each evaluated problem; otherwise, these metrics are unavailable. D.2

Public-Set Results and Failure Composition

On the 30-problem public set, OpenHands yields five-model means of 86.90% ATS, 79.36% CTPR, and 50.89% ASR (Table 6). The 28.47-point CTPR–ASR difference contrasts average core-task acceptance with joint application acceptance, which also requires operability. Table 6: Aggregate OpenHands scores (%) on the 30-problem public set. CTPR−ASR contrasts average core-task pass rate with joint application success. Model

ATS

CTPR

ASR

CTPR−ASR

Qwen3.8 Max (Aug. 2026) Kimi K3 GPT-5.6 Sol DeepSeek-V4-Pro-0813 GLM-5.3

88.57 87.87 90.53 84.99 82.54

82.19 81.52 82.33 79.06 71.69

57.78 55.56 55.56 46.67 38.89

24.41 25.96 26.78 32.39 32.80

Mean

86.90

79.36

50.89

28.47

Table 2 reports failure mechanisms among unsuccessful OpenHands runs on the public set with valid criterion judgments. A mandatory failure occurs when any core task fails a mandatory criterion. A completion failure occurs when a core task passes all of its mandatory criteria but falls below 80% completion. “Mandatory only” and “completion only” indicate one mechanism; “both” indicates a mandatory failure and a completion failure in different core tasks. The final row pools valid failed runs across models. The three categories cover the valid failed runs in this analysis. Invalid reports are excluded from this conditional analysis and retain their zero scores in the scheduled-run metrics. D.3

Representative Cases for the Three Metric Layers

Table 3 gives the per-run scores behind the two public-set runs discussed in Section 4.3. Both reports contain audited criterion judgments; each run fails its core-task acceptance conditions and has Sqr = 0. D.3.1

Case A: High Task Scores with Failed Mandatory Rules

The specification requires a personal-finance dashboard that lets users complete a transfer while keeping the balance, transaction list, and derived statistics consistent. The minimum workflow for that goal includes amount validation, secondary confirmation, transaction submission and write-back, balance deduction, and updates to the derived views. Two mandatory criteria require invalid amounts to be rejected before confirmation and all derived views to reflect a completed transfer. Each criterion is required for its core task to pass. This run fails at both points. Entering -5 silently removes the minus sign and leaves 5 in the amount field without an error (Figure 6), then proceeds to a $5.00 secondary confirmation (Figure 4(a)). After a valid transfer, the “Transfer” entry in the category breakdown rises from $120.00 over one transaction to $132.34 over two. The transaction therefore appears in some derived views, but the category budgets still have four rows with no Transfer budget, and overall use remains at 49%. The budget view does not reflect the updated transaction state. Across all seven tasks, five score 100 and two score 66.67, so ATS is (5×100+2×66.67)/7 ≈ 90.48. The two failures lower ATS by only 9.52 points, but each failed mandatory criterion prevents its core task from passing. Only two of four core tasks pass; thus the per-run core-task pass fraction is Cqr = 50% and Sqr = 0.

15

Figure 6: Case A: after entering -5, only 5 remains in the amount field with no validation error. Figure 4(a) shows the resulting $5.00 confirmation. D.3.2

Case B: Passing Mandatory Criteria with Incomplete Core Tasks

The problem asks for a declaratively animated octopus playing a jazz drum kit in a single native-SVG file. Its core tasks require eight arms with distinct playing roles and stick heads that reach their corresponding drums or cymbals. Six mandatory criteria check whether this basic action sequence and its implementation constraints hold. All six pass in this run. The core tasks also include non-mandatory details. Task 2 scores 75.00%. The eight arms have distinct roles and remain attached, but their equal-width paths lack natural taper (Figure 7). Task 4 scores 50.00%. The sticks reach the snare, rack tom, ride, and crash, but the struck surfaces do not deform or move in synchrony (Figure 4(b)). In runtime samples, the struck targets’ bounding boxes are bit-identical across nine instants, while the kick beater changes width in the same data. This confirms that the sampling procedure captured motion elsewhere in the same artifact. The remaining defects concern the specified arm shape and synchronized target response.

Figure 7: Case B: the eight animated arms have equal-width paths and lack natural taper (sampled at t = 0.12). The ride contact frame at t = 2.34 is shown in Figure 4(b). Although all six mandatory criteria pass, each core task must also reach 80% completion. Tasks 2 and 4 score 75.00% and 50.00%, so only one of three core tasks passes: Cqr = 33.33% and Sqr = 0. Appendix D.4 examines the 80% threshold. Case A violates mandatory transfer rules; Case B satisfies its mandatory criteria but has incomplete core tasks. The cases illustrate the distinct roles of mandatory criteria and the completion threshold in application acceptance. D.4

Completion-Threshold Sensitivity

We compared τ ∈ {0.75, 0.80, 0.90} on the 30-problem public set under OpenHands across the five evaluated models. Raising the threshold from 0.75 to 0.80 lowers the five-model mean ASR by 7.33 percentage points and mean CTPR by 3.56 points. Raising it from 0.80 to 0.90 lowers them by a further 6.00 and 3.01 points, respectively. All changes are computed before rounding. 16

Table 7: Completion-threshold sensitivity on the 30-problem public set under OpenHands. Means average five models; ASR range is the maximum minus minimum model ASR, in percentage points. All scores are percentages. Threshold 0.75 0.80 0.90

Mean ASR

ASR range

Mean CTPR

58.22 50.89 44.89

17.78 18.89 21.11

82.91 79.36 76.34

The completion threshold operates discretely because each core task contains a finite number of criteria. On the public set, a threshold of 0.90 requires all criteria to pass for 99.1% of core tasks. At 0.80, this proportion is 86.6%, while the remaining 13.4% permit one non-mandatory criterion to fail. Table 7 shows how the alternative thresholds affect aggregate scores. Considering these score changes and the discrete behavior of the criteria, we use 0.80 as the default completion threshold.

E

Additional Results and Diagnostics

The additional test-set results cover application categories, capability dimensions, and repeated-run outcomes across 70 problems. Generation-resource statistics use the public set. E.1

ASR by Application Category

Figure 8 displays category-level ASR, problem counts, and cross-model means.

(A

6) 02 .2 g u

x Ma 3 3.8 n iK e Qw K im

GP

ol 6S . 5 T-

De

e Se ep

k-V

13 -08 o r 4-P 5.3 ML G

an Me

Web and Business Interfaces (21)

36.5

47.6

54.0

49.2

31.7

43.8

SVG and Vector-Graphics Generation (9)

59.3

51.9

63.0

29.6

18.5

44.4

Data Visualization (9)

63.0

59.3

29.6

40.7

70.4

52.6

Applications (15)

62.2

71.1

55.6

48.9

53.3

58.2

Demonstrations and Concept Visualizations (16)

77.1

56.3

68.8

58.3

33.3

58.8

20

30

40

50

60

70

80

Application Success Rate (%)

Figure 8: Category-wise ASR (%) on the 70-problem test set under OpenHands. Cells show scores and cross-model means; row labels give problem counts. Color represents ASR. Across the evaluated test problems, Web and Business Interfaces and SVG and Vector-Graphics Generation have the lowest cross-model means (43.8% and 44.4%).

17

E.2

Capability-Dimension Scores

Table 8 gives task counts, full capability scores, and dimension means for Figure 3. Table 8: Capability scores (%) on the 70-problem test set under OpenHands. Parentheses give task counts; bold marks the highest value within each dimension. Component Completeness (75)

Representation Fidelity (30)

Qwen3.8 Max (Aug. 2026) Kimi K3 GPT-5.6 Sol DeepSeek-V4-Pro-0813 GLM-5.3

88.44 90.67 88.89 86.22 87.56

94.44 94.44 96.67 93.33 83.33

73.85 72.31 83.08 59.49 63.59

98.77 92.59 95.06 96.30 93.83

Mean

88.36

92.44

70.46

95.31

Model

Computational Rendering (77)

Model

Rule Simulation (167)

Visual Quality (65)

State Management (84)

Interaction Responsiveness (27)

Data Exchange (20)

Qwen3.8 Max (Aug. 2026) Kimi K3 GPT-5.6 Sol DeepSeek-V4-Pro-0813 GLM-5.3

92.64 91.34 91.77 88.74 86.15

92.42 91.42 92.61 88.82 84.63

90.48 91.27 89.29 90.87 88.49

98.33 95.00 86.67 91.67 93.33

Mean

90.13

89.98

90.08

93.00

E.3

Generation Resource Use

For runs on the 30-problem public set, mean generation latency is 39.65 minutes under OpenCode and 59.24 minutes under OpenHands; mean token use per run is 6.18M and 9.52M, respectively. These describe generation resources for harnesses with different agent loops (Appendix B). E.4

Repeated-Run Outcome Distributions

Table 1 summarizes repeated-run outcomes on the 70-problem test set. Each of the five models has three scored OpenHands generation-and-evaluation runs for every test problem, giving 70 model– problem conditions and 210 scored runs per model. Each condition is categorized as all successful, mixed, or all failed. These distributions describe end-to-end outcome consistency across three runs, while ASR summarizes per-run success under the scoring protocol in Appendix D. E.5

Additional End-to-End Failure Cases

The following public-set cases supplement the detailed metric examples in Appendix D.3. In Q2, restoring a field to its template value should also remove the override and restore inheritance; 13 of 15 OpenHands reports record a failure on this criterion. The visible value can be correct while its source remains wrong. In Q20, 13 of 15 reports record synchronization failures after a transaction: the list and balance may update without matching changes to charts, budgets, timelines, or aggregate statistics. Both cases expose inconsistent state semantics despite working local interactions. Across the evaluated artifacts for Q12, Q18, and Q29, observed failures involve contact, phase, direction, or dependence on velocity and gravity. Examples include timeline-driven vapor-film effects, projectile arcs whose landing points are disconnected from input parameters, and moving drumsticks without actual contact with their targets. These failures concern the specified mechanisms and their consistency over time.

18

F

Agreement with the Adjudicated Human Reference

F.1

Comparison Scope

The analysis covers all 30 public-set problems and 726 pointwise atomic criteria. The pointwise track compares Pass/Fail judgments, while the pairwise track compares Good/Same/Bad judgments on 15 tasks. Score-level comparisons use aggregated pointwise judgments. The comparison measures agreement in criterion verdicts and pointwise scores; application-level human agreement is a separate measurement target. Human reference. Two evaluators independently annotate each unit under a shared protocol, without access to the identity of the generating model or one another’s decisions. Agreed labels become the reference directly; a third evaluator adjudicates disagreements. The automated judge completes its assessment without access to human labels or rationales. Automated outputs are matched to the final human reference after both assessments are complete. F.2

Agreement Results

Table 9 consolidates the reported label- and score-level comparisons. Table 9: Agreement between the automated judge and the adjudicated human reference on 30 publicset problems. Pointwise uses Pass/Fail atomic criteria; pairwise uses Good/Same/Bad judgments on 15 tasks. Problem- and task-level scores aggregate pointwise judgments. Pooled micro agreement combines the two label tracks and is supplementary. Formulas appear in Appendix F.3. Metric

Evaluation unit

Label-level agreement Exact agreement Cohen’s κ Macro-F1 Exact agreement Micro exact agreement

Pointwise criteria Pointwise criteria Pointwise criteria Pairwise tasks (n = 15) Pooled tracks

83.47% 0.651 0.825 86.67% 83.54%

Score association and error Pearson r Spearman ρ RMSE/SD Mean error Pearson r Spearman ρ

Problem Problem Problem Problem Task Task

0.807 0.766 0.631 +0.0362 0.709 0.731

F.3

Value

Metric Formulas

Label-level metrics in Section 4.4 compare pointwise Pass/Fail criteria or pairwise Good/Same/Bad tasks as separate tracks. Score-level metrics aggregate pointwise judgments, with xi the final human reference score and yi the judge score on the same problem or task; n denotes the number of comparable units.

Exact agreement. Nmatch , (4) Ncomparable where Nmatch is the number of units assigned exactly the same label by both sides and Ncomparable is the number of units for which both sides produced a valid label. This metric compares final labels; κ additionally accounts for agreement expected by chance. Agreement =

Cohen’s κ. po − p e , (5) 1 − pe where po is the observed agreement (i.e. exact agreement) and pe is the chance agreement expected from the two Pass/Fail marginal distributions. κ = 0 indicates that observed agreement equals the chance agreement expected from the marginal label distributions; κ = 1 indicates perfect agreement. We compute κ for pointwise Pass/Fail judgments. κ=

19

Macro-F1. Macro-F1 =

1 2

X c∈{Pass,Fail}

2 T Pc . 2 T Pc + F Pc + F Nc

(6)

With adjudicated human labels as the reference and judge labels as predictions, T Pc , F Pc , and F Nc are the true positives, false positives, and false negatives obtained when class c is taken as the positive class. Macro-F1 weights the Pass and Fail classes equally.

Pearson correlation. Pn (xi − x̄)(yi − ȳ) r = pPn i=1 , (7) P 2 (x − x̄)2 n i i=1 i=1 (yi − ȳ) P P where x̄ = n1 i xi and ȳ = n1 i yi are the two score means. Pearson r measures linear association and is invariant to additive shifts and positive rescaling; RMSE/SD and mean error quantify score discrepancies.

Spearman correlation. Pn ρ = qP

i=1 (Ri − R̄)(Si − S̄)

,

Pn n 2 2 i=1 (Ri − R̄) i=1 (Si − S̄)

(8)

where Ri = rank(xi ) and Si = rank(yi ) are the ranks of the two score series and R̄, S̄ are their means. Spearman ρ measures monotonic association through the ordering of scores.

Normalized root-mean-square error. q P n 1

i=1 (yi − xi ) n RMSE = q , P n SD(x) 1 (x − x̄)2 n−1

2

i=1

(9)

i

The numerator is the root-mean-square error of the judge scores against the human scores; the denominator is the sample standard deviation of the human scores. The ratio expresses score error relative to variation in the human reference. A value below 1 indicates that RMSE is smaller than this standard deviation; a value near 1 indicates comparable magnitudes.

Mean error (ME). n

ME =

1X (yi − xi ). n i=1

(10)

ME > 0 indicates that judge scores are higher on average, and ME < 0 indicates lower scores. ME is the signed average error; positive and negative discrepancies may cancel.

20

Related documents

• JoyFrontBench: Evaluating Front-End Coding Agents through Goal-Oriented Acceptance Contracts#1648982
• From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software#346591
• Coding-Agent Benchmarks Should Match Their Users' Task Flows#1234001
• Coding-Agent Benchmarks Should Match Their Users' Task Flows#1227411
• SWE-Residual: Evaluating Starting-State Robustness of Coding Agents#1401090