OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development

O PEN H ARMONY B ENCH : E VALUATING LLM S AND C ODING AGENTS ON O PEN H ARMONY A PP D EVELOPMENT AI-Assisted R&D TMG, Huawei SMAT Lab, Beihang University CodeWisdom Lab, Fudan University

Official website: https://bench.matrix.openharmony.cn/

arXiv:2608.16022v1 [cs.SE] 17 Aug 2026

A BSTRACT We present OPENHARMONY BENCH, an app-level coding benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS applications. Unlike function-level code generation benchmarks, O PEN H ARMONY B ENCH evaluates complete app-level changes: each task requires an agent to modify a buildable ArkTS project so that a requested application behavior works end to end, which in turn requires coordinated edits across project structure, UI state, local data persistence, build configuration, and platform APIs. The benchmark then installs and drives the delivered application on a device to check whether that behavior is actually observable. The benchmark covers three input sources: natural-language feature requests (new-feature), structured scenario specifications (spec-driven), and bug descriptions (bug-fix). The benchmark contains 153 top-level inputs and 242 Feature points (F-points), where an F-point is one executable behavior check. The current snapshot contains 32 new-feature tasks, 50 spec-driven tasks with 139 F-points, and 71 bug-fix tasks. The main leaderboard is scored over top-level tasks rather than by independently weighting F-points. We describe the benchmark construction, statistics, and build-andtest evaluation pipeline, and evaluate DevEco Code with eight LLMs across three independent full-suite runs per configuration, reporting the arithmetic mean of top-level Task Completion across runs. Three findings emerge. First, within the evaluated model-family pairs, newer generations complete more tasks than their predecessors. Second, buildability is close to saturated while behavioral correctness is not: mean Final Build Success Rate lies between 94.77% and 100.00%, whereas mean Task Completion lies between 48.36% and 58.39%, so a large fraction of delivered projects compile without implementing the requested behavior. Third, spec-driven tasks have the lowest Task Completion under all-checks task scoring, with no configuration exceeding 35%. The code, data, tasks, reference solutions, tests, evaluation scripts, and leaderboard are released through the official O PEN H ARMONY B ENCH website at https://bench.matrix. openharmony.cn/.

1

I NTRODUCTION

Large language models (LLMs) and coding agents are increasingly expected to operate inside real software projects rather than solve isolated programming exercises. Existing benchmarks have made substantial progress on function-level code generation (Chen et al., 2021; Austin et al., 2021; Hendrycks et al., 2021; Lu et al., 2021), repository-level software maintenance (Jimenez et al., 2024; Yang et al., 2024; Xia et al., 2025; Zan et al., 2025; Xu et al., 2025), and visual or web-oriented development tasks (Yang et al., 2025; Cui, 2024; Xiao et al., 2025). However, these settings do not fully capture application-level delivery. Many coding benchmarks evaluate a patch with function-level, issue-level, or unit-test-style checks, while mobile and OpenHarmony application tasks require the submitted project to build, launch, and exhibit the expected GUI behavior in a platform runtime. In this setting, correctness depends on ArkUI components, page routing, UI state, persistence, platform APIs, resources, and build configuration working together inside a runnable application. A benchmark for OpenHarmony app development therefore needs to test whether a coding agent can turn 1

source-level changes into buildable and behaviorally correct applications, rather than only satisfy narrower localized checks. OpenHarmony is a useful app-development ecosystem for this style of evaluation. Its ArkTS and ArkUI ecosystem (Huawei Developer, 2026a) is widely used in HarmonyOS application development, but it poses several practical challenges for benchmark construction. Tasks must be evaluated inside buildable applications rather than standalone files. UI behavior must be checked through emulators or devices. Platform APIs, project layouts, and build tools evolve quickly, so benchmark tasks can become stale for reasons unrelated to agent ability. In addition, many failures only appear after the application is built, installed, launched, and tested through realistic GUI interactions. These properties make OpenHarmony app development a concrete setting for evaluating whether coding agents can deliver platform-specific application behavior under realistic engineering requirements. We present OPENHARMONY BENCH, an app-level coding benchmark for OpenHarmony development. The benchmark is organized around three related input sources that coding agents commonly face in OpenHarmony app development: natural-language feature requests, structured scenario specifications, and bug descriptions. The new-feature source contains 32 tasks in existing projects, where each task starts from a concise natural-language description of a feature to implement. The spec-driven source contains 50 structured tasks and 139 F-points, where each task starts from a scenario-level specification that may cover multiple observable behaviors. The bug-fix source contains 71 real bug-fix tasks, where each task starts from a description of an observed application failure. We also characterize the platform capabilities exercised by the benchmark through aggregate Kit/API coverage analysis over all 153 tasks. O PEN H ARMONY B ENCH evaluates submitted projects through a shared build-and-test pipeline. For each task, an agent produces or modifies an OpenHarmony project, the pipeline checks buildability, and task-specific executable UI tests validate observable behavior. Task Completion is the primary effectiveness metric: one top-level benchmark input is marked complete only when the final project builds and all required executable checks pass. The spec-driven source still contains F-points as finegrained checks inside each task, and all F-point checks must pass for the corresponding spec-driven task to be marked complete. Build pass rates and raw resource-use diagnostics provide complementary signals. This separation is important because an implementation may build successfully while still failing required interaction logic, state transitions, or persistence behavior. This report describes the benchmark construction, statistics, evaluation pipeline, and benchmarklevel baseline results. Each Agent–LLM configuration is evaluated through three independent fullsuite runs. We first compute Task Completion for each run and then report the arithmetic mean across runs. The leading configuration, DevEco Code with GLM-5.2, achieves a mean Task Completion of 58.39%, with an observed run-level range of [56.21%, 60.78%]. Mean Final Build Success Rate remains high across the evaluated configurations, ranging from 94.77% to 100.00%, while mean Task Completion ranges from 48.36% to 58.39%. This gap indicates that behavior-level correctness, rather than compilation alone, remains the main challenge for OpenHarmony app-level coding agents. Figure 1 summarizes the current leading Task Completion result over the complete benchmark and the three task sources introduced above. Here “Overall” aggregates all 153 tasks across new-feature, spec-driven, and bug-fix. The top row is the complete-benchmark aggregate, while the lower rows report the three task sources separately. Each bar reports the best mean Task Completion among the evaluated DevEco Code–LLM configurations for that scope; the label names the corresponding leading model, and the whisker shows that configuration’s observed run-level min–max range. The contributions of this report are: • An OpenHarmony app-level benchmark. We define a benchmark structure spanning requirement-style incremental development, specification-driven development, and bug fixing. • Executable evaluation for ArkTS applications. We describe a build-and-test pipeline based on OpenHarmony project builds and executable UI behavior checks, with Task Completion as the primary metric and buildability as a complementary metric. • Benchmark-level agent and model findings. Under the fixed DevEco Code agent framework, we report four benchmark-level observations: evaluated configurations complete 2

Overall (GLM-5.2)

58.39

New-feature (GLM-5.2)

63.54

Spec-driven (Kimi-K3)

34.67

Bug-fix (Kimi-K3 )

75.12 20

40 60 Task Completion (%)

80

Figure 1: Leading Task Completion by benchmark scope. Bars show the best mean Task Completion across evaluated DevEco Code–LLM configurations for each scope, labels identify the corresponding leading model, and whiskers show the observed run-level min–max range for the same configuration. The x-axis starts at 20% to focus on the observed score range. roughly half of the 153 top-level tasks; newer generations complete more tasks than their predecessors within the evaluated model-family pairs; Final Build Success Rate is high while Task Completion remains much lower; and spec-driven tasks have the lowest Task Completion under all-checks scoring. Availability, versioning, and reproducibility. The benchmark artifacts needed to inspect and reproduce the current evaluation snapshot are released through the official O PEN H ARMONY B ENCH website, including task descriptions, source-code scaffolds or repository pointers, reference solutions, executable tests, evaluation scripts, and result aggregation code. The official project URL is https://bench.matrix.openharmony.cn/. The results in this report correspond to the OpenHarmony Bench v1.0 task snapshot dated 2026.7.31. Future leaderboard entries should report the benchmark version, task snapshot, evaluation script snapshot, DevEco Code version, model identifier, execution environment, and run configuration so that scores remain comparable over time.

2

B ENCHMARK C ONSTRUCTION

O PEN H ARMONY B ENCH is an OpenHarmony app-level coding benchmark covering repositorylevel incremental development and bug fixing. The new-feature source captures requirement-style incremental feature implementation, the spec-driven source captures standardized specificationdriven incremental development, and the bug-fix source captures bug fixing from real bug-fix tasks. Together, these task sources move beyond isolated code snippets by requiring agents to produce buildable ArkTS applications and validate the resulting behavior through build and test pipelines. 2.1

TASK D EFINITION

We formulate O PEN H ARMONY B ENCH as an app-level coding benchmark for OpenHarmony applications. Each instance provides a task description paired with a runnable ArkTS project at a base commit. Given this input, a coding agent must modify the project to satisfy the requested application behavior while preserving a buildable OpenHarmony project. The agent-facing objective is the stated user requirement rather than the executable checks. The executable checks are not exposed to the agent. They are used only after generation to determine whether the delivered project satisfies the requested behavior. This separation prevents agents from optimizing against the oracle and keeps the task close to how development requirements are given in practice. The benchmark therefore rewards behaviorally correct app-level implementations rather than token-level similarity to a reference patch. The benchmark uses two units. The scoring unit is one top-level benchmark input: one new-feature requirement, one spec-driven specification, or one bug-fix report. The 153 inputs are therefore 153 tasks, and the leaderboard is computed over these top-level tasks regardless of task source. The checking unit is the Feature point (F-point), which is one executable behavior check. Most tasks have exactly one F-point, so the scoring unit and checking unit coincide. Spec-driven tasks are 3

Task source

Task type

Source and evaluation signal

new-feature

Requirement-style feature implementation

spec-driven

Specification-driven implementation

bug-fix

Bug fixing

Realistic requirement-style tasks with executable UI tests. Standardized scenario-level tasks with 139 Fpoints implemented as Hypium checks. PR-derived bug-fix tasks whose tests fail before the fix and pass after the fix.

Table 1: Benchmark task sources in O PEN H ARMONY B ENCH. the exception: a single specification can describe several observable behaviors, so it carries several F-points (2.78 on average). Such a task is scored all-or-nothing: it is marked complete only if the project builds and every one of its F-points passes. F-points therefore provide finer-grained diagnostic evidence within a task, but they are not an independent leaderboard unit. Compared with function-level code generation benchmarks, OpenHarmony app tasks require reasoning over project structure, routing, state management, persistence, UI components, build configuration, and platform APIs across multiple files. The task boundary is intentionally practical: the agent is expected to implement a specified application behavior, extend an existing project, or fix a known bug, while open-ended product design and requirement discovery are out of scope. 2.2

B ENCHMARK TASK S OURCES

O PEN H ARMONY B ENCH organizes these task families into three benchmark task sources. Table 1 summarizes their roles and evaluation signals. new-feature. The new-feature source evaluates incremental feature implementation in OpenHarmony applications. Its 32 tasks are built primarily from Huawei official template projects and requirement-style development scenarios, covering new business features and functional iterations. Each task is screened to have a clear requirement, a bounded implementation scope, and executable UI tests. This source measures whether a coding agent can understand an existing project, modify the relevant files, and deliver a feature that works in the application context. spec-driven. The spec-driven source contains 50 structured tasks and 139 F-points. The 50 specifications are derived from open-source OpenHarmony projects through preprocessing, manual screening, and curator-guided merging of related requirements. Unlike requirement-style tasks, specdriven tasks use standardized scenario-level specifications as the model-facing input. Each task is paired with a runnable base project and a curator-reviewed reference implementation that defines the expected behavior. The agent must translate each task into a buildable, runnable, and verifiable OpenHarmony application implementation. bug-fix. The bug-fix source evaluates engineering-level bug fixing for OpenHarmony ArkTS projects in a repository-level setting exemplified by SWE-bench (Jimenez et al., 2024). Its 71 tasks are derived from bug-fix pull requests in real open-source OpenHarmony projects. Candidate fixes are retained only when the bug is reproducible, the fix scope is bounded, and the associated tests can distinguish the buggy and fixed versions. This source targets fault localization, code understanding, and bug-fixing effectiveness rather than new feature construction. 2.3

TASK C ONSTRUCTION

All task sources are built around runnable OpenHarmony applications rather than isolated code snippets. For each candidate input, we identify a base project state, define the expected behavior, and attach task-specific automated checks. Candidate tasks are retained only when the requirement is clear, the behavior is reproducible, the required change is bounded, and the project remains compatible with the evaluation environment. For bug-fix, we additionally screen for focused underlying changes so that the task emphasizes fault localization and repair rather than broad refactoring.

4

new-feature: requirement-style feature implementation. Add a Text component at the first position below the text-type components on the home page. The home-page component list is data-driven, so the Text entry must be added to both the data source and the code configuration. The new component should follow the existing component pattern, such as TextInput, and include a preview area plus a control panel with font weight, font size, text shadow, and letter spacing controls. The preview defaults to “Hello Developer”. spec-driven: wx-combo-chat-input task. Scenario 1: tapping the “+” icon expands the bottom action tray with an animation and reveals options such as Album, Camera, Voice Call, and Location. Scenario 2: long text automatically expands the input box to multiple lines. Scenario 3: entering text shows the green “Send” button, and tapping “Send” appends the message to the chat list. Scenario 4: deleting all text hides “Send” and restores the “+” icon. bug-fix: Penkit crash task. When the application is launched and the user clicks the “Try Penkit” button, the application crashes immediately, preventing normal entry into the corresponding feature module. The agent must localize and fix the bug so that the Penkit feature opens normally.

Figure 2: Representative task examples from the three benchmark task sources in O PEN H ARMONY B ENCH. The new-feature source is primarily built from Huawei official template projects1 and covers common development scenarios such as cross-layer feature implementation, standalone UI feature implementation, HarmonyOS Kit integration, third-party SDK integration, feature-parameter extension, and scenario coverage extension. The bug-fix source is built from Huawei official open-source repositories by selecting historical pull requests that cover bug fixes, feature restoration, and stability optimization. The spec-driven source is built from manually curated, standardized scenario-level specifications over curated OpenHarmony base projects. For each specification, curators select a compatible base project, implement a reference solution, and retain only tasks whose expected behavior can be exercised reliably in the shared OpenHarmony evaluation environment. The new-feature and spec-driven sources differ in the task input given to the agent. The newfeature source preserves compact development requests that resemble real incremental requirements, while spec-driven uses standardized scenario-level specifications with explicit acceptance structure. This separation lets the benchmark cover realistic requirement-style feature work and fine-grained specification-following analysis. The bug-fix source extends the same project-level setting to tasks grounded in real bug-fix changes. 2.4

F- POINT AND T EST C ONSTRUCTION

For all three task sources, benchmark curators manually construct executable oracles and Hypium test scripts from the task description, expected behavior, and project context. Each test is written to check user-visible behavior rather than implementation-specific code structure. Spec-driven tasks require additional normalization because one document often combines main paths, alternative flows, and boundary conditions. We normalize each spec-driven specification into a hierarchy of specification, scenario, and F-point. A specification defines the full task scope, a scenario represents a coherent user path, and an F-point corresponds to one executable behavior check that should be implemented and verified independently. This construction yields 139 F-points for the spec-driven source. A spec-driven task is marked complete only when the project builds and all F-point checks under the corresponding task pass. Every retained F-point must satisfy a fail-to-pass criterion. For new-feature and spec-driven tasks, the corresponding test must fail on the base revision where the requested behavior is absent and pass on the reference solution. For bug-fix tasks, the test must reproduce the reported failure on the buggy revision and pass on the fixed revision. Tests that do not distinguish the pre-change and post-change states are discarded. At least two curators independently validate each retained test, and each F-point must pass on the reference solution in at least two validation runs before the task is admitted into the benchmark. 1

https://developer.huawei.com/consumer/cn/market/landing/component

5

(a) Common benchmark composition Task source Input form

Repos Inputs F-points Mean F-points per input Difficulty (E/M/H)

new-feature Requirement-style request spec-driven Structured task bug-fix Bug report Overall

–

12 10 31

32 50 71

32 139 71

1.00 2.78 1.00

11/18/3 11/25/14 36/29/6

42 repos

153

242

1.58

58/72/23

(b) Input, reference, and repository-concentration statistics Task source Input length [IQR] Ref. files Ref. LOC Largest repo (%) Top-3 repos (%) new-feature 118.5 [61.8–165.2] spec-driven 589 [496.5–794.5] bug-fix 107 [40–248.5]

4 12 2

88 489.5 83

31.2% 38.0% 14.1%

53.1% 74.0% 31.0%

Table 2: Benchmark-wide and task-source-level statistics. A repository source is one OpenHarmony code repository from which benchmark inputs are drawn. Mean F-points per input is the total number of executable checks divided by top-level inputs, and difficulty reports easy/medium/hard. Input length reports median [IQR] in non-whitespace characters. Ref. files and Ref. LOC are medians of changed files and added-plus-deleted lines in the reference patch. Largest repo and Top3 repos report the percentage of inputs coming from the most frequent repository and the three most frequent repositories; percentages are used because the three task sources contain different numbers of inputs. Kit/API coverage is reported separately at the benchmark level in Section 3.2. Figure 2 illustrates how this construction maps to concrete tasks. In the new-feature example, the task has one F-point: the delivered application must expose the requested Text component with the expected preview and controls. In the bug-fix example, the task also has one F-point: the Penkit entry must no longer crash, and the target feature module must open normally. The spec-driven example is different because the specification contains four required behaviors. Curators therefore define four F-points for the chat-input task: expanding the action tray from the “+” icon, expanding the input box for long text, showing the Send button and appending the sent message to the chat list, and restoring the “+” icon after the input is cleared. The spec-driven task is counted as complete only if the project builds and all four F-points pass.

3

B ENCHMARK S TATISTICS

We analyze OPENHARMONY BENCH at two levels. First, we report benchmark-wide composition and task-source scale using dimensions with a common denominator. We then examine workload scale, source concentration, difficulty, and aggregate Kit/API coverage for the benchmark as a whole. 3.1

B ENCHMARK C OMPOSITION AND S CALE

Table 2 provides a single statistical view of all three task sources. The benchmark contains 153 top-level inputs and 242 F-points drawn from 42 unique repositories. The repository total is a union: new-feature contributes 12 repositories, bug-fix contributes 31, these two sources share 11 repositories, and spec-driven contributes ten additional repositories, yielding 12+31−11+10 = 42 unique repositories. The spec-driven source has denser evaluation, with 2.78 F-points per top-level task on average, while each new-feature requirement and bug-fix bug report has one F-point. This difference reflects verification granularity rather than a change in the leaderboard unit: the leaderboard is scored over top-level tasks, and F-points are independently verifiable behavior checks within those tasks. Input size shows the same distinction. The median spec-driven task contains 589 non-whitespace characters, compared with 119 for new-feature and 107 for bug-fix. These values measure how much behavior is stated to the agent. They are not interpreted as direct difficulty scores. The median reference patches change four files and 88 lines for new-feature, 12 files and 489.5 lines for specdriven, and two files and 83 lines for bug-fix. The spec-driven values are computed over the 50 reference patches associated with the spec-driven inputs. The 50 spec-driven inputs are distinct task 6

Most frequent covered Kits 94.1%

ArkUI

Coverage buckets

35.9%

Basic Services

32.7%

Ability

29/39 Kits covered

ArkData

19.6%

Performance Analysis

17.6%

ArkTS

14.4%

Core File

13.1% 9.8%

Image Very frequent >=50% tasks: 1 Kit Frequent 10-49% tasks: 6 Kits Occasional 3-9.9% tasks: 7 Kits Rare <3% tasks: 15 Kits Uncovered 0 tasks: 10 Kits

ArkWeb

5.2%

Media Library

5.2%

0

20

40

60

80

100

Tasks covered (%)

Figure 3: Coverage of 39 high-frequency OpenHarmony Kits. The donut groups Kits by task-level coverage frequency, i.e., the percentage of the 153 golden patches in which each Kit appears. The bar chart shows the ten most frequent covered Kits. Coverage describes the platform surface represented by the workload, not model performance. specifications with separate task files and Hypium checks. When a single app state contains several independently verifiable behavior groups, curators may split the original specification into multiple smaller tasks to keep each task bounded and interpretable. Such tasks can share the same base and reference state, but they remain separate evaluation inputs because they expose different behavior requirements and different executable checks. This footprint describes the reference implementation associated with an evaluation input rather than an independent edit size for each F-point. Patch size also does not capture the repository exploration needed to diagnose a bug. Repository concentration reveals how broadly each task source samples the OpenHarmony ecosystem rather than over-representing a small number of projects. The new-feature source draws 31.2% of its inputs from its largest repository and 53.1% from its three largest repositories. The corresponding spec-driven shares are 38.0% and 74.0%. The bug-fix source is less concentrated, at 14.1% and 31.0%, and therefore provides the broadest repository spread among the three task sources. Across the complete benchmark, 58 inputs (37.9%) are labeled easy, 72 (47.1%) medium, and 23 (15.0%) hard. Difficulty is descriptive metadata assigned during curation from reference-patch size, changed-file count, cross-Kit/API involvement, and curator judgment of the development scenario. It summarizes benchmark composition and is not used as a calibrated performance predictor. 3.2

K IT /API C OVERAGE

Kit/API coverage characterizes the OpenHarmony platform surface represented by the benchmark. We analyze all instances against 39 high-frequency Kits commonly used in OpenHarmony application development. Their names follow the Huawei Developer taxonomy (Huawei Developer, 2026b). A Kit is counted for an input when the gold patch imports an @kit.* module or a recognized legacy @ohos.* module, or when it references a Kit-specific API class or method such as PixelMap, AVPlayer, or cameraManager. Tests, generated output, dependencies, documentation, and resources are excluded. The analysis is performed over the 153 task-level golden patches used by the benchmark. Overall, the benchmark covers 29 out of 39 high-frequency Kits (74.4%). Figure 3 explains this coverage in two ways. The number at the center of the left donut, “29/39 Kits covered”, reports the number of high-frequency Kits touched by at least one gold patch. In the donut, Kits are grouped by their task-level coverage frequency. A Kit is placed into exactly one bucket according to the percentage of the 153 benchmark tasks whose gold patch touches that Kit. “Very frequent” means the Kit appears in at least 50% of tasks, “Frequent” means 10–49%, “Occasional” means 3–9.9%, 7

“Rare” means greater than 0% but less than 3%, and “Uncovered” means the Kit appears in zero tasks. Under this definition, one Kit is very frequent, six are frequent, seven are occasional, fifteen are rare, and ten are uncovered. This shows that the benchmark has broad Kit coverage but that most covered Kits appear in relatively few tasks. The right bar chart lists the ten covered Kits with the highest task-level coverage frequency. Its horizontal axis, “Tasks covered (%)”, is the percentage of the 153 tasks whose gold patch touches the corresponding Kit. ArkUI appears in 144 of 153 tasks (94.1%), reflecting the fact that most patches involve ArkUI declarative UI components or DSL constructs such as struct, build(), $r(), and @Component. Basic Services Kit and Ability Kit are the next most common platform surfaces, appearing in 35.9% and 32.7% of tasks, followed by ArkData (19.6%), Performance Analysis Kit (17.6%), ArkTS (14.4%), and Core File Kit (13.1%). Image Kit, ArkWeb, and Media Library Kit complete the top-ten list but appear in fewer than 10% of tasks. A single task may exercise multiple Kits, so these percentages describe workload composition rather than mutually exclusive categories or model performance.

4

E VALUATION P IPELINE

4.1

P IPELINE OVERVIEW

O PEN H ARMONY B ENCH evaluates coding agents as end-to-end app developers. For each benchmark instance, the evaluation pipeline prepares a clean workspace, provides the task input to the agent, records the submitted code changes, builds the resulting OpenHarmony project, and then runs executable checks whenever the project is buildable. Every scored benchmark input has taskspecific executable checks; static and code-quality diagnostics are separate from the leaderboard score. This pipeline is shared across incremental feature implementation, specification-driven development, and bug fixing. The task input differs by source, but the delivered artifact is always a buildable ArkTS application project. The pipeline has six stages: 1. Workspace initialization. The pipeline initializes a clean workspace from the task scaffold or base project. 2. Agent execution. The agent receives the task description, relevant project context, system instruction, prompt template, and tool access, then produces a modified workspace. 3. Diff recording. The pipeline records whether the run produced a non-empty diff. This diagnostic separates no-output failures from later build or behavior failures, but it is not used as a correctness signal. 4. First build. The evaluation protocol records the first project build before any compilerepair step. 5. Compile repair. If the first build fails, the agent may enter a bounded compile-repair loop. The final submitted workspace is then built again. 6. Behavior validation. If the final build succeeds, the dynamic evaluation scripts install the application, launch it as required, and run the corresponding Python Hypium checks. 4.2

E VALUATION D IMENSIONS

No single evaluation dimension can adequately assess app-level code generation. A UI-only pass may miss incorrect persistence or platform integration, whereas a static-only pass cannot confirm whether user-visible behavior actually works. O PEN H ARMONY B ENCH therefore records complementary dimensions: buildability, dynamic behavior verification, and resource use. The current leaderboard is ranked by top-level Task Completion. Buildability and resource use are reported as diagnostics, while static functional coverage and code-quality checks are used during benchmark construction or retained as optional diagnostics when available. Buildability checks whether the delivered ArkTS project can be compiled into a runnable OpenHarmony application. Dynamic behavior verification executes scenario-specific UI tests on OpenHarmony emulators or devices. Resource metrics record the token use and agent wall-clock time 8

needed to produce the submitted workspace. Static functional coverage and code-quality checks are construction and diagnostic signals rather than leaderboard metrics. Only the build and dynamic behavior dimensions determine the current Task Completion scores. 4.3

E XECUTION E NVIRONMENT

All experiments are executed with DevEco 6.1 and API 23 on an OpenHarmony emulator configured with the Pura 90 Pro Max device profile, a resolution of 1308×2880, and a 6.9-inch screen. For compilation, we use the OpenHarmony API 23 SDK. The build pipeline is based on hvigorw Huawei Developer (2026c). Dynamic behavior verification uses OpenHarmony’s Hypium UI testing framework Huawei Developer (2026d). The dynamic evaluation scripts install application packages with hdc OpenHarmony (2026), launch the application when needed, and invoke the Python Hypium runner for the task-specific UI checks. When compile repair is enabled, we allow at most five compile-and-fix iterations per task. A repair iteration receives only build output, such as compiler diagnostics and build logs. Hypium outputs, oracle results, reference patches, and task-specific expected-check details are not returned to the agent during repair. Tokens and agent wall-clock time consumed by these repair iterations are included in the resource metrics reported in Table 4. The evaluated snapshot does not impose an additional benchmark-level token or wall-clock cap beyond the shared five-iteration repair limit and provider/tool constraints. If compilation still fails after the allowed repair budget, the case is marked as a build failure and receives zero dynamic passes. This separation lets us distinguish generation failures, build failures, and behavioral failures after a successful build. 4.4

AGENTS AND M ODELS

Unless otherwise stated, all reported results use DevEco Code OpenHarmony-SIG (2026) as the fixed agent framework. The main comparison varies the underlying model, identified by the display names configured in DevEco Code: GLM-5.1 and GLM-5.2 (Zhipu AI, 2026a;b), Qwen3.7Max and Qwen3.8-Max-Preview (Alibaba Cloud, 2026b;a), Kimi-K2.7-Code and Kimi-K3 (Moonshot AI, 2026b;a), MiniMax-M3 (MiniMax, 2026), and DeepSeek-V4-Pro (Alibaba Cloud, 2026b; DeepSeek, 2026). This organization focuses the current report on model differences under one shared OpenHarmony coding-agent scaffold. All runs for a given comparison use the same task split, system instruction, prompt template, tool access, environment configuration, build scripts, and scoring scripts. This keeps the DevEco Code agent framework fixed and isolates the effect of the underlying model as much as possible. The main result table reports the concrete model cells evaluated over the complete 153-task suite. Results should therefore be read as DevEco Code–model configurations rather than as a comparison of multiple independent agent frameworks. 4.5

E VALUATION S NAPSHOT AND R EPRODUCIBILITY

Table 3 summarizes the exact snapshot evaluated in this report. We report these identifiers because leaderboard scores depend not only on the model, but also on the task set, dynamic evaluation scripts, agent framework, SDK, device profile, and run configuration.

9

Item

Snapshot used in this report

Benchmark version Task snapshot date Official project URL Number of top-level tasks Task sources Executable F-points DevEco Code version Prompt/template

OpenHarmony Bench v1.0 2026.7.31 https://bench.matrix.openharmony.cn/ 153 new-feature 32; spec-driven 50; bug-fix 71 242 total; spec-driven 139 0.1.0-1.3 (1) Task definition; (2) Spec content and environment; (3) Code generation instructions Temperature=0, Context-window=200K, Reasoning-budget=65.5K API 23 DevEco 6.1 Pura 90 Pro Max emulator, 1308×2880, 6.9-inch 3 independent full-suite runs; at most 5 compile-repair iterations per task

Run parameters OpenHarmony SDK DevEco Studio / toolchain Device profile Run configuration

Table 3: Evaluation snapshot used for the results in this report. Open artifacts and contamination safeguards. The public release includes the task descriptions, benchmark metadata, reference patches or solutions, executable tests, evaluation scripts, and aggregation scripts. This open design allows developers to inspect task quality, reproduce scores, and run additional Agent–LLM configurations. Public release may introduce training-contamination risk over time, and the bug-fix source may also carry pre-training exposure risk because it is derived from historical public pull requests. We therefore treat the reported numbers as a timestamped snapshot, record benchmark versions and evaluation script snapshots for every leaderboard entry, and recommend that future evaluations use updated task snapshots and trajectory audits to check for reward hacking or overfitting to released tests. 4.6

M ETRICS

We use top-level Task Completion as the primary effectiveness metric and leaderboard score. One evaluation task is one benchmark input: a new-feature request, a spec-driven task, or a bug-fix input. A task is marked complete only when the final project builds and all executable checks associated with that input pass. Buildability is therefore necessary but not sufficient for completion: a project can compile successfully while still failing required UI behavior, state transitions, persistence writes, or platform integration. The spec-driven source still contains multiple F-point checks inside each task; a spec-driven task is marked complete only if all F-point checks for that task pass. This toplevel definition is stricter than counting individual F-point checks independently because a task is marked complete only when all of its checked behaviors pass. It mirrors the all-tests task success view used in repository-level benchmarks. The metric has an explicit floor and ceiling in the released snapshot. A no-op baseline that submits the base or buggy project unchanged completes 0 of the 153 top-level tasks, because every retained executable check is required to fail before the reference change and pass after it. The reference solutions define a 100% ceiling: a task is admitted only when its reference solution builds and all retained executable checks pass under curator validation. For the current benchmark, the overall leaderboard aggregates 153 top-level tasks: 32 new-feature tasks, 50 spec-driven tasks, and 71 bug-fix tasks. Each DevEco Code–LLM configuration is evaluated with three independent full-suite runs. We compute every rate separately for each run and then report the arithmetic mean across the three runs. Tables and prose report rates as percentages. For a reported scope S (either all 153 tasks or one task source), let N = |S| and k = 3. For first final configuration c and run r, let Cc,r be the number of completed tasks in S, and let Bc,r and Bc,r be the numbers of tasks whose first and final build attempts succeed.

10

Task Completion. Task Completion is the fraction of top-level tasks that build and pass all required executable checks in a run: Cc,r TCc,r (S) = , N

k

1X TCc (S) = TCc,r (S). k r=1

The main leaderboard reports TCc (S) with S equal to all 153 tasks. Source-level tables apply the same rule with S restricted to one task source. This run-level averaging treats each completesuite execution as one repeated measurement of the same Agent–LLM configuration, and each run contributes equally to the reported mean. Run range. Because each configuration is evaluated three times, we report the observed minimum and maximum run-level Task Completion beside the mean score:   Rangec (S) = min TCc,r (S), max TCc,r (S) . r∈{1,...,k}

r∈{1,...,k}

This range is descriptive rather than an inferential interval, and no best-of-three rule is applied. For the overall 153-task table, each full-suite run first aggregates all task sources, and the minimum and maximum are then taken across the three overall run scores. The overall range is not formed by independently combining task-source-level extrema. Final Build Success Rate. Final Build Success Rate measures whether the delivered workspace builds after the complete agent run and any allowed compile-repair iterations. It is the fraction of tasks whose submitted ArkTS project builds successfully at the end of the agent run, averaged across the three runs: k final Bc,r 1X FBRc,r (S) = FBRc,r (S). , FBRc (S) = N k r=1 The Final Build column in Table 4 reports this metric over all 153 tasks. First Build Success Rate. First Build Success Rate measures whether the project builds successfully on the first build attempt, before any compile-repair iteration. The runner records this first build outcome before source repair so that it can be distinguished from the final build outcome: IBRc,r (S) =

first Bc,r , N

k

IBRc (S) =

1X IBRc,r (S). k r=1

The First Build column reports the mean first build success rate across the three runs. Run-level resource metrics. We report token usage and agent wall-clock time as descriptive resource dimensions for every Agent–LLM configuration. For each complete-suite run, we first average the raw per-task resource values over all 153 inputs and then average those per-run values across (q) the three runs. For resource dimension q (tokens or agent wall-clock time), let xc,r,i be the raw value for task i in run r: k 1 X (q) 1 X (q) (q) (q) Xc,r = xc,r,i , Xc = X . N k r=1 c,r i∈S

For presentation, Table 4 converts tokens to millions per task and agent wall-clock time to seconds per task. Token usage is the provider-recorded total tokens value accumulated over every conversation turn. It includes cache-read or cache-creation tokens when the provider reports them in total tokens; component fields are retained in the raw records but are not added again. Agent wall-clock time is measured from the start of the Agent process until it completes. It includes model interactions, tool use, source modification, and any builds initiated by the Agent. Post-run validation builds, installation, and Hypium execution are part of correctness evaluation but are excluded from agent wall-clock time. This time definition deliberately measures the execution time of the complete Agent–LLM system, including local Agent actions. Because agent wall-clock time includes local tool use, source modification, and builds, it can vary with hardware performance, build-server load, filesystem state, and 11

emulator/device state. We therefore use wall-clock time as a within-environment diagnostic and do not treat it as directly comparable across different execution environments. These dimensions measure gross resource use rather than normalized efficiency. A configuration that fails early can appear faster or cheaper because it does less work. We therefore interpret wall-clock time and token use together with Task Completion.

5

R ESULTS

Figure 4 visualizes the main ranking, and Table 4 reports the full evaluation matrix. Each row represents one DevEco Code–LLM configuration evaluated on all 153 tasks. Task Completion reports the arithmetic mean across three independent full-suite runs, with the run-level minimum and maximum shown in brackets. Build and resource columns are averaged over the same three runs. 1 GLM-5.2

58.39

2 Kimi-K3

57.30

3 Qwen3.8-Max-Preview

55.99

4 Qwen3.7-Max

53.38

5 GLM-5.1

52.29

6 Kimi-K2.7-Code

52.07

7 DeepSeek-V4-Pro

52.07

8 MiniMax-M3

48.36

44

48

52

56

60

64

Task Completion (%)

Figure 4: Overall ranking over all 153 OpenHarmony tasks. Bars show mean Task Completion across three independent full-suite runs, and whiskers show the observed run-level min–max range. The x-axis starts at 44% to focus on the observed score range.

Rank 1 2 3 4 5 6 7 8

Model

Task Completion First Build mean [min, max] (%) (%)

GLM-5.2 Kimi-K3 Qwen3.8-Max-Preview Qwen3.7-Max GLM-5.1 Kimi-K2.7-Code DeepSeek-V4-Pro MiniMax-M3

58.39 [56.21, 60.78] 57.30 [53.59, 62.09] 55.99 [54.90, 57.52] 53.38 [51.63, 55.56] 52.29 [50.33, 54.90] 52.07 [50.98, 52.94] 52.07 [50.33, 54.25] 48.36 [45.75, 49.67]

67.10 87.58 80.61 69.06 73.20 62.53 64.71 59.26

Final Build Tokens/Task (%) (M) 98.04 98.91 99.56 98.47 99.57 99.35 100.00 94.77

2.57 1.02 0.98 1.27 1.60 1.88 1.33 3.69

Time/Task (s) 1936.4 652.3 394.3 738.3 1097.8 767.5 1048.6 870.8

Table 4: Main results over all 153 OpenHarmony tasks. Task Completion reports the mean across three independent runs, with the run-level min–max range shown in brackets. First Build and Final Build are agent-run build outcomes averaged over the same runs. Tokens/Task uses providerrecorded total tokens; Time/Task reports mean per-task agent wall-clock time. 5.1

OVERALL P ERFORMANCE

Across three independent full-suite runs over all 153 tasks, DevEco Code with GLM-5.2 achieves the highest mean Task Completion at 58.39% (approximately 89.3/153 tasks). Kimi-K3 ranks second at 57.30%, and Qwen3.8-Max-Preview ranks third at 55.99%. These values are arithmetic means 12

of run-level rates and do not apply a best-of-three rule. The bracketed ranges in Table 4 show the minimum and maximum run-level Task Completion values for each model. The rank column should be read as a descriptive ordering for this three-run snapshot rather than as a strict statistical separation between adjacent configurations. The observed run range of Kimi-K3, [53.59, 62.09], contains the mean of GLM-5.2, and Kimi-K2.7-Code and DeepSeek-V4-Pro differ only beyond the two decimal places shown in the table. The table therefore supports a leading group view: GLM-5.2, Kimi-K3, and Qwen3.8-Max-Preview form the top group under the current DevEco Code setting, while adjacent fine-grained ranks should be interpreted with caution. 5.2

M ODEL P ERFORMANCE

Task Completion ranges from 48.36% for MiniMax-M3 to 58.39% for GLM-5.2. The corresponding mean Final Build Success Rate ranges from 94.77% for MiniMax-M3 to 100.00% for DeepSeekV4-Pro. These results show that a high final build rate does not by itself imply high behavioral correctness: DeepSeek-V4-Pro achieves the highest Final Build Success Rate, while GLM-5.2 achieves the highest overall Task Completion. Within the evaluated model families under the fixed DevEco Code setting, the newer model variants obtain higher mean Task Completion than their earlier counterparts. This observation is limited to the evaluated configurations and does not by itself establish a general trend across all OpenHarmony coding agents or model families. The complete 153-task result combines the new-feature, spec-driven, and bug-fix task sources using the same top-level Task Completion definition. We use this overall task-level aggregation as the primary leaderboard unit. We also report the three task sources separately to explain where model differences arise. 5.3

TASK -S OURCE R ESULTS

Figure 5 and Tables 5–7 decompose the 153-task result into the three task sources. These sourcelevel results are diagnostic rather than separate benchmark leaderboards: the main ranking remains the 153-task aggregation in Table 4. The decomposition follows the report-wide task-source order: new-feature, spec-driven, and bug-fix. In the current snapshot, bug-fix tasks have the highest observed completion rates, new-feature tasks sit in the middle, and spec-driven tasks have the lowest observed completion rates under the task-level completion metric. 74.2 75.1 70.0 68.5 63.5

ax niM Mi

2.7

eek

i-K

epS

Kim

De

.7

5.1

en3

GL M-

Qw

3

.8 en3 Qw

5.2

i-K Kim

GL M-

ax niM

eek

28.0

Mi

2.7 i-K

epS

Kim

.7

spec-driven

24.0

De

GL M-

en3

28.7 27.3

5.1

30.7

Qw

3

.8

i-K

5.2

34.7 34.0

Kim

GL M-

ax niM Mi

2.7

eek

i-K

epS

Kim

new-feature

De

.7

5.1

en3

GL M-

Qw

3

.8 en3 Qw

i-K

5.2

32.7

GL M-

70.4

53.1

40

20

67.1 67.6

60.6 55.2 56.2 56.2 55.2

en3

53.1

Qw

59.4

60

Kim

Task Completion (%)

80

bug-fix

Figure 5: Task Completion by task source for the evaluated DevEco Code–LLM configurations. Each vertical bar represents one configuration. Each panel uses the same model order and the same y-axis range; bars report mean Task Completion across three independent full-suite runs. The y-axis starts at 20% to focus on the observed score range. New-feature tasks require agents to implement requirement-style changes in existing OpenHarmony applications. GLM-5.2 achieves the highest mean Task Completion, but it also uses substantially more time than most other configurations. Qwen3.8-Max-Preview ranks second while keeping both token use and wall-clock time low. It completes 19 of 32 tasks in each of the three runs, which is

13

Rank 1 2 3 4 5 6 7 8

Model

Task Completion First Build mean [min, max] (%) (%)

GLM-5.2 Qwen3.8-Max-Preview GLM-5.1 Kimi-K2.7-Code Qwen3.7-Max DeepSeek-V4-Pro MiniMax-M3 Kimi-K3

63.54 [59.38, 65.62] 59.38 [59.38, 59.38] 56.25 [53.12, 59.38] 56.25 [53.12, 59.38] 55.21 [46.88, 62.50] 55.21 [53.12, 56.25] 53.13 [43.75, 59.38] 53.13 [43.75, 59.38]

60.42 91.66 89.58 71.88 72.92 61.46 61.46 83.33

Final Build Tokens/Task (%) (M) 96.88 100.00 100.00 100.00 100.00 100.00 93.75 100.00

2.08 1.01 1.42 1.77 1.07 1.00 3.57 0.84

Time/Task (s) 1597.0 359.3 941.0 521.7 368.0 328.3 463.4 633.4

Table 5: Results on the new-feature task source, covering 32 tasks. why its new-feature run range has the same minimum and maximum. The wider min–max ranges for several models show that this source is more sensitive to run-level variation than bug-fix.

Rank 1 2 3 4 5 6 7 8

Model

Task Completion First Build mean [min, max] (%) (%)

Kimi-K3 Qwen3.8-Max-Preview GLM-5.2 Qwen3.7-Max GLM-5.1 MiniMax-M3 Kimi-K2.7-Code DeepSeek-V4-Pro

34.67 [28.00, 40.00] 34.00 [32.00, 36.00] 32.67 [30.00, 34.00] 30.67 [28.00, 32.00] 28.67 [28.00, 30.00] 28.00 [28.00, 28.00] 27.33 [26.00, 28.00] 24.00 [20.00, 26.00]

92.67 68.00 63.33 62.67 57.33 50.00 53.33 61.33

Final Build Tokens/Task (%) (M) 96.67 100.00 100.00 100.00 99.33 98.67 100.00 100.00

1.54 1.34 4.47 2.06 2.36 5.56 2.86 2.17

Time/Task (s) 928.8 610.6 2971.6 1633.9 1857.9 1746.7 1412.7 2580.0

Table 6: Results on the spec-driven task source, covering 50 tasks. Spec-driven has the lowest observed completion under top-level completion scoring. Its mean Task Completion ranges from 24.00% to 34.67%, lower than the other two sources. This gap is consistent with the stricter task-level rule for this source: a spec-driven task is marked complete only when all of its required F-point checks pass. The source therefore stresses precise specification following and end-to-end UI behavior, not just buildability: several configurations reach a 100% mean Final Build Success Rate while remaining below 35% Task Completion. This source-level comparison should be read at the common top-level task unit. Spec-driven tasks contain multiple F-points, whereas new-feature and bug-fix tasks contain one executable check each. We therefore report source-level results with the same task-completion rule used by the main leaderboard, rather than adding a separate F-point-level ranking that would not align with the other two task sources. Bug-fix has the highest observed completion rates in the current benchmark snapshot. Kimi-K3 and GLM-5.2 form the leading group, both exceeding 74% mean Task Completion. DeepSeek-V4Pro and Qwen3.8-Max-Preview follow closely, with Qwen3.8-Max-Preview using the lowest mean tokens and wall-clock time on this source. The high final build rates indicate that remaining failures are mostly behavioral rather than compilation failures. 5.4

B UILDABILITY AND F UNCTIONAL C ORRECTNESS

Final Build and Task Completion measure different outcomes. Across the evaluated DevEco Code configurations, mean Final Build Success Rate ranges from 94.77% to 100.00%, while mean Task Completion ranges from 48.36% to 58.39%. The gap indicates that buildability is necessary but not sufficient: many submitted projects compile after repair but still fail to satisfy the requested behavior-level checks.

14

Rank 1 2 3 4 5 6 7 8

Model

Task Completion First Build mean [min, max] (%) (%)

Kimi-K3 GLM-5.2 DeepSeek-V4-Pro Qwen3.8-Max-Preview Qwen3.7-Max Kimi-K2.7-Code GLM-5.1 MiniMax-M3

75.12 [70.42, 78.87] 74.18 [71.83, 77.46] 70.42 [66.20, 73.24] 69.95 [66.20, 73.24] 68.54 [66.20, 70.42] 67.61 [67.61, 67.61] 67.14 [64.79, 71.83] 60.56 [59.15, 61.97]

85.91 72.77 68.54 84.51 71.83 64.79 77.00 64.79

Final Build Tokens/Task (%) (M) 100.00 97.18 100.00 99.06 96.71 98.59 99.53 92.49

0.74 1.46 0.89 0.71 0.80 1.24 1.14 2.43

Time/Task (s) 466.1 1360.3 294.8 257.6 274.5 423.9 633.2 437.4

Table 7: Results on the bug-fix task source, covering 71 tasks. The highest mean First Build Success Rate is 87.58% for Kimi-K3, while the lowest is 59.26% for MiniMax-M3. For the leading overall configuration, GLM-5.2, the mean First Build Success Rate is 67.10%, mean Final Build Success Rate is 98.04%, and mean Task Completion is 58.39%. These values indicate how often compile repair recovers first-build failures and how often a buildable result also implements the required interaction, state transition, and UI behavior. 5.5

R ESOURCE U SE AND RUN - TO -RUN VARIATION

Mean per-task resource use varies across DevEco Code configurations. The lowest mean token use is 0.98 million tokens per task for Qwen3.8-Max-Preview, while the highest is 3.69 million tokens per task for MiniMax-M3. The lowest mean agent wall-clock time is 394.3 seconds per task for Qwen3.8-Max-Preview, while the highest is 1936.4 seconds per task for GLM-5.2. The leading effectiveness configuration, GLM-5.2, uses a mean of 2.57 million tokens and 1936.4 seconds per task. Token counts are not directly comparable across models, since each provider may use a different tokenizer and accounting convention for cached prefixes; the column indicates order of magnitude rather than a like-for-like ratio. Raw resource totals must be interpreted together with effectiveness because a configuration that stops early can consume fewer resources by completing less work. The bracketed ranges in Table 4 summarize observed run-to-run variability across the three full-suite runs. We report this descriptive min–max range instead of an inferential interval because the current evaluation uses three repeated runs per configuration.

6

R ELATED W ORK

Code and repository-level benchmarks. Early code benchmarks evaluate self-contained synthesis from docstrings, signatures, or programming-problem statements, including HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), and APPS (Hendrycks et al., 2021). CodeXGLUE (Lu et al., 2021) broadens the scope to multiple code-understanding and generation tasks, while DS1000 (Lai et al., 2023) and BigCodeBench (Zhuo et al., 2025) stress library use and complex instructions. Repository-level benchmarks such as SWE-bench (Jimenez et al., 2024), SWE-agent (Yang et al., 2024), Agentless (Xia et al., 2025), SWE-bench Multimodal (Yang et al., 2025), Multi-SWEbench (Zan et al., 2025), and SWE-Compass (Xu et al., 2025) move toward realistic issue resolution and agentic coding. These benchmarks are important for measuring program synthesis, bug fixing, and repository maintenance, but they do not directly evaluate mobile app feature implementation from behavioral specifications in a runnable platform-specific project. O PEN H ARMONY B ENCH targets this missing setting for OpenHarmony ArkTS applications. Mobile, GUI, and app-generation benchmarks. Mobile-agent benchmarks such as Android in the Wild (Rawles et al., 2023), AndroidWorld (Rawles et al., 2025), SPA-Bench (Chen et al., 2025), Mobile-Agent (Wang et al., 2024), B-MoCA (Lee et al., 2025), and Mobile-Bench (Deng et al., 2024) evaluate agents operating existing apps through device or GUI actions. Mobile GUI datasets also support data-driven research on interface understanding and cross-device GUI development (Hu et al., 2023). MobileDev-Bench (Fakorede et al., 2026) brings issue resolution to Android Native, 15

React Native, and Flutter apps. On the generation side, WebApp1K (Cui, 2024), DesignBench (Xiao et al., 2025), WebCoderBench (Liu et al., 2026), PlayCoder/PlayEval (Peng et al., 2026), and AppEvalPilot (Bian et al., 2025) study web, front-end, GUI, or generated-application evaluation. Mobile app quality can also depend on performance-sensitive build and binary properties, such as compiler optimization settings in native libraries (Hu et al., 2026). These efforts share our interest in interactive behavior, but most evaluate either action completion in existing apps, issue-level patch correctness, or coarse generated-application quality. They do not test whether an agent can modify OpenHarmony source code so that a specified app feature becomes buildable and behaviorally correct. OpenHarmony-specific evaluation. Work targeting the OpenHarmony ecosystem remains sparse. ArkEval (Xie et al., 2026) benchmarks automated repair for ArkTS, and framework-aware code generation studies synthesize training data from API knowledge graphs to improve OpenHarmony code generation (Liu et al., 2025). These efforts operate mainly at the function, API, or repair level. In contrast, OpenHarmony app development requires coordination across ArkUI components, routing, state, persistence, platform APIs, command-line build tooling, device connectors, and UI testing infrastructure (Huawei Developer, 2026a;c; OpenHarmony, 2026; Huawei Developer, 2026d). Public benchmark coverage remains limited for app-level feature implementation in complete OpenHarmony projects. Benchmark gap. Across these lines, existing evaluation typically reduces to final patch correctness, outcome-level UI task completion, or broad generated-application quality. O PEN H ARMONY B ENCH addresses a different gap: it evaluates source-level implementation of platform-specific app features from behavioral task inputs. For the spec-driven source, each task is decomposed into fine-grained F-points, and observable behavior is checked through executable OpenHarmony UI tests. Static functional coverage and code-quality assessment are retained as construction-time checks or optional diagnostics rather than primary leaderboard metrics (subsection 4.2).

7

L IMITATIONS

The current report has three main limitations. First, all reported results use DevEco Code as the fixed agent framework. This design keeps the agent scaffold, tool access, prompt template, build scripts, and scoring pipeline fixed, so the main comparison isolates differences among the evaluated underlying models as much as possible. At the same time, the reported numbers should be interpreted as DevEco Code–model configurations rather than as a direct comparison of multiple independent coding-agent frameworks. Second, each configuration is evaluated with three independent full-suite runs. This repeated-run design is sufficient to report an observed run-level range and to avoid a best-of-three score, but it does not provide an inferential confidence interval or a statistical separation between adjacent leaderboard entries. Fine-grained rank differences should therefore be read cautiously, especially when run-level ranges overlap or rounded mean values are close. Third, the bug-fix source is derived from historical public pull requests. This choice grounds the tasks in real OpenHarmony development, but it also means that possible pre-training exposure cannot be excluded for that source. We therefore treat the reported numbers as a dated benchmark snapshot and recommend recording benchmark versions, task snapshots, evaluation-script snapshots, and model identifiers for future leaderboard entries.

8

C ONCLUSION

We introduced O PEN H ARMONY B ENCH, an app-level benchmark for evaluating LLM-based coding agents on OpenHarmony ArkTS development. The benchmark spans three complementary task sources: new-feature, spec-driven, and bug-fix. Together, they contain 153 top-level inputs and 242 F-points across 42 OpenHarmony repositories. The leaderboard is scored over top-level tasks; F-points provide the executable verification granularity within those tasks.

16

O PEN H ARMONY B ENCH evaluates each Agent–LLM configuration through a shared build-and-test pipeline. Task Completion measures whether each top-level task builds and satisfies all required executable checks. In the spec-driven source, this means all F-point checks inside a task must pass. First Build Success Rate, Final Build Success Rate, and run-level resource metrics provide complementary views of delivery reliability and evaluation resource use. This design separates code that merely builds from code that implements the required application behavior. Each Agent–LLM configuration is evaluated through three independent full-suite runs. We compute every effectiveness and build rate separately for each run and report the arithmetic mean. Over the combined 153-task suite, DevEco Code with GLM-5.2 achieves the highest mean Task Completion at 58.39%, with an observed run-level range of [56.21%, 60.78%]. Kimi-K3 and Qwen3.8-MaxPreview follow at 57.30% and 55.99%, respectively. These results show substantial model- and task-level variation under the fixed DevEco Code agent framework used in this report. The leading configuration uses a mean of 2.57 million tokens and 1936.4 seconds per task. Qwen3.8Max-Preview uses the fewest tokens and shortest wall-clock time in the current table, at 0.98 million tokens and 394.3 seconds per task. The results also quantify the relationship between buildability and functional correctness. Mean Final Build Success Rate ranges from 94.77% to 100.00%, while mean Task Completion ranges from 48.36% to 58.39%. Successful compilation does not by itself demonstrate correct UI behavior, state transitions, persistence, or platform integration. Executable application-level checks are therefore necessary for measuring whether a coding agent has completed the requested work. By combining realistic OpenHarmony projects, F-points, and Agent–LLM evaluation, O PEN H ARMONY B ENCH supports reproducible comparison of incremental implementation, specification following, and bug fixing in the OpenHarmony ecosystem.

C ONTRIBUTORS Li Li, Han Hu, Tianjian Zhang, Xin Peng, Fangzhu Mao, Qingyu Zhang, Xiaoheng Xie, Zhongmin Tang, Zhihao Lin, Haolin Ruan, Miaomiao Dong, Liuchuan Zhu, Yue Li, Chi Chen, Wenkang Zhong, Mingfei Zhang, Yang Yu, Bo Sun, Chaorui Zhang, Weixi Zhang, Wei Han, Bo Bai, Kui Liu, Gang Fan, Siru Liu, Jiaqian Zhou, Jiali Sun, Yunbiao Dong, Wenhao Zhong, Yunhong Xu.

ACKNOWLEDGMENTS We thank the Huawei Hong Kong Research Center, the Huawei Developer Platform Department, and related experts for their valuable guidance on dataset design, real-world OpenHarmony application evaluation, and trajectory analysis. In particular, we thank Ning Jiang, Jiamin Wu, Jinyu Chen, Heqing Huang, and Yu Liu for their helpful feedback and support.

17

R EFERENCES Alibaba Cloud. Alibaba cloud model studio. https://modelstudio.alibabacloud. com/, 2026a. Accessed: 2026-08-17. Alibaba Cloud. Alibaba cloud model studio: Recommended models. https://www. alibabacloud.com/help/en/model-studio/models, 2026b. Accessed: 2026-0817. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, Xiangru Tang, Yongxin Ni, Sirui Hong, and Chenglin Wu. You don’t know until you click: Automated GUI testing for production-ready software evaluation. arXiv preprint arXiv:2508.14104, 2025. Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, and Kun Shao. SPA-Bench: A comprehensive benchmark for smartphone agent evaluation. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. arXiv:2410.15164. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Yi Cui. WebApp1K: A practical code-generation benchmark for web app development. arXiv preprint arXiv:2408.00019, 2024. DeepSeek. Transparency center: Published models. https://www.deepseek.com/en/ transparency/, 2026. Accessed: 2026-08-17. Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile-Bench: An evaluation benchmark for LLM-based mobile agents. arXiv preprint arXiv:2407.00993, 2024. Moshood A. Fakorede, Krishna Upadhyay, A. B. Siddique, and Umar Farooq. MobileDev-Bench: A comprehensive benchmark for evaluating language models on mobile application development. arXiv preprint arXiv:2603.24946, 2026. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. Measuring coding challenge competence with APPS. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. Han Hu, Haolan Zhan, Yujin Huang, and Di Liu. Pairwise GUI dataset construction between Android phones and tablets. In Advances in Neural Information Processing Systems 36, 2023. URL https://openreview.net/forum?id=8gDJXL652A. Han Hu, Xiaoheng Xie, Bo Sun, Jian Gu, Gang Fan, and Li Li. Source-free detection and impact analysis of compiler optimization problems in mobile applications. In Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, 2026. To appear. arXiv:2606.23512. Huawei Developer. ArkUI application framework. https://developer.huawei.com/ consumer/en/doc/harmonyos-guides/arkui, 2026a. Accessed: 2026-06-25. Huawei Developer. HarmonyOS NEXT development documentation. https://developer. huawei.com/consumer/cn/doc/, 2026b. Accessed: 2026-07-13. Huawei Developer. Using hvigorw. https://developer.huawei.com/consumer/en/ doc/harmonyos-guides/ide-hvigor-commandline, 2026c. Accessed: 2026-06-25. 18

Huawei Developer. UITest user guide. https://developer.huawei.com/consumer/ en/doc/harmonyos-guides/uitest-guidelines, 2026d. Accessed: 2026-06-25. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In Proceedings of the International Conference on Learning Representations (ICLR), 2024. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen tau Yih, Daniel Fried, Sida Wang, and Tao Yu. DS-1000: A natural and reliable benchmark for data science code generation. In Proceedings of the International Conference on Machine Learning (ICML), 2023. Juyong Lee, Taywon Min, Minyong An, Dongyoon Hahm, Haeone Lee, Changyeon Kim, and Kimin Lee. Benchmarking mobile device control agents across diverse configurations. In Proceedings of the Conference on Lifelong Learning Agents (CoLLAs), 2025. Chenxu Liu, Yingjie Fu, Wei Yang, Ying Zhang, and Tao Xie. WebCoderBench: Benchmarking web application generation with comprehensive and interpretable evaluation metrics. arXiv preprint arXiv:2601.02430, 2026. Mingwei Liu, Zheng Pei, Yanlin Wang, Zihao Wang, Zikang Li, Enci Lin, Xin Peng, and Zibin Zheng. Framework-aware code generation with API knowledge graph–constructed data: A study on OpenHarmony. arXiv preprint arXiv:2512.00380, 2025. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin B. Clement, Dawn Drain, Daxin Jiang, Duyu Tang, Ge Li, Lidong Zhou, Linjun Shou, Long Zhou, Michele Tufano, Ming Gong, Ming Zhou, Nan Duan, Neel Sundaresan, Shao Kun Deng, Shengyu Fu, and Shujie Liu. CodeXGLUE: A machine learning benchmark dataset for code understanding and generation. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2021. MiniMax. MiniMax M3. https://www.minimaxi.com/models/text/m3, 2026. Accessed: 2026-08-17. Moonshot AI. Kimi code docs: Model configuration. https://www.kimi.com/code/docs/ en/kimi-code/models.html, 2026a. Accessed: 2026-08-17. Moonshot AI. Kimi K2.7 Code: Open-source agentic coding model. https://www.kimi. com/resources/kimi-k2-7-code, 2026b. Accessed: 2026-08-17. OpenHarmony. HDC: Openharmony device connector. https://github.com/ openharmony/developtools_hdc, 2026. GitHub repository. Accessed: 2026-06-25. OpenHarmony-SIG. Deveco code: An open-source ai agent for openharmony application development. https://gitcode.com/openharmony-sig/deveco-code/tree/ develop, 2026. Accessed: 2026-06-19. Zhiyuan Peng, Wei Tao, Xin Yin, Chenhao Ying, Yuan Luo, and Yiwen Guo. PlayCoder: Making LLM-generated GUI code playable. arXiv preprint arXiv:2604.19742, 2026. Christopher Rawles, Alice Li, Daniel Rodriguez, Oriana Riva, and Timothy Lillicrap. Android in the Wild: A large-scale dataset for android device control. In Advances in Neural Information Processing Systems, Datasets and Benchmarks Track, 2023. Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Marybeth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, Daniel Toyama, Robert Berry, Divya Tyamagundlu, Timothy Lillicrap, and Oriana Riva. AndroidWorld: A dynamic benchmarking environment for autonomous agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2025. arXiv:2405.14573. Junyang Wang, Haiyang Xu, Jiabo Ye, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, and Jitao Sang. Mobile-Agent: Autonomous multi-modal mobile device agent with visual perception. In Proceedings of the International Conference on Learning Representations (ICLR), 2024. 19

Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents. Proceedings of the ACM on Software Engineering, 2 (FSE):801–824, 2025. Jingyu Xiao, Ming Wang, Man Ho Lam, Yuxuan Wan, Junliang Liu, Yintong Huo, and Michael R. Lyu. DesignBench: A comprehensive benchmark for MLLM-based front-end code generation. arXiv preprint arXiv:2506.06251, 2025. Bang Xie, Senjian Zhang, Zhiyuan Peng, Wei Chen, Chenhao Ying, and Yuan Luo. ArkEval: Benchmarking and evaluating automated code repair for ArkTS. arXiv preprint arXiv:2602.08866, 2026. Jingxuan Xu, Ken Deng, Weihao Li, Songwei Yu, Huaixi Tang, Haoyang Huang, Zhiyi Lai, Zizheng Zhan, Yanan Wu, Chenchen Zhang, et al. SWE-Compass: Towards unified evaluation of agentic coding abilities for large language models. arXiv preprint arXiv:2511.05459, 2025. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. SWE-bench Multimodal: Do AI systems generalize to visual software domains? In Proceedings of the International Conference on Learning Representations (ICLR), 2025. Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-SWE-bench: A multilingual benchmark for issue resolving. arXiv preprint arXiv:2504.02605, 2025. Zhipu AI. GLM-5.1: Model documentation. https://docs.bigmodel.cn/cn/guide/ models/text/glm-5.1, 2026a. Accessed: 2026-08-17. Zhipu AI. GLM-5.2: Model documentation. https://docs.bigmodel.cn/cn/guide/ models/text/glm-5.2, 2026b. Accessed: 2026-08-17. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, et al. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.

20

Related documents

• OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development#617695
• OpenHarmony Bench: Evaluating LLMs and Coding Agents on OpenHarmony App Development#617195
• LLM-based Hardware Development with Hierarchical IRs and End-to-End Multi-Agent Workflow#818715
• LLM-based Hardware Development with Hierarchical IRs and End-to-End Multi-Agent Workflow#808723
• Using LLMs in Software Design: An Empirical Study of GitHub and A Practitioner Survey#1036689