SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents

SynWeaver: Website-Prior Task and Trajectory Co-Synthesis for Web Agents Ruitao Wang and Yuwen Hao and Menglin Yang† Hong Kong University of Science and Technology (Guangzhou) {rwang356, yhao481}@connect.hkust-gz.edu.cn menglinyang@hkust-gz.edu.cn †

Corresponding author.

arXiv:2608.12429v1 [cs.SE] 12 Aug 2026

Abstract

seen websites, as multimodal models lack websitespecific priors about page structure, interaction affordances, and action consequences (Tang et al., 2025). Therefore, web-agent data synthesis is important not only for alleviating the supervision bottleneck, but also for injecting target-website knowledge that improves generalization. Existing web agent data synthesis methods can be broadly divided into two families. The first family leverages external resources, such as tutorials or documentation, to instantiate tasks and replay trajectories (Zhang et al., 2026a; Xu et al., 2025b; Su et al., 2025; Shen et al., 2024). These methods can provide useful supervision when such resources are abundant, but their coverage is fundamentally bounded by what already exists in the internet. For many websites, relevant materials are sparse, outdated, or entirely unavailable (Mereu et al., 2026). In this work, we instead focus on the second family, which mines tasks directly from website interaction and is therefore applicable even when external artifacts are missing (Sun et al., 2025a; Pahuja et al., 2025; Gandhi and Neubig, 2025). Within this family, a central challenge is generating tasks that are both realistic and executable. Some methods synthesize tasks from local interactions, which is efficient but often short-sighted: lacking sufficient website-specific knowledge (Shi et al., 2025), the model may propose tasks that appear plausible on the current page yet become infeasible during subsequent interaction. NNetNav (Murty et al., 2024) alleviates this issue through multi-step LLM exploration, but at the cost of substantial redundant interaction. SynthAgent (Wang et al., 2025b) further introduces task refinement during execution and trajectory refinement after execution, partially mitigating infeasible initial tasks. However, because these refinements are decoupled, the task is often rewritten to fit the collected trajectory, producing step-heavy descriptions that read like procedural instructions rather than natural user

Web agents often struggle to generalize to unseen websites because they lack websitespecific supervision. Recent exploration-based data synthesis methods reduce manual annotation, but they still face two key limitations: they often fail to cover the full functionality of a website, and without sufficient website prior knowledge, they tend to propose hallucinated tasks, which in turn limits the diversity and efficiency of downstream trajectory synthesis. We present SynWeaver, a website-prior tasktrajectory co-synthesis framework designed to address these challenges. SynWeaver first performs structured website exploration and constructs a website map that covers a broad set of functionally distinct page states and executable interactions on the target website. It then derives page-level and transition-level supervision from this map to train a UI-aware model with website-specific priors, enabling more grounded task proposals. Finally, SynWeaver performs collaborative task-trajectory synthesis, jointly updating the task and execution trajectory when they become inconsistent, and then verifies and repairs the collected results to produce executable, semantically aligned supervision. Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms strong synthesis baselines and yields more effective supervision for both in-domain and out-of-domain generalization. The code is publicly available at https://github.com/Eilok/SynWeaver.

1

Introduction

End-to-end web agents have shown strong potential on realistic tasks, but training effective agents still requires substantial task-trajectory supervision (Qin et al., 2025; Wang et al., 2025a). Obtaining such data at scale is challenging because manual trajectory annotation is costly and time-consuming (Deng et al., 2023). Moreover, even with interaction data, agents often transfer poorly to un1

requests. To address these limitations, we propose SynWeaver, a website-prior task and trajectory co-synthesis framework. SynWeaver places explicit website knowledge at the center of synthesis and proceeds in three stages. First, it constructs a website map of functionally distinct states and executable transitions, improving coverage while reducing redundant exploration. Second, it derives user interface (UI) supervision from the map to fine-tune a UI-aware model, enabling task proposal grounded in website-specific priors rather than local observations alone. Third, it performs collaborative task-trajectory synthesis, where teacher model jointly refines the task and execution prefix whenever intent and execution become inconsistent. This collaborative refinement preserves the natural form of user requests while maintaining execution feasibility. Experiments on WebArena and WebVoyager confirm the effectiveness of SynWeaver. Using only 822 validated task-trajectory pairs synthesized from website maps, SynWeaver achieves the best overall WebArena success rate on both Qwen3-VL8B-Instruct and InternVL3-8B, and reaches 27.06 success rate on WebVoyager, outperforming the strongest baseline by 4.64 points. Our contributions are threefold: 1. We introduce a structured website exploration mechanism that reduces redundant interactions while preserving coverage of functionally distinct states, compared to random-walk exploration. 2. We propose SynWeaver, a website-priorcentered framework for collaborative tasktrajectory synthesis that couples grounded task proposal with joint refinement of task intent and execution traces. 3. Extensive experiments and ablations show that SynWeaver produces more effective and data-efficient supervision for web agents, yielding strong in-domain and out-of-domain gains over prior synthesis methods.

2

et al., 2023; Su et al., 2025). Recent web-centric methods instead interact with target websites: OSGenesis samples random-walk actions (Sun et al., 2025a), SynthAgent reduces redundancy via function categorization (Wang et al., 2025b), and GoBrowse builds an exploration graph to discover pages and induce tasks (Gandhi and Neubig, 2025). However, these approaches still provide limited guarantees of functional coverage. Graph-based systematic exploration has addressed similar coverage issues in apps, where KG-RAG and GUIXplore use DroidBot for exhaustive exploration (Guan et al., 2025; Sun et al., 2025b; Li et al., 2017); earlier web research also uses Crawljax to traverse website states with depth-first search (DFS) (Mesbah et al., 2012; Peng et al., 2012). Inspired by graph-structured web exploration (Chen et al., 2025; Zhang et al., 2026b), we construct a website map covering functionally distinct interactions and use it as an LLM fine-tuning prior, enabling task synthesis with richer website context. Trajectory Data Synthesis for Web Agents. Web agent training requires trajectory-level supervision (Nguyen et al., 2025), and synthetic trajectories often rely on stronger LLMs for labeling (Tang et al., 2025). Existing methods filter tutorials with FastText before LLM annotation (Xu et al., 2025a; Bojanowski et al., 2017), roll out trajectories with learned web-world models (Gao et al., 2025), or create verifiable synthetic websites to collect interaction traces (Wu et al., 2026). For LLM demonstrations on real websites, early approaches retain only trajectories passing binary filters (Lin et al., 2026; Lù and Reddy, 2026), while OS-Genesis scores trajectories for weighted training instead of discarding low-quality data (Sun et al., 2025a). Explorer refines tasks during execution (Pahuja et al., 2025), and SynthAgent adds post-execution trajectory refinement (Wang et al., 2025b). Yet decoupled task and trajectory refinement cannot reliably repair execution-time failures, reducing data efficiency and task-trajectory alignment. We therefore introduce collaborative refinement, which jointly optimizes tasks and trajectories throughout synthesis to preserve their semantic consistency.

Related Work

Executable Task Synthesis. Existing synthesis pipelines usually acquire environment knowledge before generating grounded tasks. Some methods mine executable tasks from online tutorials (Zhang et al., 2026a; Xu et al., 2025b; Shen et al., 2024; Zhou et al., 2025); Learn-by-interact further augments such seeds with Self-Instruct (Wang

3

Method

Problem Formulation. We model a website as a partially observable web environment W. At interaction step t, the agent receives an observation ot ∈ O (e.g., textual and visual content), 2

makes a reasoning rt ∈ R and performs an action at ∈ A, yielding an execution history h = (o1 , r1 , a1 , . . . , oH , rH , aH ) of length H. We formulate web agent data synthesis as two coupled generation processes. Task synthesis generates a task instruction x from a partial context hc , i.e., x ∼ pθ (x | hc ), where hc may be a local page state or an exploration prefix ending at the current state. Trajectory synthesis then generates an execution trajectory he conditioned on x by sequentially selecting actions according to at ∼ π(a | ot , x, h<t ) until termination. Our goal is to synthesize a dataset D = {(xi , hei )}N i=1 of N executable and semantically aligned task-trajectory pairs, rather than trajectories in isolation. In other words, the task and the trajectory are treated as co-equal objects that must express the same underlying intent. As shown in Figure 1, our SynWeaver framework proceeds in three stages: website map construction, website-prior learning, and tasktrajectory collaborative synthesis. We describe each stage in detail in the following. 3.1

that have already been triggered. DFS recurses only when a new state is confirmed; once a branch is exhausted, the crawler restores the previous context and resumes the remaining unexplored interactions. Full construction details are provided in Appendix A. 3.2

Website-Prior Learning

Existing works often propose tasks from local states (Sun et al., 2025a; Wang et al., 2025b; Pahuja et al., 2025) or a short interaction prefix (Murty et al., 2024). Without sufficient website knowledge, however, the model may overfit local cues and propose tasks that are overly narrow or even infeasible. We therefore learn website-specific UI priors from the website map before task synthesis. Supervision Signals. We design five supervision formats, grouped into page-level and transitionlevel supervision. Let q, img, a11y, img som , op, and (img ′ , a11y ′ ) denote the question, current screenshot, accessibility tree, SoM-annotated screenshot (Yang et al., 2023), executed action, and post-interaction state, respectively. Page-level supervision contains two forms: page description, which takes (img, a11y) as input and predicts a detailed description of the page content, layout, and style to provide a holistic understanding of the page; and page question answering (QA), which takes (img, a11y, q) as input and predicts the answer to an information-seeking question, thereby encouraging targeted information extraction. Transition-level supervision contains three forms: element description, which takes (img som , a11y) as input and predicts how the marked element should be interacted with and what function it serves, thereby grounding elementlevel affordances; forward transition description, which takes (img som , a11y, op) as input and predicts how the page changes after executing op, thereby modeling action-conditioned state changes; and inverse transition description, which takes (img, a11y, img ′ , a11y ′ ) as input and predicts a description of the observed change together with the action most likely to have caused it, thereby modeling change summarization and action inference. The data examples are shown in Appendix G.

Website Map Construction

Existing works often explore websites via random walks (Sun et al., 2025a) or LLM-guided probing (Murty et al., 2024), but such strategies may repeatedly traverse elements with the same functionality while missing interactions with distinct effects, yielding redundant and incomplete traces that limit downstream task diversity. We therefore introduce a DFS-based crawler that constructs a website map G = (S, T ) for each target website, where each node s ∈ S denotes a functionally distinct state together with its page screenshot and accessibility tree, and each edge τ ∈ T denotes an executable interaction and records the corresponding action. Rather than enumerating superficial page variations, G captures the website’s functional topology, reducing redundant exploration and providing the foundation for subsequent website-prior learning and task-trajectory synthesis. We construct G through depth-first exploration from a seed URL. At each visited state, the crawler enumerates executable candidates, performs the corresponding interactions, and records the resulting transitions. To avoid over-exploring superficial page variations, we introduce a unified state processor with two functions: progressive state comparison, which determines whether a post-action page should be treated as a new state, and duplicatetrigger detection, which suppresses interactions

Data Construction. We construct the supervision data by querying a teacher model M1 over page states and recorded transitions in the website map. For each node, one call to M1 with (img, a11y) yields a page description and one 3

Stage 1: Website Map Construction DFS

...

...

State Processor

Backtrack

s> ck se Cli es dr Ad ge na a <M

Click <My Account>

duplicate with

Cli ck <E dit >

Website

...

Click <General Subscription>

Stage 2: Website-Prior Learning

Construct Stage 3: Task-Trajectory Collaborative Synthesis Update reasoning

Select <Product Name>

Website Map

1 2

Trajectory Transition-level

1

le d

Refine task ?

Reorganize Page-level

Refine trajectory ?

6 Refine

.. .

Difference Analysis

k

.. .

Fine-tune

4

Verify

il

Question & Answer

3

Re fi ne Generate

Fa

Generate

St al

As

Environment Generate

Element Description

2

Observation Action

Extract

Page Description

Swap step

Delete step

1 Execution

5 Post-verification

Pass

2

Tasks

3

Verifier

Data

Figure 1: Overview of SynWeaver. The framework first constructs a website map, then learns website-specific UI priors from the collected states and transitions, and finally performs task-trajectory collaborative synthesis to produce executable and semantically aligned task-trajectory pairs.

QA pair, which are then reorganized into pagedescription data and page-QA instances. For each recorded transition, one teacher call with (img som , a11y, op, img ′ , a11y ′ ) yields an element function description and a transition change analysis, which are then recombined into element description, forward transition, and inverse transition supervision.

therefore synthesize tasks and trajectories collaboratively so that intent alignment and executability are maintained throughout data collection. Reverse Task Synthesis. We follow the reverse task synthesis strategy of OS-Genesis (Sun et al., 2025a) by extracting transition triplets (s, op, s′ ) from the website map and prompting Mui to infer a high-level task x0 for which the observed transition is a plausible intermediate consequence. Unlike OS-Genesis, however, Mui already encodes website-specific UI priors, allowing it to propose tasks that are better grounded in verified website behavior and less prone to hallucinated affordances. The synthesized task x0 then serves as the initial task specification.

Training. We aggregate all supervision sources into a unified dataset Dui and shuffle it. We then fine-tune the target model M with LoRA (Hu et al., 2022) on Dui to obtain a UI-aware model Mui that captures website-specific UI knowledge. 3.3

Task-Trajectory Collaborative Synthesis

Collaborative Trajectory Synthesis. Let xt denote the task specification at step t, and let ht = (o1 , r1 , a1 , . . . , ot , rt , at ) denote the executed prefix up to that point. Starting from x0 , a teacher model M1 interacts with the website online to collect a trajectory. We trigger collaborative refinement whenever the current task lacks essential details, becomes incompatible with the observed website state, or execution stalls after multiple unsuccessful attempts. In such cases, we in-

Prior work such as SynthAgent (Wang et al., 2025b) largely treats task refinement and trajectory refinement as separate optimization problems. This decoupling can be problematic: if one only optimizes the task, the task description may gradually align to the collected trace, yielding procedure-heavy phrasing instead of a natural user request; if one only optimizes the trajectory against a fixed task, partially useful traces may be discarded once they no longer fit the specification, reducing data efficiency. We 4

3 Scroll down 4 Scroll down 5 Scroll up 6 Stop

Find a one-star review of Q Mixers Premium Ginger Ale and summarize its key points.

1 Click

Joint refinement: (1) Refine task: Find the first five-star review of Q Mixers Premium Ginger Ale and summarize its key points. (2) Update step 2 reasoning. (3) Delete steps 3-6.

3 Done

Step 2: Click ‘12 Reviews’ to check all reviews and find the first five-star.

2 Click

3 4 5 6 The ... is in the 2nd position from the left on the 1st row. Open it first.

Click ‘12 Reviews’ to view all reviews and find the first one-star review.

Perform several scrolls and check the reviews twice, but no one-star reviews were found.

The first review is five-star rated. Its key comments are: 1..., 2..., 3...

Figure 2: Case study of collaborative trajectory synthesis. When the target review is non-existent, SynWeaver adopts joint mode to refine the task and trajectory, ensuring the task can be successfully completed.

voke a stronger teacher M2 to co-optimize the task and the collected prefix. We consider two refinement modes. In the task-only mode, M2 applies Update(xt ) while keeping ht fixed, producing a revised task that better matches the current observations and can be completed within the next 23 steps. In the joint mode, used when task-only refinement would preserve an excessively noisy prefix, M2 edits both xt and ht . Following SynthAgent (Wang et al., 2025b), we allow two trajectory edits over step indices i and j: Delete(i) removes an obsolete or redundant step, and Reorder(i, j) swaps two locally commutable steps. In addition, we introduce Update(rt ) to revise affected reasoning so that the refined execution remains logically coherent. Execution then resumes under the refined pair until termination, yielding a candidate trajectory he . A case is shown in Figure 2.

to heuristic verification, and the reconstructionverification loop repeats until the pair passes. If M2 cannot reconstruct the data, the pair is filtered out. Validated pairs are retained in D as semantically aligned task-trajectory supervision.

Post-Verification. Although co-optimization improves alignment during collection, some candidate trajectories may still contain failed attempts, redundant operations, or improper termination. We therefore perform post-verification after execution with access to the final task and the full trajectory. The first stage is heuristic verification, which applies three programmatic checks: (1) termination effectiveness, requiring an explicit completion action; (2) trajectory validity, checking successful completion within the predefined step budget; and (3) trajectory consistency, detecting invalid repetitive action patterns. Additional implementation details are deferred to Appendix B. Pairs that pass are accepted directly. Otherwise, they enter a reconstruction stage, where M2 performs targeted repair under the same joint mode as above by applying Update(xt ), Delete(i), Reorder(i, j), and Update(rt ) as needed; we additionally allow Add(none) when an explicit completion action is missing. The repaired trajectory is then returned

We evaluate SynWeaver on standard web agent benchmarks under matched fine-tuning settings.

Dataset Expansion. After generating an initial set of tasks from all extracted transitions, we resample transitions and prepend previously synthesized tasks associated with the same transition to the prompt, then ask Mui to propose an additional task that is distinct from the existing ones. Repeating the above synthesis and verification process further expands the final dataset D. Further details are shown in Appendix C.

4

Experiments

4.1

Experimental Setup

Benchmarks. We evaluate on WebArena (Zhou et al., 2024), a self-hostable benchmark spanning five websites, and WebVoyager (He et al., 2024), an online benchmark covering diverse real-world websites. Following prior work (Wang et al., 2025b), we sample one task per WebArena template and remove cross-website tasks, yielding 226 evaluation tasks. For WebVoyager, we apply executability and overlap filtering, leaving 388 evaluation tasks. We report task success rate (SR) under the official protocols. SynWeaver synthesizes data on the five WebArena websites and is evaluated in-domain on WebArena and out-of-domain on WebVoyager. Baselines & Models. We compare against NNetNav (Murty et al., 2024), OS-Genesis (Sun et al., 2025a), and SynthAgent (Wang et al., 2025b), which represent self-exploration, random-walk synthesis, and categorized exploration with online re5

Statistics States Transitions

of the synthesized task-trajectory pairs rather than from differences in model initialization. Performance varies across websites and model backbones. For Qwen3-VL-8B-Instruct, our method achieves the best SR on Shopping, Reddit, and GitLab, while SynthAgent remains strongest on CMS and Maps. For InternVL3-8B, our method performs best on Shopping, CMS, and GitLab and ties with SynthAgent for the best result on Reddit. A notable exception is Maps, where our method underperforms SynthAgent across both backbones. A likely reason is that Maps contributes only 48 transition instances for data synthesis, which is substantially fewer than the other websites and therefore provides a much smaller amount of supervision.

Shopping CMS Reddit GitLab Maps 90 148

169 213

131 203

153 214

16 16

Table 1: Website map statistics retained for SynWeaver data synthesis on WebArena.

finement, respectively. We fine-tune two opensource multimodal backbones: Qwen3-VL-8BInstruct (Bai et al., 2025) and InternVL3-8B (Zhu et al., 2025). We use Gemini-3-Flash as the base teacher M1 and Gemini-3.1-Pro as the stronger teacher M2 . Data Synthesis Statistics. Rather than relying on random exploration, SynWeaver first constructs a structured website map for each WebArena website. Table 1 reports the retained states and transitions used for synthesis, totaling 559 states and 794 transitions. From this topology, we synthesize 3,500 UI samples and 822 validated task-trajectory pairs comprising 4,500 action steps. For fair comparison, each baseline is trained on 1,000 public trajectories, so SynWeaver uses a smaller but more structured and validated training set.

Out-of-domain Results on WebVoyager. Table 3 shows that SynWeaver also generalizes well to unseen websites. Using Qwen3-VL-8B-Instruct, our method achieves an overall SR of 27.06 on the evaluated WebVoyager websites, outperforming SynthAgent by 4.64 points and the vanilla model by 13.66 points. It attains the best SR on seven of the nine websites, namely Amazon, Apple, ArXiv, BBC, ESPN, Hugging Face, and WA. However, it does not outperform the strongest baseline on Allrecipes and GitHub, indicating that cross-domain transfer remains uneven on these websites. Overall, the results on both benchmarks support our central hypothesis that high-quality web agent data should couple task intent with executable trajectories. Compared with baselines that rely on self-exploration, random walks, or decoupled tasktrajectory refinement, SynWeaver consistently delivers stronger overall performance. Notably, this improvement is achieved with 822 validated tasktrajectory pairs after filtering, which is fewer than the 1,000 trajectories used for each baseline. These findings suggest that our gains come not only from better alignment, but also from better data efficiency.

Training Details. All methods are trained under the same two-stage LoRA pipeline for fair comparison. We first adapt each backbone with SynWeaver’s UI data to obtain a shared UI-aware initialization Mui , and then fine-tune Mui using the synthesized supervision of each method. Detailed benchmark filtering, environment setup, data sampling rules, and hyperparameters are provided in Appendix D. 4.2

Main Results

In-domain Results on WebArena. As shown in Table 2, SynWeaver achieves the best overall SR on both model backbones, despite using only 822 validated task-trajectory pairs after filtering, versus the 1,000 trajectories used for each baseline. On Qwen3-VL-8B-Instruct, our method reaches 19.91, outperforming the strongest baseline, SynthAgent, by 3.10 points and the vanilla model by 7.96 points. On InternVL3-8B, it attains 14.16, improving over the strongest baseline, SynthAgent, by 1.33 points and over the vanilla model by 5.31 points. These gains are noteworthy because, under our matched training setup, all methods are fine-tuned from the same UI-aware initialization Mui . This indicates that the advantage mainly comes from the quality

5

Analysis

5.1

Ablation Study

We conduct an ablation study on Qwen3-VL-8BInstruct over the WebArena subset, as shown in Table 4. The upper block ablates components of the full SynWeaver pipeline, while the lower block compares the Qwen baseline with training on UI data only (+UI) or trajectory data only (+Traj). Relative to Ours, replacing website map sampling with random walk (−Map) reduces the overall SR by 6

Model

Method

Shopping

CMS

Reddit

GitLab

Maps

Overall

Qwen3-VL-8B-Instruct

Vanilla NNetNav OS-Genesis SynthAgent Ours

18.18 20.00 20.00 18.18 23.64

12.28 14.04 12.28 15.79 14.04

7.69 11.54 11.54 15.38 23.08

3.57 5.36 10.71 5.36 12.50

18.75 21.88 28.13 37.50 34.38

11.95 14.16 15.93 16.81 19.91

InternVL3-8B

Vanilla NNetNav OS-Genesis SynthAgent Ours

9.09 16.36 14.55 16.36 20.00

10.53 10.53 10.53 7.02 12.28

7.69 0.00 11.54 15.38 15.38

5.36 3.57 5.36 5.36 8.93

12.50 21.88 15.63 28.13 15.63

8.85 10.62 11.06 12.83 14.16

Table 2: Task success rate (SR, %) on WebArena. Bold numbers indicate the best result. Method

Allrecipes Amazon Apple ArXiv

Qwen +NNetNav +OS-Genesis +SynthAgent +Ours

4.44 2.22 6.67 4.44 2.22

4.88 9.76 21.95 24.39 29.27

20.93 13.95 27.91 25.58 39.53

11.63 23.26 18.60 23.26 27.91

BBC

ESPN GitHub Hugging Face

19.05 4.55 28.57 9.09 21.43 11.36 35.71 18.18 40.48 20.45

31.71 34.15 29.27 36.59 29.27

13.95 16.28 23.26 20.93 30.23

WA

Overall

10.87 21.74 15.22 15.22 26.09

13.40 17.53 19.33 22.42 27.06

Table 3: Qwen3-VL-8B-Instruct task success rate (SR, %) on WebVoyager. The abbreviations in the header are BBC = BBC News and WA = Wolfram Alpha. Bold numbers denote the best result in each column.

pipeline performs best.

Method Shopping CMS Reddit GitLab Maps Overall Ours −Map −CR −WP −PV

23.64 20.00 18.18 27.27 21.82

14.04 8.77 10.53 12.28 7.02

23.08 15.38 15.38 19.23 19.23

12.50 8.93 5.36 8.93 7.14

34.38 31.25 34.38 25.00 25.00

19.91 15.49 15.04 17.70 14.60

Qwen +UI +Traj

18.18 16.36 25.45

12.28 10.53 14.04

7.69 11.54 19.23

3.57 5.36 8.93

18.75 34.38 31.25

11.95 14.16 18.58

5.2 Impact of Website Maps on Data Diversity To verify whether tasks synthesized from website maps are more diverse, we compare the synthesized task sets in Table 5. We use NovelSum (Yang et al., 2025), which summarizes set-level novelty using both semantic difference and local density; here, K, α, and β control the neighborhood size, distance weighting, and density effect, respectively. For evaluation, we use the 1,009 human-annotated training tasks from Mind2Web (Deng et al., 2023) as the reference set, encode task texts with BGE-M3 (Chen et al., 2024), and report three settings to ensure robust results: the standard setting (K=10, β=0.5, α=1), the densityfree variant (K=10, β=0, α=1), and the variant with different neighbors (K=5, β=0.5, α=1). We also report cosine distance as a complementary measure. SynWeaver achieves the best NovelSum in all three settings (0.3805, 0.3731, and 0.3854), while remaining essentially tied with the strongest baseline on cosine distance. This shows that website maps improve task diversity without sacrificing overall semantic spread.

Table 4: Ablation results (SR, %) on WebArena using Qwen3-VL-8B-Instruct. Ours denotes the full SynWeaver pipeline. −Map replaces website map with random walk; −CR replaces collaborative refinement with decoupled refinement; −WP replaces the websiteprior proposer with a general-purpose model; and −PV removes post-verification. UI and Traj use only UI data and trajectory data, respectively, for training.

4.42 points, demonstrating the value of structured website exploration. Replacing collaborative refinement with decoupled refinement (−CR) causes a 4.87-point drop, with the largest losses on Reddit (7.70) and GitLab (7.14), confirming the benefit of jointly refining tasks and trajectories. Replacing the website-prior proposer with Gemini-3.1Pro (−WP) lowers the overall SR by 2.21 points and particularly hurts Maps (9.38), while removing post-verification (−PV) produces the largest overall drop of 5.31 points. In the lower block, +UI and +Traj improve the overall SR over Qwen by 2.21 and 6.63 points, respectively. Overall, the full

5.3

Impact of Website-Prior on Task Quality

To assess the effect of website-prior on task proposal, we compare the UI-aware model with Gemini-3.1-Pro on the same transition triplets as 7

NovelSum

Method NNetNav OS-Genesis SynthAgent Ours

Method

Cosine Distance

Setting 1 Setting 2 Setting 3

SynthAgent GPT-4.1

0.3585 0.3624 0.3725 0.3805

DR CR

0.3359 0.3528 0.3681 0.3731

0.3640 0.3673 0.3770 0.3854

0.5517 0.5606 0.5825 0.5820

Gemini-3-Flash + Gemini-3.1-Pro

Task SR (%) Retention (%) Cost ($) –

–

0.130

71.89 74.79

90.59 99.52

0.091 0.085

Table 6: Comparison between decoupled (DR) and collaborative (CR) refinement on the same task set. SynthAgent is included as a cost reference.

Table 5: Data diversity comparison across different synthesis methods. Settings 1, 2, and 3 correspond to (K=10, β=0.5, α=1), (K=10, β=0, α=1), and (K=5, β=0.5, α=1), respectively. Higher values indicate greater diversity.

whereas collaborative refinement jointly updates the task specification and executed prefix when inconsistencies arise. We report task success rate, trajectory retention, and estimated synthesis cost, with SynthAgent included as a cost reference. As shown in Table 6, collaborative refinement improves task success rate from 71.89% to 74.79% and raises trajectory retention from 90.59% to 99.52%. The main gain comes from retention: far fewer trajectories are discarded after verification, so more collected data can be retained as supervision. This higher retention directly lowers synthesis cost, reducing it from $0.091 to $0.085 despite better task success and data quality. Cost is further reduced by the hierarchical teacher design, where the lightweight teacher M1 handles routine synthesis and the stronger teacher M2 is used only for difficult refinements and repairs. As a result, SynWeaver is also cheaper than SynthAgent ($0.13). Overall, collaborative refinement improves both data efficiency and economic efficiency.

Figure 3: Effect of website-prior on task proposal quality across WebArena websites.

Table 1, changing only the proposer. Tasks from both models undergo the same Collaborative Trajectory Synthesis procedure. We report final task success rate and the average number of task refinement, where higher success and fewer refinements indicate better-grounded proposals. Figure 3 shows that the UI-aware proposer achieves higher task success rates with fewer optimizations across all WebArena websites. We do not control for task difficulty, so simpler tasks may naturally score better on both metrics. Nevertheless, Table 4 shows that replacing the website-prior proposer with a general-purpose model (−WP) reduces downstream SR from 19.91 to 17.70, indicating that website-prior proposals and their trajectories provide more effective training supervision. 5.4

Teacher Model

6

Conclusion

In this paper, we present SynWeaver, a websiteprior task-trajectory co-synthesis framework for web agents. SynWeaver addresses a key limitation of existing exploration-based synthesis methods: they often lack sufficient website-specific priors to propose realistic and executable tasks, leading to limited trajectory quality. To overcome this issue, SynWeaver combines exploration-driven website map construction with website-prior learning and collaborative task-trajectory refinement, producing supervision that is more faithfully aligned with real website interactions. Experiments on WebArena and WebVoyager demonstrate that SynWeaver consistently outperforms prior synthesis baselines while also improving data diversity and synthesis efficiency. These results highlight the importance of explicit website knowledge in scalable web agent data synthesis and provide a practical direction for improving web agent adaptation and generalization.

Impact of Collaborative Refinement on Data Efficiency and Cost

To examine how collaborative refinement affects data efficiency and synthesis cost, we compare it with decoupled refinement on the same task set. Decoupled refinement updates tasks during trajectory collection and revises trajectories after execution, 8

Limitations

2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.

SynWeaver has limitations in both deployment scope and training paradigm. Its website maps are constructed through automatic exploration; although this process is effective in self-hostable and moderately constrained environments, it becomes less reliable when websites adopt strong anti-bot or human-verification mechanisms, such as CAPTCHA systems, aggressive rate limiting, or dynamic authentication pipelines. In addition, SynWeaver mainly improves web agents through supervised fine-tuning on teacher-generated data, making the resulting student model inherently bounded by the reasoning and interaction capabilities of the proprietary teacher models used during synthesis. The current framework also does not explore more autonomous self-improving paradigms, such as online reinforcement learning, which may further enhance agents beyond the supervision provided by the teachers. We leave these directions to future work.

Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the association for computational linguistics, 5:135–146. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. M3embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through selfknowledge distillation. In Findings of the association for computational linguistics: ACL 2024, pages 2318–2335. Weizhi Chen, Ziwei Wang, Leyang Yang, Sheng Zhou, Xiaoxuan Tang, Jiajun Bu, Yong Li, and Wei Jiang. 2025. Pg-agent: An agent powered by page graph. In Proceedings of the 33rd ACM International Conference on Multimedia, pages 6878–6887. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091–28114. Apurva Gandhi and Graham Neubig. 2025. Go-browse: Training web agents with structured exploration. arXiv preprint arXiv:2506.03533.

Ethical Considerations All experiments in this work are conducted in controlled, compliant, and authorized web environments, such as benchmarks and self-hosted websites. We do not crawl third-party websites without permission, bypass access controls, collect private user data, or disrupt online services. Because automated website exploration could be misused for unauthorized crawling, large-scale scraping, or other harmful activities, we strongly oppose such uses and call on researchers and practitioners to obtain proper authorization, respect website policies and rate limits, and use these methods only for lawful and responsible purposes.

Yifei Gao, Junhong Ye, Jiaqi Wang, and Jitao Sang. 2025. Websynthesis: World-model-guided mcts for efficient webui-trajectory synthesis. arXiv preprint arXiv:2507.04370. Ziyi Guan, Jason Chun Lok Li, Zhijian Hou, Pingping Zhang, Donglai Xu, Yuzhi Zhao, Mengyang Wu, Jinpeng Chen, Thanh-Toan Nguyen, Pengfei Xian, and 1 others. 2025. Kg-rag: Enhancing gui agent decision-making via knowledge graph-driven retrieval-augmented generation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5396–5405. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-toend web agent with large multimodal models. arXiv preprint arXiv:2401.13919.

Use of AI Assistants AI assistants were used only for grammar checking and language polishing during manuscript preparation. All scientific ideas, methods, experiments, results, analyses, and conclusions were developed and verified by the authors. The authors manually reviewed and edited all AI-assisted text and take full responsibility for the final manuscript.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3.

References

Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2017. Droidbot: a lightweight ui-guided test input generator for android. In 2017 IEEE/ACM 39th international conference on software engineering companion (ICSE-C), pages 23–26. IEEE.

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, and 1 others.

Zichuan Lin, Feiyu Liu, Yijun Yang, Jiafei Lyu, Yiming Gao, Yicheng Liu, Zhicong Lu, Yangbin Yu, Mingyu Yang, Junyou Li, and 1 others. 2026. Ui-voyager: A

9

self-evolving gui agent learning via failed experience. arXiv preprint arXiv:2603.24533.

on Learning Representations, volume 2025, pages 89044–89097.

Xing Han Lù and Siva Reddy. 2026. Structured distillation of web agent capabilities enables generalization. arXiv preprint arXiv:2604.07776.

Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, and 1 others. 2025a. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5555–5579.

Jacopo Mereu, Alessandro Carcangiu, Valentino Artizzu, Federico Maria Cau, and Lucio Davide Spano. 2026. Specializing llms to low-documented domains with rag: An analysis across models and retrieval depths. SN Computer Science, 7(4):320.

Yuchen Sun, Shanhui Zhao, Tao Yu, Hao Wen, Samith Va, Mengwei Xu, Yuanchun Li, and Chongyang Zhang. 2025b. Gui-xplore: Empowering generalizable gui agents with one exploration. In Proceedings of the computer vision and pattern recognition conference, pages 19477–19486.

Ali Mesbah, Arie Van Deursen, and Stefan Lenselink. 2012. Crawling ajax-based web applications through dynamic analysis of user interface state changes. ACM Transactions on the Web (TWEB), 6(1):1–30. Shikhar Murty, Hao Zhu, Dzmitry Bahdanau, and Christopher D Manning. 2024. Nnetnav: Unsupervised learning of browser agents through environment interaction in the wild. arXiv preprint arXiv:2410.02907.

Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865.

Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, and 1 others. 2025. Gui agents: A survey. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22522–22538.

Haoming Wang, Haoyang Zou, Huatong Song, Jiazhan Feng, Junjie Fang, Junting Lu, Longxiang Liu, Qinyu Luo, Shihao Liang, Shijue Huang, and 1 others. 2025a. Ui-tars-2 technical report: Advancing gui agent with multi-turn reinforcement learning. arXiv preprint arXiv:2509.02544.

Vardaan Pahuja, Yadong Lu, Corby Rosset, Boyu Gou, Arindam Mitra, Spencer Whitehead, Yu Su, and Ahmed Hassan. 2025. Explorer: Scaling explorationdriven web trajectory synthesis for multimodal web agents. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6300–6323.

Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508.

Zhaomeng Peng, Nengqiang He, Chunxiao Jiang, Zhihua Li, Lei Xu, Yipeng Li, and Yong Ren. 2012. Graph-based ajax crawl: Mining data from rich internet applications. In 2012 International Conference on Computer Science and Electronics Engineering, volume 3, pages 590–594. IEEE.

Zhaoyang Wang, Yiming Liang, Xuchao Zhang, Qianhui Wu, Siwei Han, Anson Bastos, Rujia Wang, Chetan Bansal, Baolin Peng, Jianfeng Gao, and 1 others. 2025b. Adapting web agents with synthetic supervision. arXiv preprint arXiv:2511.06101.

Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, and 1 others. 2025. Uitars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12326.

Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zijie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, and 1 others. 2026. Autowebworld: Synthesizing infinite verifiable web environments via finite state machines. arXiv preprint arXiv:2602.14296.

Junhong Shen, Atishay Jain, Zedian Xiao, Ishan Amlekar, Mouad Hadji, Aaron Podolny, and Ameet Talwalkar. 2024. Scribeagent: Towards specialized web agents using production-scale workflow data. arXiv preprint arXiv:2411.15004.

Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C Ho, Carl Yang, and Dong Yu. 2025a. Retrieval-augmented gui agents with generative guidelines. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17877–17886.

Chenrui Shi, Zedong Yu, Zhi Gao, Ruining Feng, Enqi Liu, Yuwei Wu, Yunde Jia, Liuyu Xiang, Zhaofeng He, and Qing Li. 2025. Gui knowledge bench: Revealing the knowledge gap behind vlm failures in gui tasks. arXiv preprint arXiv:2510.26098.

Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. 2025b. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In International Conference on Learning Representations, volume 2025, pages 79822–79843.

Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Arik. 2025. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. In International Conference

10

A

Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441.

Website Crawling Details

This appendix describes the construction of the website map G. Starting from a seed URL, the crawler incrementally explores reachable pages with depth-first search (DFS) and records both states and transitions in a persistent graph. Algorithm 1 is the main controller: it acquires the current state, inserts the state and transition into G, filters candidate actions, and recursively explores newly discovered states. Algorithm 2 is called inside this DFS loop to prune redundant lists, forms, and atomic elements before execution. Algorithm 3 is applied after each interaction to determine whether the resulting page should be treated as a new state. Algorithm 4 is executed after recursive exploration returns, so that the crawler can recover the previous action context and continue exploring the remaining actions from the correct state. Table 8 summarizes the notation shared by these algorithms, and Table 9 defines the principal graph and browser operations used in their pseudocode.

Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2025. Measuring data diversity for instruction tuning: A systematic analysis and a reliable metric. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 18530–18549. Bofei Zhang, Zirui Shang, Zhi Gao, Wang Zhang, Rui Xie, Xiaojian Ma, Tao Yuan, Xinxiao Wu, SongChun Zhu, and Qing Li. 2026a. Tongui: Internetscale trajectories from multimodal web tutorials for generalized gui agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 12367–12375. Xuanwang Zhang, Yuteng Han, Jinnan Qi, Mulong Xie, Zhen Wu, and Xinyu Dai. 2026b. Webnavigator: Global web navigation via interaction graph retrieval. arXiv preprint arXiv:2603.20366. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606.

Why we adopt DFS? We adopt DFS as the outer exploration strategy because it is substantially more efficient than breadth-first search (BFS) in an interactive browser environment. Under BFS, exploration proceeds level by level. After expanding one state, the crawler must repeatedly return to earlier states and then restore the browser to another frontier state in order to continue expansion. In practice, this requires frequent rollback and replay operations, such as navigating back, reopening pages, or deterministically replaying prior actions, which introduces considerable browser overhead. By contrast, DFS continues along the current branch and performs recovery mainly when the branch has been exhausted, thereby greatly reducing the number of restoration operations and the associated resource cost.

Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Li Erran Li. 2025. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. In Forty-second International Conference on Machine Learning. Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, and 1 others. 2025. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479.

State. A page state sd = (u, E, L, F, a11y, img) is a structured snapshot of the current web interface. Specifically, u is the current URL, E is the set of actionable elements detected on the page, L is the set of extracted list containers, F is the set of extracted form containers, and (a11y, img) denote the accessibility tree and screenshot, respectively. Here, a list container is detected from DOM containers such as ul, ol, table/tbody, or list-like ARIA containers (e.g., role="list", role="listbox", and role="menu"). A container is treated as a valid list only if it contains over three actionable list items 11

overall. A form container is detected only from a genuine ancestor <form> element. Elements are grouped by their nearest form ancestor, and a container is retained as a valid form only if it contains at least one input field and one submit button.

vide the same functionality even when their item texts or item counts differ. Therefore, Xl additionally records visited list-container XPaths so that functionally equivalent lists are not re-explored solely because their contents change. For forms, repeated functionality is less likely to reappear across different page structures, so the crawler adopts a lighter strategy and records only XPaths of form containers in Xf , which saves memory while remaining sufficient in practice. In Algorithm 2, signatures are computed by GetSign as lightweight identifiers rather than full semantic hashes. For an atomic element, the signature is formed by concatenating its tag, name, aria-label, and text, i.e., sig(e) = tag | name | aria_label | text. For a list container, the signature is computed by concatenating the container tag with the texts of its list items, equivalently the container tag plus the string obtained by prefixing each item text with a separator and concatenating them in order. This design keeps signature computation simple and efficient, while the combination of signature and XPath prevents redundant exploration more robustly than any single mechanism alone.

Transition. A transition is defined as τd = (sd , sd+1 , newP ageOpened, opd ), where sd is the pre-action state, sd+1 is the resulting state after execution, newP ageOpened indicates whether the action creates a new page, and opd records the executed operation. This representation makes each edge in the website graph self-contained: it captures not only how the crawler moves from one state to another, but also the information needed for later recovery and replay. In practice, transitions are produced by three kinds of execution routines: form execution, atomic element execution, and page-level scrolling. Among them, atomic execution is rule-based: the crawler chooses an action according to the target element’s tag, type, text, and accessibility attributes. Table 7 summarizes the action categories used in our implementation, together with their high-level semantics and the tag patterns to which they apply. If the element does not match any action, click is executed by default.

Constrained Exploration. To ensure finite and effective website exploration, the crawler relies not only on the duplicate-trigger detection mechanism described above, but also on three additional constraints applied during each recursive step: ShouldStop, IsExitGuarded, and SampleItems. First, ShouldStop determines whether the current branch should terminate immediately. In practice, this happens when the crawler fails to obtain the current state, when the graph G detects that the state is already known, when the current URL leaves the target root domain and enters an external domain, when the current URL matches a manually configured blocked URL, or when the predefined exploration budget, such as the maximum depth or the maximum number of states, has been reached. Second, IsExitGuarded checks whether a candidate element belongs to a predefined set of forbidden interactions before the crawler executes it. This mechanism is mainly used to block elements that may prematurely terminate or disrupt the session, such as logout, sign out, or account deletion controls, and thus prevents the crawler from leaving the intended exploration workflow through unsafe actions. Third, SampleItems constrains exploration within list containers by selecting only representative elements rather than exhaustively triggering

Duplicate-trigger Detection. Duplicate-trigger detection is required to ensure that the website graph covers each distinct functional state while avoiding repeated exploration of equivalent functionality. To support this goal, the crawler maintains an exploration registry M = (Ve , Vl , Xl , Xf , Du ) across recursive calls. The set Ve records element signatures so that once an atomic element with the same functional profile has been triggered, the crawler does not need to trigger it again on later pages. However, element signatures alone are not sufficient, because two elements may serve the same function while carrying different local text. A typical example is a content feed such as Reddit. There are many posts sharing the same structure and affording the same interactions, but their texts differ, so they may still be treated as different elements by signature matching. To address this case, Du records visited element XPaths under the same URL template, allowing the crawler to deduplicate elements that appear at the same structural position on templateequivalent pages. The same design principle is used for lists. The set Vl stores list signatures to suppress obviously identical list containers, but lists on structurally similar pages may still pro12

Action click

Description Clicks the element and then checks whether the interaction opens a new tab.

fill

select hover

scroll form_submission

Applicable Tags / Targets Direct click targets: <button>, <a>, <summary>, <details>, <img>, <span>, <video>, <audio>, <iframe>, <label>; <input> with clickable types (checkbox, radio, button, submit, reset, image) Fills an input control with <textarea>; <input> with input types (text, a default value based on email, password, tel, url, number, and date), exthe field type and textual cluding search-like fields attributes. Selects an option from a <select> dropdown menu. Hovers over the target el- <li> with submenu cues in text or aria-label; fallement to reveal hidden back elements whose aria-label indicates menu or menus. dropdown behavior Scrolls the page by one N/A viewport. Executes a composite Extracted <form> containers, typically with <input>, form action: fills fields <textarea>, or <select> fields and a submit control and then submits form. such as <button> or <input type="submit">

Table 7: Browser actions for website map construction and their applicable tag patterns.

every repeated item. The key observation is that many lists contain multiple items with the same functional structure, so exploring all of them would add cost without revealing new behavior. In our framework, sampling is primarily guided by an LLM: if the list items are judged to provide different functions, all of them are retained; if they are judged to contain repeated functional groups, only one representative group is kept. For example, in a product-card list, each card may contain the same interaction pattern, such as product image, product name, add-to-cart, and add-to-wishlist, while differing only in product content, in which case exploring a single representative card is sufficient.

while still preserving a fallback for dynamic or non-reversible interactions.

B

Heuristic Verification Details

Heuristic verification is the first stage of postverification and is applied to a completed candidate trajectory he = (o1 , r1 , a1 , . . . , oH , rH , aH ). The procedure is purely offline: it does not re-execute the browser or query the environment again, but instead diagnoses trajectory quality from the finished interaction record. For non-empty trajectories, it applies three deterministic checks in order: (1) termination effectiveness, (2) trajectory validity, and (3) trajectory consistency. A candidate passes heuristic verification if and only if no issue is detected.

Backtrack. After exploring the subtree induced by one action, the crawler must restore the preaction context before it can continue with the remaining unexplored actions. Algorithm 4 therefore uses three recovery strategies in order. The first strategy closes a newly opened tab, which directly handles actions that spawn a separate browsing page. The second strategy uses browser history navigation when the interaction changes the current page within the same tab. The third strategy replays the shortest path from the root state, which is more expensive but more robust when local rollback is unavailable or unreliable. This prioritized design keeps common recovery cases lightweight

Termination Effectiveness. A valid trajectory is required to end with an explicit completion action, denoted by none. This convention provides a canonical marker that the task objective has been achieved and aligns the trajectory with a final completion statement rather than a mere cessation of interaction. Therefore, if the last step is not none, the trajectory is judged to be missing an explicit ending signal and fails termination effectiveness. Trajectory Validity. Beyond the explicit ending marker, heuristic verification checks whether the 13

Symbol Γ B G M d sd τd

ωl ∈ L ωf ∈ Ωf ωe ∈ Ωe u E L F a11y img Ve Vl Xl Xf Du

Description Exploration configuration Browser interaction agent that executes browser operations Website graph Exploration registry, M = (Ve , Vl , Xl , Xf , Du ) Current exploration depth State at depth d, where sd = (u, E, L, F, a11y, img) Transition generated at depth d, where τd = (sd , sd+1 , newP ageOpened, opd ) List container in the current state Form in the current state Atomic element in the current state URL Set of all actionable elements on the webpage Set of all list containers on the webpage Set of all form containers on the webpage Accessibility tree Screenshot of the webpage Set of visited element signatures Set of visited list signatures Set of visited list XPaths Set of visited form XPaths Map from a URL template to visited element XPaths

Function Navigate GetCurrentState AddState

AddTransition ShouldStop IsExitGuarded ExecuteForm

ExecuteAtomic

GetSign GetItems

Table 8: Notation Description.

GetShortestPath

completed trajectory is a plausible successful execution artifact. First, an empty trajectory is rejected immediately, since no executable evidence of task completion is present. Second, let n = |he | and let B denote the predefined step budget. When n ≥ B, the trajectory is flagged with an error. Excessive length often reflects program vulnerabilities because we have set the step limit before execution. Third, the program will detect the state of the environment during execution. If the execution result is labelled as failure, it fails the test. This often happens when some behaviors of the agent cause environmental errors.

Description Navigates to the specified URL u. Returns the current webpage state sd . Adds the state sd to the graph G and returns whether it is a duplicate dup. Adds the transition τd to the graph G. Determines whether current exploration branch should be stopped. Checks whether the element ωe is in the block list. Replays a recorded form operation when available; otherwise, it fills and submits the form with the LLM, and falls back to default values if needed. Replays a recorded operation when available; Otherwise, it triggers the element according to its tag. Computes the signature of the object. Returns all elements contained in the given container. Computes the shortest path from the root state to the target state using bidirectional breadth-first search.

Table 9: Function Description.

formation that determines functional behavior: the action type, the identity of the target when applicable, and the action value when the semantics of that action depend on it (e.g. type:[4480]:pencil). This abstraction suppresses incidental observation differences while preserving materially distinct operations. The loop search is activated only when n ≥ 4. It then enumerates candidate pattern lengths ℓ ∈ [1, min(⌊n/2⌋, 10)] and, for each ℓ, enumerates feasible start positions j. For each candidate

Trajectory Consistency. The consistency check is designed to detect cyclic or stalled behavior. We first map he to an action-signature sequence Σ = (σ1 , . . . , σn ), where each σi preserves only the in14

pair (j, ℓ), define

is then processed by the same collaborative synthesis and post-verification pipeline described in the main text. The resampling order follows the structural accessibility of G. Let depth(s) denote the depth of the source state s from the root, and let deg(s) denote its degree in the website map. We rank triplets lexicographically by

pj,ℓ = (σj , . . . , σj+ℓ−1 ). Starting from the next block of length ℓ, the verifier checks whether subsequent blocks match pj,ℓ exactly and contiguously. If no such repetition exists, it moves to the next candidate; if repetition is observed, it continues counting until the first mismatch, at which point the current candidate stops expanding. Let rj,ℓ denote the number of contiguous repeats observed after the initial block. The repeated block therefore satisfies

depth(s), − deg(s) , so that shallower triplets are revisited first, and among triplets at the same depth, those attached to higher-degree states are preferred. This bias is useful because shallow, high-degree regions usually offer larger interaction space and more stable execution prefixes, making them more likely to yield diverse yet completable tasks.

(σj+kℓ , . . . , σj+(k+1)ℓ−1 ) = pj,ℓ , for k = 1, . . . , rj,ℓ . Only candidates with at least two contiguous repeats are retained. For each retained candidate, the repeated-steps coverage is defined as cj,ℓ =

ℓ rj,ℓ , n

Experimental Details

D.1

Environment

Benchmarks and Filtering. We evaluate on WebArena (Zhou et al., 2024), which contains 812 tasks derived from 241 templates across five websites, and WebVoyager (He et al., 2024), which contains 643 task queries across 15 websites. For WebArena, we sample one task from each template and remove cross-website tasks following prior work (Wang et al., 2025b), leaving 226 evaluation tasks. For WebVoyager, we exclude tasks from Google Flights and Booking because they are no longer executable, remove Cambridge Dictionary, Coursera, and Google Search because of strict robot verification, and exclude Google Maps to avoid overlap with the websites used for training in WebArena. This leaves 388 evaluation tasks.

and its priority score is   1 sj,ℓ = cj,ℓ 1 + , ℓ which favors both larger coverage and shorter repeated blocks. Among all retained candidates, the verifier selects the one with the highest score as the dominant loop pattern. A trajectory is then judged inconsistent when this dominant loop covers more than 40% of the trajectory, or when its repeat count is at least four even if the coverage is lower. This criterion captures both short repetitive failures, such as repeated clicking or scrolling, and longer multi-step cycles, while remaining restricted to contiguous repetition rather than scattered recurrence.

C

D

Environment Configuration. We host the online web environment on an Amazon EC2 instance (t3a.xlarge) with a 1000GB EBS root volume. Both benchmarks provide screenshots and accessibility trees as observations.

Dataset Expansion Details

After the initial synthesis round, we continue expansion by revisiting transition triplets (s, op, s′ ) extracted from the website map G. The user specifies the number of additional tasks to synthesize, after which the system samples the same number of triplets for a second proposal pass. For each sampled triplet, previously accepted tasks associated with it are prepended to the prompt so that Mui is encouraged to propose a new task that remains compatible with the observed transition while avoiding near-duplicate outputs. Each newly proposed task

Action Space. Table 10 lists all actions available in the trajectories. The stop action terminates execution when a task is infeasible or the agent is stalled. The none action denotes the final execution step and returns the task answer. D.2

Baselines

We compare against NNetNav (Murty et al., 2024), OS-Genesis (Sun et al., 2025a), and SynthAgent 15

Action

Method

Description

NNetNav (full) SynthAgent (full)

click [id]

Clicks the element tagged with the given id. type [id] [content] Types the provided content into the element tagged with the given id. hover [id] Hovers over the element tagged with the given id. press [key comb] Presses a key combination (e.g., Cmd+V). scroll [up|down] Scrolls the page up or down. new_tab Opens a new browser tab. tab_focus [tab_index] Switches focus to the browser tab with the specified index. close_tab Closes the current active tab. goto [url] Navigates to the specified URL. go_back Navigates to the previously viewed page. go_forward Navigates to the next page. stop [reason] Stops execution and records the reason. none [summary] Terminates execution with the final answer.

Shopping CMS Reddit GitLab Maps Overall 20.00 23.64

15.79 14.04

15.38 19.23

7.14 7.14

34.38 37.50

17.26 18.58

Table 11: Supplemental full-data baseline results for UIaware model (Qwen3-VL-8B-Instruct) on WebArena. Statistics States Transitions

Shopping CMS Reddit GitLab Maps 90 148

646 750

131 203

426 606

16 16

Table 12: Original website map statistics for WebArena.

and 18.58%, respectively, improving upon their balanced 1,000-trajectory results of 14.16% and 16.81%. Nevertheless, both remain below SynWeaver’s 19.91%, indicating that data quality and structural coverage remain important beyond scale alone. D.3

Table 10: Available browsing actions.

SynWeaver Data Synthesis

Website Map Construction and Filtering. For the five WebArena websites, we construct a website map using Qwen3-VL-235B-A22B-Instruct to sample representative elements within list containers and Qwen3-VL-Plus as a fallback for state equivalence judgment. Table 12 reports the original graph statistics. CMS and GitLab exhibit substantially higher graph complexity, so we apply controlled sampling to balance the final UI and trajectory data. For page-level UI synthesis on these two websites, we retain one state per URL. For transition-level synthesis, we keep one transition every two hops and cap the number of scrolling operations at 25, discarding additional scrolling transitions after the cap is reached. The filtered statistics used for data synthesis are reported in Table 1 in the main paper.

(Wang et al., 2025b), and construct matched training sets from their released data. OS-Genesis provides 1,000 trajectories, which we use in full. For NNetNav and SynthAgent, we sample 1,000 trajectories from each method. To balance the data across websites, we sample 200 trajectories per website. For NNetNav, we first merge multi-step records into single trajectory records according to data indices, and then use regular-expression matching on the first-step text of each trajectory to identify the corresponding website. For SynthAgent, we perform indexed sampling over the task list, whose keys follow the format task@website. Specifically, we select the first 200 task indices for each website and then extract the corresponding trajectories to construct the training dataset. Because NNetNav and SynthAgent release more trajectories than the 1,000 examples used in our balanced main comparison, we further conduct a supplemental full-data experiment for these two baselines. This experiment fine-tunes the UIaware model (Qwen3-VL-8B-Instruct) on the full released NNetNav or SynthAgent data using the same LoRA configuration as the main experiments, and evaluates the resulting models on the same WebArena benchmark. Table 11 reports the resulting website-level and overall success rates. Using the full released data, NNetNav and SynthAgent achieve overall success rates of 17.26%

Synthesized Data. Using the retained states, we synthesize two page-level datasets, page description and page QA, with 559 instances each. Using the retained transitions, we synthesize three transition-level datasets, element description, forward transition description, and inverse transition description, with 794 instances each. After shuffling these five subsets, we obtain a UI dataset containing 3,500 entries. We then synthesize one task-trajectory pair for each retained transition. To mitigate sample imbalance, we perform task sampling three times for the Maps website; additional details are provided in Appendix C. This process yields 826 candidate task-trajectory pairs. During post-verification, four pairs fail validation and 16

are removed, leaving 822 validated task-trajectory pairs for training.

Scale Data construction S M

D.4

Training L

We train on a cluster of eight Ascend 910C NPUs. We use Gemini-3-Flash 1 as the base teacher M1 and Gemini-3.1-Pro 2 as the stronger teacher M2 . In the first stage, M1 synthesizes page-level UI data for each retained state and transition-level UI data for each retained transition. We then LoRA finetune each backbone to obtain Mui using a learning rate of 5e-5, batch size 16, rank 8, alpha 16, and 3 epochs. In the second stage, we LoRA fine-tune Mui into the final experimental model using the synthesized data of each method, with a learning rate of 1e-4, batch size 32, rank 16, alpha 32, and 3 epochs.

E

XL

# Traj.

Rel.

Half of the standard sam413 0.50× ple Standard sampling config826 1.00× uration M with full CMS and Git- 1,755 2.12× Lab sampling L plus one additional M- 2,581 3.12× sized sampling pass

Table 13: Construction of the four trajectory scales.

For all four settings, we start from the same UI-aware Qwen3-VL-8B-Instruct initialization and keep the synthesis, collaborative refinement, postverification, fine-tuning hyperparameters, and WebArena evaluation split unchanged. Thus, the comparison varies only the amount and coverage of the task-trajectory supervision induced by the sampling configurations above. We report task success rate (SR) on each website and over all 226 evaluation tasks.

Data Scaling

40

Success rate (%)

Experimental Setup. We investigate how SynWeaver scales with the amount of synthesized tasktrajectory supervision. Because each sampled transition in the website map serves as an anchor for task-trajectory synthesis, the number of sampled or resampled transitions directly determines the trajectory budget. We construct four operational data scales, summarized in Table 13. The small setting (S) uses half of the standard sample, whereas the medium setting (M) uses the standard sampling configuration. To test whether broader graph coverage provides further gains, the large setting (L) starts from M and exhaustively samples the transitions of Shopping Admin (CMS) and GitLab. Finally, the extra-large setting (XL) augments L with one additional sampling pass whose budget equals that of M. This last setting increases the number of trajectories generated from the available website-map structure and tests whether additional task diversity remains beneficial after expanding transition coverage. Relative to M, the four settings contain 0.50×, 1.00×, 2.12×, and 3.12× as many trajectories, respectively. These points characterize practical scaling behavior under the available website maps rather than fitting a parametric power law.

Shopping CMS

Reddit GitLab

Maps Overall

30 23.89 19.91

20.80

20 14.16

10

0

S

M

L

XL

Training data scale

Figure 4: Performance of SynWeaver across different websites with varying data amounts.

Results. Figure 4 shows a consistently positive aggregate scaling trend. Increasing the training set from 413 trajectories (S) to 826 trajectories (M) raises the overall SR from 14.16 to 19.91, a gain of 5.75 percentage points. Expanding transition coverage in L further improves the overall SR to 20.80, and the additional sampling pass in XL yields the best result of 23.89. Overall, scaling from S to XL improves SR by 9.73 points, corresponding to a relative improvement of 68.7%. The website-level results show a broadly positive but heterogeneous scaling pattern. Although performance on individual websites fluctuates across the intermediate scales, most websites benefit when the data are expanded from S to XL, and the largest setting achieves the strongest or tied-strongest performance on four of the five websites. The nonmonotonic changes at intermediate scales likely

1

https://storage.googleapis.com/deepmind-media/ModelCards/Gemini-3-Flash-Model-Card.pdf 2 https://deepmind.google/models/model-cards/gemini-31-pro/

17

reflect shifts in data composition: L primarily expands transition coverage for CMS and GitLab, whereas XL further increases task diversity through an additional sampling pass. Taken together with the steadily increasing overall SR, these results indicate that SynWeaver can effectively exploit additional website-map coverage and repeated transition-conditioned sampling, while the distribution of the synthesis budget across websites remains important for domain-specific gains.

F

Model Qwen3-VL-8B-Instruct +UI Gemini-3.1-Pro

Accuracy (%) 74.5 89.0 87.5

Table 14: Accuracy on the held-out UI knowledge test.

Results. As shown in Table 14, website-prior learning improves Qwen3-VL-8B-Instruct from 74.5% to 89.0%, an absolute gain of 14.5 percentage points. The resulting UI-aware model also exceeds Gemini-3.1-Pro by 1.5 points on this test. Because all evaluation interactions are unique and excluded from the website maps used for UI data construction, the improvement cannot be attributed to direct memorization of the tested transitions. Instead, the results indicate that the UI supervision enables the model to acquire transferable knowledge on the target websites.

UI Test

Evaluation Data Construction. To determine whether website-prior learning improves the model’s understanding of website-specific UI knowledge, we construct a held-out UI questionanswering benchmark from online interactions in WebArena. We first run the teacher model, Gemini3-Flash, on all 226 WebArena evaluation tasks and record each interaction step in real time. For every step, we retain the screenshots before and after the interaction, the executed action, and the target element on which the action is performed. We then randomly sample 200 unique interaction steps, subject to the constraint that none of them appears in the website maps used to construct the UI training data. This separation prevents direct overlap between the UI training supervision and the evaluation examples. Following the evaluation protocol of GUI Knowledge Bench (Shi et al., 2025), we use the sampled interactions to construct four categories of UI knowledge questions: Widget Function, which evaluates whether the model understands the purpose of a UI element; Layout Semantics, which evaluates its understanding of the element’s role and spatial relationship within the page layout; Interaction Effect, which asks the model to predict or identify the state change caused by an interaction; and Interaction Type, which tests whether the model can infer the appropriate operation for a state change. Each category contains 50 questions generated from a disjoint subset of 50 interactions. Consequently, the benchmark contains 200 questions in total, with each sampled interaction contributing to exactly one question.

G

UI Dataset Cases

This appendix shows five types of UI data: page description, page QA, element description, forward transition description, and inverse transition description.

Models and Metric. We evaluate the base Qwen3-VL-8B-Instruct model, its UI-aware counterpart obtained after website-prior learning (+UI), and Gemini-3.1-Pro. We report question-answering accuracy over all 200 examples. 18

Algorithm 1 DFS-based Website Crawling Require: Seed URL u0 , configuration Γ Ensure: Persistent graph G 1: Initialize B ← InitAgent(Γ) 2: Initialize G ← InitGraph(Γ) 3: Initialize M ← InitRegistry() 4: B.Navigate(u0 ) 5: Explore(⊥, ⊥, 0, B, G, M, Γ) 6: return G

Algorithm 2 Duplicate-trigger Detection Require: Current state sd , registry M, configuration Γ Ensure: Forms Ωf , raw elements Ωe 1: procedure F ILTER S TATE(sd , M, Γ) 2: Ωf ← ∅, Ωe ← ∅, ∆ ← ∅ where ∆ stores elements pruned from further exploration. 3: (ud , E, L, F, _, _) ← sd 4: (Ve , Vl , Xl , Xf , Du ) ← M 5: t ← GetTemplate(ud ) ▷ Get the URL template

7: procedure E XPLORE(sd−1 , τd−1 , d, B, G, M,

Γ) sd ← B.GetCurrentState() (ud , _, _, _, _, _) ← sd 10: dup ← G.AddState(sd ) where dup indicates whether sd has already been visited. 11: if sd−1 ̸=⊥ ∧τd−1 ̸=⊥ then 12: G.AddTransition(τd−1 ) 13: end if 14: if ShouldStop(dup, sd , d, G, Γ) then 15: return 16: end if 17: (Ωf , Ωe ) ← FilterState(sd , M, Γ) 18: for ωf ∈ Ωf do 19: (sd+1 , τd ) ← B.ExecuteForm(ωf , sd ) 20: eq ← StateEquiv(sd , sd+1 , Γ) 21: if ¬eq then 22: Explore(sd , τd , d + 1, B, G, M, Γ) 23: Backtrack(τd , B, G, Γ) 24: end if 25: end for 26: for ωe ∈ Ωe do 27: if IsExitGuarded(ωe , Γ) then 28: continue 29: end if 30: (sd+1 , τd ) ← B.ExecuteAtomic(ωe , sd ) 31: eq ← StateEquiv(sd , sd+1 , Γ) 32: if ¬eq then 33: Explore(sd , τd , d + 1, B, G, M, Γ) 34: Backtrack(τd , B, G, Γ) 35: end if 36: end for 37: (sd+1 , τd ) ← B.ExecuteScroll(sd ) ▷ Scroll the page 38: eq ← StateEquiv(sd , sd+1 , Γ) 39: if ¬eq then 40: Explore(sd , τd , d + 1, B, G, M, Γ) 41: B.Navigate(ud ) 42: end if 43: end procedure 8: 9:

// List-level pruning for ωl ∈ L do 7: if GetSign(ωl ) ∈ Vl ∨ ωl .xpath ∈ Xl then 8: ∆ ← ∆ ∪ GetItems(ωl ) 9: else 10: Iωl ← SampleItems(ωl , Γ) ▷ Sample representative items 11: ∆ ← ∆ ∪ (GetItems(ωl ) \ Iωl ) 12: end if 13: Vl ← Vl ∪ {GetSign(ωl )} 14: Xl ← Xl ∪ {ωl .xpath} 15: end for 6:

// Form-level pruning for f ∈ F do 17: Ef ← GetItems(f ) 18: if f.xpath ∈ / Xf then 19: Ωf ← Ωf ∪ {f } 20: end if 21: ∆ ← ∆ ∪ Ef 22: Xf ← Xf ∪ {f.xpath} 23: end for 16:

// Element-level deduplication for e ∈ E do 25: if e ∈ ∆ ∨ GetSign(e) ∈ Ve e.xpath ∈ Du [t] then 26: continue 27: end if 28: Ωe ← Ωe ∪ {e} 29: Ve ← Ve ∪ {GetSign(e)} 30: Du [t] ← Du [t] ∪ {e.xpath} 31: end for 32: return (Ωf , Ωe ) 33: end procedure 24:

19

∨

Algorithm 3 Progressive State Comparison Require: Current state sd , candidate state sd+1 , configuration Γ Ensure: eq 1: procedure S TATE E QUIV (sd , sd+1 , Γ) 2: (ud , _, _, _, a11yd , imgd ) ← sd 3: (ud+1 , _, _, _, a11yd+1 , imgd+1 ) ← sd+1 4: (a11yT olerance, useV LM ) ← Γ // Level 1: URL consistency 5: if ud ̸= ud+1 then 6: return false 7: end if // Level 2: Accessibility-tree similarity 8: Cd ← [ ], Cd+1 ← [ ] 9: for all row r in a11yd do 10: append the normalized row r to Cd 11: end for 12: for all row r in a11yd+1 do 13: append the normalized row r to Cd+1 14: end for 15: Nd ← |Cd |, Nd+1 ← |Cd+1 |, N ← max(Nd , Nd+1 ) if |Nd − Nd+1 | < N · (1 − a11yT olerance) 16: then 17: m←0 18: for i = 1 to min(Nd , Nd+1 ) do 19: remove all digits from Cd [i] and Cd+1 [i] if the transformed rows are equal then 20: 21: m←m+1 22: end if 23: end for 24: sim ← m/N 25: if sim ≥ a11yT olerance then 26: return true 27: end if 28: end if // Level 3: Visual fallback 29: if useV LM then 30: if CompareByVLM(imgd , imgd+1 ) = true then 31: return true 32: else 33: return false 34: end if 35: end if 36: return false 37: end procedure

Algorithm 4 Backtracking to a Target State Require: Current transition τd , agent B, graph G, configuration Γ Ensure: Backtrack success flag 1: procedure BACKTRACK(τd , B, G, Γ) 2: (sd , sd+1 , newP ageOpened, opd ) ← τd // Strategy 1: close newly opened page 3: if newP ageOpened = true ∧ B.BackToPreviousPage() then ▷ Close the new page and back to the previous page 4: s′ ← B.GetCurrentState() 5: if StateEquiv(sd , s′ , Γ) then 6: return true 7: end if 8: end if // Strategy 2: browser history back 9: (ud , _, _, _, _, _) ← sd 10: (ud+1 , _, _, _, _, _) ← sd+1 11: if ud ̸= ud+1 ∧ B.NavigateBack() then ▷ Back to the previous page. 12: s′ ← B.GetCurrentState() 13: if StateEquiv(sd , s′ , Γ) then 14: return true 15: end if 16: end if // Strategy 3: deterministic replay from root 17: P ← G.GetShortestPath(sd ) 18: u0 ← B.GetSeedURL() 19: B.ResetContext() 20: B.Navigate(u0 ) 21: for τ ∈ P do 22: (_, _, _, op) ← τ 23: if op.isForm then 24: B.ExecuteForm(op) 25: else 26: B.ExecuteAtomic(op) 27: end if 28: end for 29: s′ ← B.GetCurrentState() 30: return StateEquiv(sd , s′ , Γ) 31: end procedure

20

Page Description

Input Describe this web page in detail based on the screenshot and accessibility tree. Accessibility Tree: ``` {accessibility_tree} ``` Include information about: - Visual layout and structure (header, navigation, main content, sidebar, footer) - All visible UI elements and their apparent functions - Visual styling (colors, typography, spacing)

Output The screenshot displays a web page titled 'List of forums' from a site called Postmill. The header is a solid red bar containing the site name 'Postmill', links for 'Forums' and 'Wiki', a central search bar, and user-related icons for notifications, 'Submit', and a user profile dropdown for 'spiderman'. Below the header, a white main content area includes a sub-navigation bar with tabs for 'Forums', 'Tags', and 'Alphabetical', along with a 'Submissions' sort dropdown. A prominent red 'Create forum' button is located on the right. The main content is a vertical list of forums (e.g., AskReddit, relationship_advice, worldnews), each showing its '/f/' path, title, a short description, a 'Subscribe' button with a subscriber count (currently 0 for all), and the total number of submissions. The typography is clean and sans-serif, with forum titles in bold black and paths in red.

Figure 5: Example of page description data in the UI dataset.

21

Page Question & Answering

Input Look at this web page screenshot and accessibility tree, then answer the following question: Accessibility Tree: ``` {accessibility_tree} ``` Question: How many comments are listed for this post according to the text link below the deleted post content?

Output 17 comments

Figure 6: Example of page QA data in the UI dataset.

22

Element Description

Input The highlighted element in this screenshot is about to be interacted with. Current page accessibility tree: ``` {accessibility_tree} ``` Target element: <button>Post</button> Describe the functionality of the interacted element and how users would typically interact with it in the interface.

Output The 'Post' button is a submit element for the comment form, allowing users to publish their typed text as a new comment on the discussion thread.

Figure 7: Example of element description data in the UI dataset.

23

Forward Transition Description

Input Given the current page state and the action to be performed, predict how the page will change. Current page accessibility tree: ``` {accessibility_tree} ``` Action to perform: Click on [1298] link 'allentown' url: http://localhost:9999/f/allentown Describe only what changes you expect to see on the page after this action.

Output The page has transitioned from an alphabetical directory of all forums to the specific '/f/allentown' forum page. The new view displays a list of submissions and comments specific to that forum, a sidebar with a 'Subscribe' button, forum statistics, and a 'Toolbox' section for moderators.

Figure 8: Example of forward transition description data in the UI dataset.

24

Inverse Transition Description

Input Compare these two screenshots showing a web page before and after a user action. Accessibility Tree (Before): ``` {accessibility_tree_before} ``` Accessibility Tree (After): ``` {accessibility_tree_after} ``` Based on the visual changes and accessibility tree differences, predict what action the user took. Your answer should include: 1. A description of what changed 2. The predicted action including the action and the node (e.g., 'Click on [576] link 'add the blog')

Output Changes observed: The user profile popover menu for 'BzhizhkMard' has disappeared from the screen. In the comment section below, a new reply interface has appeared under the first comment, consisting of a text input area, a 'Formatting help' link, and a 'Post' button. Additionally, sorting tabs for 'Nested' and 'Linear' views have become visible above the comment thread. Predicted action: Click on [3048] strong ''

Figure 9: Example of inverse transition description data in the UI dataset.

25

H

Prompts

Input: ``` <ul> [21]<a> Black Friday </a> [22]<a> Amazon Haul </a> [23]<span> Medical Care </span> [24]<button> </button> [25]<a> Best Sellers </a> [26]<a> Amazon Basics </a> [27]<a> New Releases </a> [28]<a> Books </a> [29]<a> Registry </a> [30]<span> Groceries </span> [31]<button> </button> [32]<span> Gift Cards </span> </ul> ```

This appendix provides the main prompts used in SynWeaver. You are a webpage structure analysis model. Your task is to determine whether the elements inside an HTML list (such as `<ul>` or `<ol>`) represent **repeated functionality** (only one element or one group of elements needs to be clicked) or **distinct functionality** (all elements must be clicked). You will be given the textual structure of a list, including each element's index and text content. Your goal is to analyze the semantics and structural patterns to decide whether the list items lead to **similar ** or **different** functional outcomes.

Output: ```json { "decision": "all", "reason": "The list items represent distinct functional entry points rather than repeated or symmetric options. Items such as 'Black Friday', 'Amazon Haul', 'Medical Care ', 'Best Sellers' all lead to different content categories or features. They do not share a repeated structure or identical functional behavior. Therefore, each item must be treated as a unique function.", "selected_indices": [] } ```

## Objective 1. **If the list items represent repeated or symmetric functionality**: Only **one** representative item or **one group of items** should be clicked. 2. **If the list items represent distinct functionality**: All items must be clicked. ## Classification Criteria ### **1. Treat the items as *functionally identical* ( decision = "single") if:**

### Example 2

All or most items exhibit these characteristics:

Input: ``` <ul> [12]<img> </img> [13]<span> Outstanding Fiction of the past 25 years </span> [14]<img> </img> [15]<span> Unforgettable Nonfiction of the past 25 years </ span> [16]<img> </img> [17]<span> Most buzzworthy books of the past 25 years </ span> [18]<img> </img> [19]<span> Editors' personal favorites </span> </ul> ```

- Text content represents members of the same category (e.g ., product categories, product titles, tags, results) - Elements have a highly repetitive structure (e.g., repeated `<li>`, repeated `<a>`, repeated `<div>` templates) - Items correspond to the same type of resource (e.g., products, categories, search results) - Clicking each item typically leads to pages of the **same structure but different content** (e.g., different product detail pages) - The list visually or semantically presents a "set of options of the same type" If these conditions are met, treat the list as **repeated functionality**.

Output: ``` { "decision": "single", "reason": "The list items represent a set of thematically similar options: each entry pairs an image with a descriptive label for a specific curated book collection. The structure is repetitive, and clicking any item is expected to lead to a page of the same functional type (a book collection page), differing only in content. Therefore , these items exhibit repeated functionality.", "selected_indices": [12, 13] } ```

### **2. Treat the items as *functionally different* ( decision = "all") if:** Any of the following are true: - Text content clearly indicates different action targets ( e.g., "new communication", "Sign In", "search communication ", "profile") - Items represent different task categories (e.g., login, registration, account management) - The list acts as a navigation menu with distinct entry points - Clicking each item evidently results in **different functionality**, not symmetrical options

Prompt 1: Prompt for sampling representative items from the list container.

If any such condition applies, treat the list as **distinct functionality**. ## Output Format You must output a JSON object in the following format: ```json { "decision": "single" or "all", "reason": "Explanation of the decision", "selected_indices": when the decision is "single", return [list of indices]. Otherwise, return an empty list. } ```

You are an intelligent form filling assistant. Your task is to generate reasonable test values for each input field in a web form based on the form context and field semantics. ## Context You will receive: 1. **Page URL**: The current page URL to help understand the context 2. **Accessibility Tree**: A structured representation of the page content around the form 3. **Form Fields**: A list of input fields in the form, each with:

## Example ### Example 1

26

} ```

- Index number (1, 2, 3, ...) - Tag type (input/textarea/select) - Input type (for input elements: text, email, password, etc.) - Name attribute - Aria-label - Placeholder text - Current text content

Prompt 2: Prompt for filling the form container.

You are an expert web page analyzer. Your task is to determine whether two screenshots represent the **same functional state** of a web page, even if some minor visual elements differ.

## Your Task Generate appropriate test values for each form field. The values should be: 1. **Semantically appropriate**: Match the expected content type (email for email fields, phone for phone fields, etc .) 2. **Realistic**: Look like real user input, not obviously fake 3. **Valid**: Pass basic validation (proper email format, valid phone format, etc.) 4. **Contextual**: Consider the form's purpose based on URL and accessibility tree

# Background During web crawling and state replay, certain transient UI elements (like toast notifications, temporary alerts, or dynamic content) may appear or disappear. However, the underlying page state remains the same. We need to identify whether two screenshots represent the same functional page state while ignoring these superficial differences. # Task

## Output Format

Compare the two provided screenshots and determine if they represent the **same web page state**.

Return a JSON object with a "values" array containing the fill value for each field **in order**:

# What to Consider as the SAME State

```json { "values": ["value_for_field_1", "value_for_field_2", " value_for_field_3", ...], "reasoning": "Brief explanation of the values chosen" } ```

Two screenshots should be considered the **same state** if they share: 1. **Core Page Structure**: - Same page layout and main content areas - Same navigation elements (header, footer, sidebar) - Same primary content sections

**Important**: - The array length MUST match the number of input fields - Each value corresponds to the field at that index position - For select elements, provide the option text to select - For password fields, use a strong test password like " TestPass123!" - For date fields, use ISO format (YYYY-MM-DD)

2. **Functional Elements**: - Same interactive elements (buttons, links, forms, input fields) - Same product listings, article content, or data tables - Same menu items and navigation options 3. **Page Identity**: - Same URL path (if visible) - Same page title or heading - Same main purpose/function

## Examples ### Example 1: Login Form

# What to IGNORE (Acceptable Differences)

Input: - URL: https://example.com/login - Fields: 1. input[type=email] name="email" placeholder="Enter your email" 2. input[type=password] name="password" placeholder=" Password"

The following differences should be **ignored** when comparing states: 1. **Transient Notifications**: - Toast messages (success/error/warning notifications) - Temporary alert boxes - Popup hints or tooltips - Tutorial/onboarding overlays - Cookie consent banners

Output: ```json { "values": ["testuser@example.com", "TestPass123!"], "reasoning": "Standard login form with email and password fields" } ```

2. **Dynamic Content**: - Timestamps (e.g., "2 minutes ago" vs "5 minutes ago") - Real-time counters (shopping cart quantity, notification badges) - Live data feeds (stock prices, weather updates) - Advertisement content - Personalized recommendations

### Example 2: Registration Form Input: - URL: https://shop.example.com/register - Fields: 1. input[type=text] name="first_name" aria-label="First Name" 2. input[type=text] name="last_name" aria-label="Last Name" 3. input[type=email] name="email" 4. input[type=tel] name="phone" placeholder="Phone number "

3. **Visual State Changes**: - Hover effects or focus states - Loading spinners or progress indicators - Scroll position - Animation states - Dropdown menu open/closed states 4. **Session-Specific Elements**: - Session IDs or tokens in URLs - CSRF tokens - Temporary promotional banners - A/B testing variations (minor UI tweaks)

Output: ```json { "values": ["John", "Smith", "john.smith@example.com", "555-123-4567"], "reasoning": "E-commerce registration form requiring personal contact information"

# Decision Criteria - **SAME**: If the core page structure, main content, and functional elements are identical (ignoring the acceptable differences listed above)

27

- **DIFFERENT**: If there are substantial differences in layout, content, navigation, or functionality

You will receive: 1. A "before" screenshot showing the page with a marked element 2. An "after" screenshot showing the page after the action was performed 3. Information about the action taken

# Output Format Provide your analysis in the following JSON format: ```json { "decision": "same" or "different", "confidence": 0.0 to 1.0, "reasoning": "Simple explanation of your decision", } ```

Your task is to provide TWO types of analysis: ## Part 1: Element Description Describe the functionality of the interacted element and how users would typically interact with it in the interface. ## Part 2: State Transition Description Describe the changes between the before and after states.

Prompt 3: Prompt for state comparison.

IMPORTANT: In the state transition description, describe ONLY what changed on the page. Do NOT mention the action taken (click, fill, etc.) - focus purely on the observable differences between the two screenshots.

You are an expert UI analyst tasked with analyzing web page screenshots and accessibility trees. Your analysis will be used to train AI models to understand web interfaces.

## Output Format You MUST respond in the following JSON format: ```json { "element_description": "Description of the element's functionality...", "state_transition": "Description of how the page changed (without mentioning the action)..." }

You will receive: 1. A screenshot of a web page 2. The page's accessibility tree Your task is to provide TWO types of analysis: ## Part 1: Page Description Provide a comprehensive description of the page that includes: - Visual layout and structure (header, navigation, main content, sidebar, footer) - All visible UI elements and their apparent functions - Visual styling (colors, typography, spacing)

Analyze this web page interaction: Action Performed: {action} Element: {element} URL Before: {from_url} URL After: {to_url}

## Part 2: Page QA Generate ONE informative question-answer pair about the page. The question should: - Ask about specific information visible on the page - Be answerable ONLY by looking at the screenshot - Focus on extracting factual information (not opinions) - Examples: "What is the main heading?", "How many products are displayed?", "What navigation options are available?"

The first image shows the page BEFORE the action (with the target element highlighted). The second image shows the page AFTER the action was performed. Please provide: 1. A description of the element's functionality 2. A description of how the page changed (without mentioning what action was taken)

## Output Format You MUST respond in the following JSON format: ```json { "page_description": "Your detailed page description here...", "question": "Your question about the page content...", "answer": "The answer based on the screenshot..." }

Remember to respond in the specified JSON format.

Prompt 5: System prompt and user prompt for transitionlevel UI data generation.

Analyze the following web page: You are a GUI (Graphical User Interface) expert capable of analyzing interface changes and envisioning executable tasks or instructions. Given a GUI interface change caused by an action (e.g., clicking or typing) and the corresponding element highlighted in red boxes, you are required to analyze the interface and generate related tasks.

URL: {url} Accessibility Tree: ``` {accessibility_tree} ``` Please provide: 1. A detailed description of this page 2. One question-answer pair about specific information visible on the page

Your task is to envision tasks based on the current action and the resulting changes in the screenshots. The output should include three components: 1. Sub-Instruction: Create a natural language instruction for the current action based on the interface changes it caused. The instruction should be concise, clear, and actionable, incorporating specific details critical to the task, such as elements, file names, timestamps, or other relevant content visible in the screenshots. For example: - "Click on the 'Add to Cart' button next to the product to add it to your shopping cart." - "Type 'OpenAI' into the search bar to find relevant articles." - "Scroll down to view the latest blog posts on the homepage."

Remember to respond in the specified JSON format.

Prompt 4: System prompt and user prompt for pagelevel UI data generation.

You are an expert UI analyst tasked with analyzing web page interactions. Your analysis will be used to train AI models to understand how UI elements work and how pages change after user actions.

2. Analysis: Carefully analyze the before-and-after

28

screenshots step by step, focusing on the changes caused by the action. Then, examine key elements in both screenshots and consider possible operations based on these elements. For example: "The previous screen displayed the main interface of a shopping website, featuring multiple product categories and several showcased items. After clicking the 'Sign Up' button, the interface transitioned to a login page where an email and password can be entered to log into an account. The login page also provides other options, such as recovering a password, creating a new account, or logging in with a Google account."

"{high_level_task}" 2. History of Actions ({hint_for_history}): {previous_actions} 3. Current Page (only current view, not full page, you may need to scroll to see more): - URL: {url} - Accessibility Tree (Page Context): {page_context}

3. High-Level Instruction: Based on the before-and-after screenshots, the action, and the analysis, generate a highlevel task that you believe can be completed within the current interface. There are three types of tasks: - Information seeking: The user wants to obtain certain information from the webpage, such as product details, reviews, map information, or route comparisons. Please propose clear and specific questions that need an explicit answer, and avoid asking for summary-type questions, such as "summarize the information about a product." - Site navigation: The user wants to navigate to a specific page or state. - Content modification: The user wants to modify the content of a webpage or its settings.

- Screenshot (only current view, not full page): {img_info} **Available Low-Level Actions (exact JSON formats)** Page Operation Actions: - CLICK: {{"type": "CLICK", "element_id": <int>, "value": ""}} - TYPE (default behavior is to press Enter after typing unless you explicitly set press_enter_after to 0): {{"type": "TYPE", "element_id": <int>, "value": "text to type <string>"}}

The high-level instruction should be creative. You need to deeply analyze the elements and executable actions on the interface to generate realistic, valuable, and executable tasks that can be completed within the current GUI. The instruction should be specific, actionable, and goaloriented, ensuring the task can be completed on the current GUI by including all critical specifics such as file names , relevant timings, or required details.

- HOVER: {{"type": "HOVER", "element_id": <int>, "value": ""}}

Below is a brief description of the current website: { website_intro}

- SCROLL: {{"type": "SCROLL", "element_id": "", "value": "up" or " down"}}

- PRESS (keyboard shortcut. The value MUST be either exactly ONE key, e.g., ArrowDown, or ONE key chord joined by "+", e.g., Ctrl+V / Cmd+V): {{"type": "PRESS", "element_id": "", "value": "key_comb < string>"}}

Here are some examples of High-Level Instruction for reference: {task_examples}

Tab Management Actions: - NEW_TAB (open a new, empty browser tab): {{"type": "NEW_TAB", "element_id": "", "value": ""}}

Current Action: {current_action_str}

- TAB_FOCUS (switch focus to a specific tab by its index, starting from 0): {{"type": "TAB_FOCUS", "element_id": "", "value": "< tab_index as int>"}}

Website Name: {website_name} Before-action Screenshot: <image is provided in the first attachment> (the action's target element is highlighted in red box if applicable)

- CLOSE_TAB (close the currently active tab): {{"type": "CLOSE_TAB", "element_id": "", "value": ""}}

After-action Screenshot: <image is provided in the second attachment>

URL Navigation Actions: - GOTO (navigate directly to a URL): {{"type": "GOTO", "element_id": "", "value": "url <string >"}}

Please generate tasks that can be completed on the current platform, and avoid tasks that are unrelated to the current website.

- GO_BACK: {{"type": "GO_BACK", "element_id": "", "value": ""}}

You MUST respond in the following JSON format: (no extra commentary): ```json {{ "Sub-Instruction": "xxx", "Analysis": "xxx", "High-Level-Instruction": "xxx" }} ``` RETURN ONLY THE JSON I ASKED FOR.

- GO_FORWARD: {{"type": "GO_FORWARD", "element_id": "", "value": ""}} Completion Actions: - NONE (use ONLY when the task is completed and you have the final answer): {{"type": "NONE", "element_id": "", "value": "summary content or final answer <string>"}} - STOP (use ONLY when the task is impossible to complete): {{"type": "STOP", "element_id": "", "value": "the reason for the STOP <string>"}}

Prompt 6: Prompt for task synthesis, adopted from OSGenesis (Sun et al., 2025a) and SynthAgent (Wang et al., 2025b).

**Critical Rules for Success** 1. You must issue only actions that are valid given the current observation (accessibility tree and screenshot). 2. Only propose ONE atomic action; actions must be executable independently. 3. Prefer actions grounded by element IDs present in the accessibility tree when using CLICK/TYPE/HOVER. 4. You MUST provide meaningful and non-empty value if the action type is in {{TYPE, SCROLL, GOTO, NONE, STOP, TAB_FOCUS}}. 5. When you believe the task is complete (e.g., you have the answer), use NONE with the final answer in value. 6. Be concise and avoid redundant or risky actions; ensure

You are a GUI (Graphical User Interface) Web Agent expert capable of long-horizon planning and executing high-level tasks on a website. Based on the observations and the highlevel task to complete, generate the next low-level instruction. **Information** 1. High-Level Task (your ultimate goal to finish):

29

each action clearly advances the task. 7. Use STOP only when: - The task lacks necessary information - The target does not exist (hallucination) - The task is harmful or inappropriate - Multiple attempts (>=3) have failed to make progress

unachievable given the current context.

8. You MUST first generate a "state_observation_summary" to observe the current environment, then take a step-by-step "reasoning" to decide the next action. 9. The high-level task often requires MULTIPLE steps to complete. Do NOT expect to finish in a single action. 10. You MUST actively decide the next step. Do NOT choose " NONE" or "STOP" unless you are sure the task is finished or impossible. 11. Choose element IDs from the accessibility tree and use them directly in CLICK/TYPE/HOVER actions. 12. If the page does not change after an action, try scrolling to see more elements. 13. When typing dates, use the format "MM/DD/YYYY".

**Example**: - Bad (Instructional): "Click the blue 'Add to Cart' button and then proceed to checkout." (Too specific, loses intent ) - Good (Intent-Driven): "Add the selected item to the shopping cart." (Clear goal, lets agent decide actions)

**Goal**: **Clarify user intent and align with the current trajectory, BUT DO NOT describe steps.** The refined task should focus on *WHAT* to achieve (the goal state), not *HOW* to do it (the actions).

**How to apply**: - Change the original intent if necessary to fit the trajectory's progress. - Clarify the *target entity* or *specific condition* required for success. - Keep `step_order` as the original order [{original_order }] (excluding the STOP step) - Set `modified_reasonings` to empty dict {{}}

**Output Requirements** You MUST return a JSON dictionary with the following format (no extra commentary): {{ "state_observation_summary": "your 1-3 sentence summary of the current state relevant to the task", "reasoning": "your step-by-step reasoning to decide the next action", "next_action": {{ "action": {{"type": "XXXX", "element_id": <int or "">, "value": <string or "">}} }} }} RETURN ONLY THE DICTIONARY WITHOUT ANY COMMENTARY.

### Strategy 2: Task-Trajectory Co-Optimization **Use when**: The trajectory contains redundant, incorrect, or out-of-order steps that prevent task completion, even with intent clarification. **Example scenario**: Agent clicked wrong tabs, navigated back and forth, or executed steps in illogical order. **How to apply**: - Refine the task description to strictly reflect the core user intent. - Change the original intent if necessary to fit the trajectory's progress and make it achievable in 2-3 steps. - Remove or reorder trajectory steps to create a coherent path: - **Delete**: Omit step numbers (e.g., [1, 3, 5] removes steps 2, 4) - **Reorder**: Change sequence (e.g., [1, 4, 3, 5] swaps steps 3 and 4) - **Combined**: [1, 5, 6] means delete steps 2-4 and keep rest - Update reasoning for affected steps in ` modified_reasonings` to ensure logical flow

Prompt 7: Prompt for task execution on the website.

You are a GUI Web Agent expert specializing in task and trajectory optimization. The execution agent has encountered a STOP action, indicating it cannot complete the current task with the existing trajectory. **Your Primary Goal**: Refine the task description so the execution agent can overcome its current obstacle and complete the task within 2-3 additional steps.

**Critical**: After modifications, the trajectory must be coherent: - Each step should logically follow the previous one - The reasoning should reflect the new context (e.g., " Since we removed the navigation step, now proceed directly to...") - The modified trajectory should lead naturally to task completion in 2-3 more steps

## Current Situation **High-Level Task**: "{current_task}" **Previous High-Level Tasks Attempted**: {previous_tasks}

--## Decision Framework

**Execution Trajectory** (steps 1-{total_steps}): {previous_actions}

Ask yourself these questions in order:

**Current Page URL**: {current_url}

1. **Root Cause Analysis**: Why did the agent issue a STOP action? - Task was impossible/invalid? - Task was unclear/ambiguous? - Trajectory led to wrong page/state? - Agent got stuck in a loop?

**Current Page Context (Accessibility Tree)**: {page_context} **Screenshot**: {img_info}

2. **Can Task-Only Optimization Succeed?** - Does the current page state enable task completion? - Will clarifying or changing the **goal state** guide the agent to success in 2-3 steps? - Are the executed steps (before STOP) generally correct ?

**Important Note**: The last step is a STOP action which produced no actual operation and will be automatically discarded. Focus your optimization on making the trajectory effective before that point. ---

- If YES, use Strategy 1 (Task-Only). - If NO, proceed to Question 3.

## Optimization Strategies

3. **What Trajectory Issues Exist?** - Are there redundant/unnecessary steps? - Are steps in the wrong order? - Did early mistakes prevent success?

Your goal is to create a refined task that enables the agent to complete its objective within 2-3 more steps. Choose between: ### Strategy 1: Task-Only Optimization **Use when**: The executed trajectory represents meaningful progress, but the original task was too ambiguous or

- If YES, use Strategy 2 (Co-Optimization).

30

4. **Coherence Check** (for Strategy 2): - After reordering/deleting, does the trajectory tell a logical story? - Do you need to update reasoning to maintain coherence? - Will the modified trajectory naturally lead to completion in 2-3 steps?

1. **Understanding the Context**: What was the agent trying to accomplish? What meaningful progress was made before the issues? 2. **Extracting Valid Steps**: Identify which steps represent meaningful progress and should be kept. You may: - **Delete** steps that are redundant, erroneous, or part of a loop - **Reorder** steps if needed for logical coherence - **Update reasoning** for steps whose context changes due to deletion/reordering - You **CANNOT** add new steps - only select from existing ones

--## Output Requirements Return a JSON object with EXACTLY this structure: {{ "analysis": "Detailed analysis: (1) Why STOP occurred (2) Root cause (3) Which strategy is appropriate (4) How refined task enables 2-3 step completion", "strategy": "task_only" or "co_optimize", "need_refine": true or false, "refined_task": "The optimized task description that enables completion in 2-3 more steps (or empty string if no refinement needed)", "step_order": [1, 2, ...], "modified_reasonings": {{ // ONLY include steps whose reasoning needs updates to maintain trajectory coherence after reordering/deletion // Key: step number (as string), Value: updated reasoning text // Example: {{"3": "Now that we've navigated back, proceed directly to click the search button"}} }}, "completion_estimate": "Specific description of the 1-3 steps needed to complete the refined task from current state" }}

3. **Handling Different Issues**: - **MISSING_END**: The trajectory lacks a proper NONE ( completion) action. If the existing steps accomplish a meaningful sub-goal, rewrite the task to match what was achieved, then indicate we need to append a NONE action. - **LOOP_DETECTED**: Remove the redundant loop iterations, keeping only the first occurrence of each unique action in the pattern. - **Failed/STOP trajectories**: Extract the meaningful portion of the trajectory that accomplishes a coherent subtask. Rewrite the task description to match this extracted trajectory. 4. **Ensuring Coherence**: The final trajectory must: - Have steps that logically follow each other - Match the (possibly rewritten) task description - End with a completion state (we'll add NONE if needed) --## Output Requirements

**Critical Rules**: 1. The refined task MUST be achievable within 2-3 more steps from the current page state. 2. **INTENT over INSTRUCTION**: The refined task must describe the **Goal State** (e.g., "Find the pricing page") , NOT a list of actions (e.g., "Click menu, then click pricing"). 3. `step_order` must only contain integers from 1 to { total_steps} (the STOP step should be removed). 4. Each step number can appear at most once in `step_order `. 5. If `strategy` is "task_only", `step_order` must be [{ original_order}]. 6. For "co_optimize" strategy, ensure modified trajectory is logically coherent. 7. Update `modified_reasonings` for any step whose context changed due to reordering/deletion.

Return a JSON object with EXACTLY this structure: {{ "analysis": "Detailed analysis of the trajectory issues and your reconstruction strategy", "can_reconstruct": true or false, "reconstructed_task": "The task description (original or rewritten to match extracted trajectory)", "step_order": [list of step numbers to keep, in order, e.g., [1, 2, 4, 5] means delete step 3], "modified_reasonings": {{ // Only include steps whose reasoning needs update // Key: original step number (as string), Value: new reasoning // Example: {{"4": "After clicking the search button, now viewing results..."}} }}, "needs_none_action": true or false, "none_action_value": "Summary/answer for NONE action if needs_none_action is true", "reconstruction_summary": "Brief summary of what was changed and why" }}

**Focus**: Prioritize enabling the agent to complete the task quickly (2-3 steps) over preserving all trajectory steps. Ensure the task remains a high-level goal, not a low -level script. RETURN ONLY THE JSON OBJECT WITHOUT ANY COMMENTARY.

**Critical Rules**: 1. `step_order` must only contain integers from 1 to { total_steps} 2. Each step number can appear at most once 3. You can only delete or reorder existing steps, NOT add new ones 4. If `can_reconstruct` is false, explain why in `analysis` 5. The reconstructed trajectory should represent a coherent , completable task 6. If the trajectory is too broken to salvage, set ` can_reconstruct` to false

Prompt 8: Prompt for task-trajectory collaborative refinement.

You are an expert at analyzing and reconstructing web agent trajectories. A trajectory has failed validation and needs to be reconstructed. ## Original High-Level Task "{high_level_task}"

**Example for LOOP_DETECTED**: If steps 3-4-5-6 form a loop pattern (3-4 repeated), keep only [1, 2, 3, 4] and update reasoning if needed.

## Trajectory ({total_steps} steps) {trajectory_history}

**Example for Failed Trajectory**: If a 10-step trajectory ended in STOP but steps 1-6 accomplished "navigating to product page", rewrite task to "Navigate to the product page" and keep [1, 2, 3, 4, 5, 6].

## Validation Issues Detected {validation_issues} ---

RETURN ONLY THE JSON OBJECT WITHOUT ANY COMMENTARY.

## Your Task Analyze the trajectory and validation issues, then reconstruct a valid task-trajectory pair by:

Prompt 9: Prompt for task-trajectory reconstruction in the post-verification.

31

Related documents

• HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions#153064
• RepoMAS: Solving Progressively Specified Tasks with Issue-Driven Multi-Agent Systems#1116821
• StateFork: Branchable Infrastructure for Agent Exploration#1169656
• StateFork: Branchable Infrastructure for Agent Exploration#1143249
• On Device Agentic Operation Caches -- Classifier-Centric NL-to-Action Generation#1117853