Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning Tianyi Men1,2 , Zhuoran Jin1,2 , Pengfei Cao1,2 , Yubo Chen1,2 , Kang Liu1,2 , Jun Zhao1,2,† 1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China {tianyi.men, zhuoran.jin, pengfei.cao, yubo.chen, kliu, jzhao}@nlpr.ia.ac.cn
Abstract
1
Introduction
The multimodal web agent is an attractive solution, which can assist humans in operating on unfamiliar websites and handling repetitive GUI tasks (Wang et al., 2024a; Ning et al., 2025; Tang et al., 2025a). The core ability of the agent is task planning, which enables it to decompose a complex task into executable actions (Li et al., 2025d; Cao et al., 2025; Wei et al., 2025). Due to the high interaction costs †
Corresponding author.
Previous Methods (Task→Trajectory)
Type
Atomic Task Training
Small MLLM
Poor Compositional Gen. Wrong distilled trajectory: yellow snowboard with 3 stars
Task: Find me a blue snowboard with 4+ stars.
Coarse Task Training
Our Method: PEEU (Task→Trajectory →Experience →Task)
arXiv:2606.27330v1 [cs.CL] 25 Jun 2026
Multimodal web agents can assist humans in operating repetitive GUI tasks, where effective task planning is essential for decomposing complex tasks into executable actions. While small open-source MLLMs are cost-efficient and privacy-preserving compared with commercial large models, they suffer from weak planning and limited cross-website generalization. To address these limitations, we introduce the planning experience exploration and utilization (PEEU) method, which autonomously explores environments to discover experiences and utilizes hindsight experience to synthesize strictly aligned, high-level training data. To quantitatively analyze the generalization behaviors driving this performance, we propose the task decomposition hierarchical analysis framework (TDHAF) to systematically study compositional generalization across three task granularities: low, middle and high levels. Our analysis reveals that mastering low-level atomic skills does not guarantee high-level planning competence, while high-level task training yields stronger OOD generalization. Experiments on real-world benchmarks demonstrate PEEU’s superior effectiveness: our 7B model achieves 30.6% accuracy, outperforming the much larger Qwen2.5-VL-32B model. These demonstrate constructing hindsight high-level tasks and leveraging experiences is crucial for OOD planning abilities of small MLLMs.
Click
Small MLLM
Misaligned Trajectory & Weak Constraints
Hindsight Procedural Experience
Task: Find me a blue snowboard with 4+ stars.
Step 1: Agent Opens shopping site. click,3 stars
Web Agent
click, site type, snowboard
Interact
Step 2: Agent Searches snowboard. Step 3: Agent clicks a product with 3 stars.
Turn trash into treasure! PEEU Task (Aligned): Find me a yellow snowboard with 3 stars.
High Quality
Step 4: Agent clicks a product in yellow color. Step 5: Agent Clicks product link.
Chrome
Exploration Tree
Small MLLM
Figure 1: The overview of planning experience exploration and utilization method.
and privacy risks of commercial large models, using small open-source multimodal large language models (MLLMs) is a promising approach (Belcak et al., 2025). However, small MLLMs exhibit weak planning ability and limited generalization. Thus, enhancing their planning with limited data is urgent (He et al., 2024). In comparison, humans can make plans by utilizing experiences from interaction and exploration with the environment (Ross, 1989; Anderson, 2013). Inspired by the human learning process, agents should (1) autonomously set their own learning goals in the environment and improve their abilities through interaction and exploration, and (2) summarize and utilize hindsight experiences from the past to guide future decisions (Silver and Sutton, 2025; Cai et al., 2025). Recent studies focus on utilizing experiences in the post-training stage to train models. As shown in Figure 1, these approaches can be categorized into two main streams: (1) Training with atomic-level tasks (Gu et al., 2024; Fan et al., 2025). These methods compare changes before and after environment observations to extract experiences. The experi-
ences are then used to synthesize atomic-level tasks such as clicking, typing, and scrolling to train the model. However, it remains unclear whether training on atomic-level tasks can effectively generalize to high-level tasks. Hence, it is urgent to propose a framework to study the compositional generalization of web agent task planning. (2) Training with coarse high-level tasks (Logeswaran et al., 2025; Trabucco et al., 2025). These methods leverage task-based exploration trajectories to train the model with coarse high-level tasks, like finding a snowboard with constraints. However, trajectories of coarse high-level tasks suffer from misalignment and a lack of stricter constraints. This limits the generalization ability in high-level tasks. Therefore, it is necessary to develop a method to synthesize trajectories that are better aligned and strictly constrained by environments. To address these limitations while ensuring a fair comparison using the same scale data, we propose the planning experience exploration and utilization method (PEEU), as shown in Figure 1. Distinct from previous methods that rely on brute-force search to match trajectories with pre-defined goals, we leverage hindsight to inversely align tasks to the collected trajectories, thereby significantly enhancing the quality of high-level data. The framework consists of two stages: planning tree exploration and planning experience utilization. (1) In the planning tree exploration stage, the exploration model autonomously sets goals adapted to the functional characteristics of diverse websites, and then conducts goal-driven exploration in the unfamiliar environment to construct an exploration tree. (2) In the planning experience utilization stage, trajectories are summarized to extract valuable experiences. These experiences are then used to create better aligned and constrained pairs of tasks and trajectories. We evaluate PEEU on seven unseen realworld websites. Under a strictly controlled setting with identical data scales for all methods, PEEU demonstrates superior cross-website generalization. PEEU based on Qwen2.5-VL-7B reaches 30.6% accuracy, marking a significant improvement over the Instruct Model’s performance of 7.8%. To further validate the advantage of high-level tasks over atomic-level tasks, we propose the task decomposition hierarchical analysis framework (TDHAF). Our analysis confirms that mastering atomic skills is insufficient for complex planning, thereby validating PEEU’s emphasis on high-level experience. This framework first defines three
levels of task granularity: low-level tasks, midlevel tasks, and high-level tasks. It further distinguishes between two types of generalization: indomain (ID) and out-of-domain (OOD). Building on this taxonomy, we analyze from three perspectives: (1) ID bottom-up generalization: whether low-level tasks can generalize to high-level tasks indomain. (2) ID top-down generalization: whether high-level tasks can generalize to low-level tasks in-domain. (3) OOD multi-level generalization: what granularity of tasks is better for out-of-domain generalization. The experiments demonstrate following conclusions: (1) Mastering individual lowlevel tasks does not necessarily imply mastery of the corresponding high-level task. (2) Using highlevel tasks makes it easier to generalize downwards in-domain with greater overall coverage. (3) Using high-level task training can enable the model to acquire stronger generalization capabilities for multilevel tasks in OOD. Overall, experiments show that in post-training stage, using low-level tasks cannot effectively generalize to high-level tasks. In summary, our contributions are as follows: (1) We propose the planning experience exploration and utilization method (PEEU), which can autonomously explores and effectively utilizes experiences to enhance the planning generalization abilities of web agents. (2) We propose the task decomposition hierarchical analysis framework (TDHAF) to analyze the compositional generalization ability of models in multimodal web navigation task planning scenarios. (3) PEEU improves cross-website OOD generalization in real online multimodal web navigation tasks, outperforming previous methods across different model scales with the same data scale and training settings.
2
Planning Experience Exploration and Utilization Method
In this section, we introduce the planning experience exploration and utilization method. This is an automatic exploration learning framework that first sets goals adaptively and explores in unfamiliar websites. Then it extracts planning experiences from trajectories and uses them to build aligned and constrained training data. Users only need to provide a URL to be explored, and the framework can freely explore the website, extract and summarize experiences, and then build better aligned and constrained data to train small MLLMs, achieving cross-website generalization capabilities.
User
Stage1: Planning Tree Exploration The user only needs to enter the URL to be explored. https://www.allrecipes.com
Explorer obtains the homepage.
Adapted Goals Task List The explorer sets exploration goals autonomously based on the content of the homepage.
Task1: Find cooking recipes using only vegan ingredients. Task2: Locate a casserole recipe that takes less than an hour to prepare. Task3: Search for recipes featuring tacos with unique fillings.
……
Task100: It must over 4.5 stars , find recipes that children can help prepare.
Stage2: Planning Experience Utilization Coarse Task: It must over 4.5 stars , find recipes that children can help prepare. (Task 100)
Exploration Tree after 10 steps
Misaligned
Lack of stricter constraints food image
food ingredients food directions
Homepage click, 9
Experience Extraction Step1: Enter 'simple recipes' into the search bar to retrieve relevant recipe suggestions.
Step2: Scroll down to view more recipe results for 'simple recipes’.
Step3: Click on the ‘Simple Strawberry Sherbet’ photos with 4 stars to view the detailed recipe page.
Aggregation
Step4: Scroll down to view the ingredients section of the recipe.
Better aligned Stricter constraints
PEEU: Find recipes that children can help prepare, including ingredients, cooking directions, and photos over 4 stars.
Step5: Scroll down to view the cooking directions for the recipe.
after 3 steps
scroll, down
OOD website after 7 steps
SFT/GRPO
after 3 steps
Figure 2: An overview of planning experience exploration and utilization method with two stages.
2.1
Method
The framework is divided into two stages: planning tree exploration and planning experience utilization, as shown in Figure 2. All prompts are shown in Appendix A. Planning Tree Exploration. The autonomous agent requires a shift from passive learning to autonomous learning. It requires self-driven tasks and self-execution exploration. For the self-driven tasks stage, given a website URL, the exploration agent interacts with the homepage s0 (obtained from the URL) through the MLLM M to generate a basic task list D = {d1 , d2 , . . . , dn }, where each task di represents a task to be explored. This process can be expressed as: D = M (s0 , URL).
(1)
Subsequently, for the self-execution exploration stage, the agent performs autonomous exploration based on the task list D, the environment Env (with basic URL as entry point), generating a directed exploration tree R = (V, E) rooted at the homepage, where V is the set of website screens, E is the set of actions between these observations. The exploration process is implemented as: R = Explore(M, D, Env, URL).
(2)
This tree can be expanded into interleaved trajectories of observations and actions, where all trajectories share the same root node. Formally, let τ = {(s0 , a0 ), . . . , (sm , am )} denote a trajectory, where s0 is the shared root state (homepage). at ∈ A represents the action at step t. st+1 ∼ P (·|st , at ) is the subsequent observation. The exploration tree R represents the collection of trajectories from tasks {τi }ni=1 , obtained via the recursive exploration process by M . Planning Experience Utilization. The agent needs to learn from past explorations and use these experiences to build high-level trajectory data. The coarse high-level tasks have two limitations. (1) The tasks and trajectories are not always aligned. For example, the task requires more than 4.5 stars, but the trajectory only reaches 4 stars. (2) The task lacks stricter constraints for unknown environments, because the websites are partially observable environments. The constraints of the unknown environment must come from real exploration, and the homepage information cannot provide them, such as ingredients and preparation directions. Using such mismatched data causes the agent to learn incorrect patterns and miss key details. Thus, recasting these explorations into accurate experiences is vital for ensuring high-quality training signals.
In the experience extraction stage, the MLLM M compares before-action state and after-action state to extract atomic experiences: ϵt = M (st , at , st+1 ),
(3)
where st and st+1 are the visual observations before and after action at , respectively. A trajectorylevel experience µ can be represented as a sequence of atomic experiences: µ = (ϵ1 , ϵ2 , . . . , ϵT ).
(4)
The agent then fuses these sequences of atomic experiences into refined high-level tasks that are both more aligned with real outcomes and stricter in the constraints. Formally, define a mapping Φ with M that aggregates the experiences into PEEU ˜ forming the collection D̃ of PEEU tasks: task d, D̃ = (d˜1 , d˜2 , . . . , d˜n ) = Φ(µ1 , µ2 , . . . , µn , M ). (5) In the training stage, the agent’s goal is to learn a policy π : S × H × D̃ → A, that maps the current state st ∈ S, the history ht ∈ H0:t , and the task description d˜ ∈ D̃, to the next action at ∈ A. We use SFT and GRPO (Shao et al., 2024) for training. The details are shown in Appendix A and B. 2.2
Exploration and training settings. (1) For the exploration phase, we use GPT-4o for exploration with a maximum step length of 15 in 0.1k or 2k exploration tasks. For the experience summarization phase, we use GPT-4o to summarize the changes in the browser’s state before and after the exploration. (2) For the training phase, all our experiments are conducted on Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7B-Instruct. For the SFT model, the batch size is 16, the learning rate is 5.0e-6, and the number of training epochs is 5, using the llama-factory (Zheng et al., 2024b) training framework. For the GRPO model, the batch size is 20, the learning rate is 1.0e-6, the rollout size is 10, and the number of training epochs is 7, using the verl (Zheng et al., 2025) framework. All experiments are performed on 4 A800 GPUs. For fair comparison, all experiments use identical trajectory scales. (3) Our experimental setup consists of two configurations: the first involves training on 0.1k trajectories derived from Allrecipes, while the second utilizes 2k trajectories from a previously unseen website. We test on seven additional websites that were entirely excluded from the training data. More details are shown in Appendix C.
Experimental Settings
Baseline. (1) Atomic-Prompt (Wang et al., 2024b) uses the input task to retrieve related atomic experiences. The number of retrieved atomic experiences is set to 10. These experiences are used as prompts to serve as contextual input. (2) TrajectoryPrompt (Wang et al., 2024b) uses the input task to retrieve one trajectory-level experience according to its query as the prompt. (3) Coarse (Logeswaran et al., 2025; Trabucco et al., 2025) uses the original exploration task as the training task. (4) Atomic (Gu et al., 2024; Fan et al., 2025) uses the atomic operation task as the training task. In addition, all the training parameters are kept the same. And all methods are controlled to use the same amount of data to ensure a fair comparison. Evaluation. We evaluate the planning capabilities of the models on real-world multimodal benchmark WebVoyager (He et al., 2024). The test set covers diverse real multimodal online websites, including cooking, shopping, research, code, map, study and other categories. Follow the standard evaluation procedure of WebVoyager (He et al., 2024), the benchmark uses the trajectory-level success rate as the final accuracy.
2.3
Results and Analysis
Adapt the task to fit the trajectory with experience. As shown in Figure 2, coarse trajectory tasks face problems of mismatch and a lack of strict constraints. For example, in the coarse task, the rating is 4.5, but the trajectory shows only 4 stars, which causes a mismatch. Therefore, constraints should be derived from exploration experience. By using experience to modify tasks, we can create more aligned and strictly constrained advanced tasks. As shown in Table 1, for the 7B model, trained with 2k trajectories, PEEU-SFT achieves a remarkable overall score of 30.6%, which not only significantly outperforms the competitive CoarseSFT of 19.0% baselines but also surpasses the much larger Qwen2.5-VL-32B Instruct model of 22.7%. This underscores the efficiency of deriving strict constraints from exploration experience to enhance model capability. Furthermore, our method consistently demonstrates superior performance and substantial gains across varying model scales (3B and 7B) and data quantities (0.1k and 2k), validating the general effectiveness of PEEU in diverse settings.
Allrecipes Amazon Apple Arxiv Github Coursera Map Wolfram ID OOD OOD OOD OOD OOD OOD OOD
Overall Total
Model
Method
GPT-4o Claude 3 Opus Qwen2.5-VL-72B Qwen2.5-VL-32B
Vanilla (Hurst et al., 2024) Vanilla (Anthropic, 2024) Vanilla (Bai et al., 2025) Vanilla (Bai et al., 2025)
56.3 45.9 6.6 0.0
53.7 58.6 58.5 39.0
56.6 58.1 25.5 16.2
60.5 55.0 32.5 32.5
57.7 56.9 17.0 21.9
65.1 68.2 21.4 19.0
56.9 55.3 36.5 34.1
65.2 51.5 36.9 19.5
59.0 56.1 29.3 22.7
Qwen2.5-VL-3B 0.1k trajectories
Vanilla (Bai et al., 2025) Atomic-Prompt (Wang et al., 2024b) Trajectory-Prompt (Wang et al., 2024b) Coarse-SFT (Logeswaran et al., 2025) Coarse-GRPO (Logeswaran et al., 2025) Atomic-SFT (Fan et al., 2025) Atomic-GRPO (Fan et al., 2025) PEEU-SFT (Ours) PEEU-GRPO (Ours)
0.0 0.0 0.0 0.0 0.0 2.2 0.0 2.2 6.6
0.0 0.0 0.0 0.0 2.4 2.4 12.1 7.3 24.3
0.0 0.0 0.0 0.0 0.0 0.0 2.3 6.9 3.0
0.0 0.0 0.0 2.3 20.9 4.6 11.6 11.6 23.2
0.0 0.0 0.0 2.4 0.0 7.3 0.0 2.4 9.7
0.0 0.0 0.0 4.7 2.3 7.1 9.5 0.0 7.1
0.0 0.0 0.0 2.4 2.4 0.0 0.0 4.8 0.0
2.1 0.0 0.0 4.3 17.3 15.2 8.6 10.8 15.2
0.2 0.0 (-0.2%) 0.0 (-0.2%) 2.0 (+1.8%) 5.6 (+5.4%) 4.8 (+4.6%) 5.5 (+5.3%) 5.7 (+5.5%) 11.1 (+10.9%)
Qwen2.5-VL-7B 0.1k trajectories
Vanilla (Bai et al., 2025) Atomic-Prompt (Wang et al., 2024b) Trajectory-Prompt (Wang et al., 2024b) Coarse-SFT (Logeswaran et al., 2025) Coarse-GRPO (Logeswaran et al., 2025) Atomic-SFT (Fan et al., 2025) Atomic-GRPO (Fan et al., 2025) PEEU-SFT (Ours) PEEU-GRPO (Ours)
2.2 2.2 4.4 0.0 0.0 15.5 2.2 8.8 4.4
7.3 0.0 0.0 4.8 17.0 17.0 19.5 24.3 26.8
9.3 6.9 0.0 0.0 7.1 11.6 0.0 18.6 18.6
4.6 4.6 4.6 4.6 20.9 23.2 18.6 16.2 20.9
9.7 2.4 2.4 0.0 4.8 0.0 0.0 7.3 21.9
16.6 9.5 9.5 7.1 4.7 7.1 11.9 16.6 33.3
0.0 0.0 2.4 4.8 12.1 4.8 0.0 7.3 12.1
13.0 4.3 6.5 17.3 26.0 19.5 28.2 26.0 21.7
7.8 3.7 (-4.1%) 3.7 (-4.1%) 4.8 (-3.0%) 11.5 (+3.7%) 12.3 (+4.5%) 10.0 (+2.2%) 15.6 (+7.8%) 19.9 (+12.1%)
Qwen2.5-VL-3B 2k trajectories
Vanilla (Bai et al., 2025) Coarse-SFT (Logeswaran et al., 2025) Atomic-SFT (Fan et al., 2025) PEEU-SFT (Ours)
0.0 0.0 13.3 8.8
0.0 12.1 26.8 46.3
0.0 6.9 4.6 13.9
0.0 6.9 23.2 13.9
0.0 9.7 7.3 9.7
0.0 14.2 9.5 14.2
0.0 17.0 17.0 21.9
2.1 39.1 32.6 30.4
0.2 13.2 (+13.0%) 16.7 (+16.5%) 19.8 (+19.6%)
Qwen2.5-VL-7B 2k trajectories
Vanilla (Bai et al., 2025) Coarse-SFT (Logeswaran et al., 2025) Atomic-SFT (Fan et al., 2025) PEEU-SFT (Ours)
2.2 6.6 13.3 17.7
7.3 34.1 51.2 53.6
9.3 20.9 6.9 16.2
4.6 20.9 25.5 25.2
9.7 17.0 0.0 19.5
16.6 14.2 9.5 35.7
0.0 17.0 39.0 48.7
13.0 21.7 28.2 28.2
7.8 19.0 (+11.2%) 21.7 (+13.9%) 30.6 (+22.8%)
Table 1: Performance across different OOD websites. Bold indicates the highest performance. Underline indicates the second-highest performance. Overall is the average accuracy of all websites.
Using higher-level tasks provides better crosswebsite generalization than lower-level tasks in real-world websites. As illustrated in Table 1, relying on atomic-level tasks, limits crosswebsite generalization. Therefore, higher-level tasks like PEEU are essential for enhancing the generalization capability of task decomposition across different websites. For the Qwen2.5-VL-7B model trained with 2k trajectories, our PEEU-SFT achieves a overall accuracy of 30.6%, outperforming the Atomic-SFT baseline of 21.7%. For the Qwen2.5-VL-7B model trained with 0.1k trajectories, our PEEU-GRPO achieves a overall accuracy of 19.9%, outperforming the Atomic-GRPO baseline of 10.0%. Furthermore, this trend is consistent across different model sizes (e.g., Qwen2.5-VL-3B and Qwen2.5-VL-7B) and data regimes (e.g., 0.1k and 2k trajectories), showing that higher-level tasks provide a more generalization ability for planning in the unseen web environments.
Without a specially designed prompt pipeline, direct training is more effective than retrieval for small models. As shown in Table 1, we apply both training and retrieval under the same experiences. Because of the limited ability of small models, using prompts without changing model parameters does not effectively help them improve in complex tasks. For example, with the retrieval method, a 7B model gets scores of 3.7% for both Atomic-Prompt and Trajectory-Prompt, which are even lower than the base model score of 7.8% because their reasoning capabilities are too limited without any training. Similarly, the 3B model fails to effectively utilize retrieved context, resulting in 0.0% accuracy across prompt-based methods. In stark contrast, training methods yield substantial gains; specifically, PEEU-GRPO boosts the 7B and 3B models to 19.9% and 11.1% respectively. This shows direct training is more effective than retrieval for small models.
Task Decomposition Hierarchical Analysis Framework (TDHAF)
ID Trajectory:
Low-level Task
Type the destination “France” into the search box.
Mid-level Task
I'm traveling alone, departing on September 15 and arriving on the 25th. Please help me filter.
click [25] 14
5
19
click [14] type [5]; “France”
25 click [19]
High-level Task
OOD Trajectory:
High-level Task
3B 7B
OOD multi-level generalization
Carefully verify and output the arXiv number.
Coverage Percentage (%)
OOD Multi-level Generalization Coverage Percentage (%)
44.8 36.4 43.5
gap
22.7
gap 22.7
29.2 9.1
Output the current arxiv id of the model in this page.
Please search on Hugging Face for all models in the “image-text to text” category, and compile a complete list. Then review that list, prioritising models that have been updated this year or in the past few months… Tell me paper citations and arXiv IDs.
72.7
63.6
Test in Mid
Jump to the drop-down list.
51.9
Test in Low
Test in High
Please help me find a comprehensive list of all the image-text to text models that are available on Hugging Face.
89.6
80.5
Check the ratings and make a reservation.
Browse and check whether there are artworks in the room and reserve it.
Click on the model to enter the opensource model list.
ID Top-down Generalization
ID Bottom-up Generalization
Accuracy (%)
ID top-down generalization
Click to set the arrival date to September 25th.
I’m planning a solo trip, departing on September 15 and returning on September 25. Please help me find a hotel within this date range that has a rating above 9.5, ideally surrounded by lush greenery and featuring artworks in the room… Please book it for me.
ID bottom-up generalization Click to enter Low-level Task the Hugging Face homepage.
Mid-level Task
Click to set the departure date to September 15th.
16.2 18.8
Test in All Trained in low
Trained in middle
3.2
25.7 18.9
24.3
29.7
33.8
37.8
9.1
Trained in low Trained in middle Trained in high
Trained in low Trained in middle Trained in high
Figure 3: This figure illustrates the task decomposition hierarchical analysis framework. The upper part shows the trajectory of ID, and the lower part shows the trajectory of OOD. Both domains contain three levels: low, middle, and high. We study three generalization dimensions, including ID bottom-up generalization, ID top-down generalization and OOD multi-level generalization.
3
Task Decomposition Hierarchical Analysis Framework
Although PEEU achieves stronger performance, a critical research question remains: what is the agent’s capacity for compositional generalization across different levels of task decomposition? To answer this and analyze the hierarchical generalization capabilities of task decomposition, we propose the Task Decomposition Hierarchical Analysis Framework (TDHAF). Illustrated in Figure 3, this framework enables a rigorous evaluation from three perspectives: ID bottom-up generalization, ID top-down generalization, and OOD multi-level generalization (Appendix D). In this section, we introduce the analysis framework, data construction, experimental settings, results and analysis. 3.1
Analysis Framework
To investigate the compositional generalization ability of models in multimodal web navigation task planning scenarios, we propose the task decomposition hierarchical analysis framework. This framework first defines three levels of task granularity: low-level tasks, mid-level tasks, and high-level tasks. It further distinguishes between two types of generalization: in-domain (ID) and out-of-domain (OOD). Building on this taxonomy, the frame-
work analyzes from three perspectives: bottom-up generalization in-domain, top-down generalization in-domain, and multi-level generalization out-ofdomain. Figure 3 provides a detailed example of the analysis framework. Table 3 illustrates the training and testing set divisions for the three generalization dimensions. Explanations of the three dimensions of generalization are presented following. ID Bottom-up Generalization. To study whether the model can generalize from low-level tasks to higher-level composite tasks in-domain, we use relatively low-level tasks as the training set and high-level tasks as the test set. For example, after the model learns single-step atomic task mapping, we test if it can generalize to multi-step subtasks and long-horizon task decomposition. We test if it can generalize to long-horizon task decomposition after learning subtasks. ID Top-down Generalization. To study whether the model can generalize from high-level tasks to lower-level tasks in-domain, we use relatively highlevel tasks as the training set and relatively lowlevel tasks as the test set, which is the opposite of the previous experiment. For example, after the model learns to decompose long-horizon tasks, we check whether it truly learns the corresponding subtasks and atomic skills.
Table 2: Accuracy comparison across different generalization dimensions. 3B Instruct refers to the Qwen2.5-VL-3BInstruct model. 3B Low refers to the Qwen2.5-VL-3B-Instruct trained at the low level (atomic level). 3B High refers to the Qwen2.5-VL-3B-Instruct trained at the high level. Test-ID-Low denotes the in-domain low-level test set. Test-OOD-Low denotes the out-of-domain low-level test set. The bolded entries indicate the model that achieves the highest Step SR among the four models on each test set under the same base model. Test-ID-Low
Model
Test-ID-Middle
Test-ID-High
Id
Action
Value
Step SR
Id
Action
Value
Step SR
Id
Action
Value
Step SR
3B Instruct 3B Low 3B Middle 3B High
30.3 81.2 72.7 77.3
39.5 99.4 98.7 98.1
85.7 100.0 95.7 95.7
17.8 80.5 71.4 75.3
17.1 28.6 66.9 57.8
6.6 83.1 95.5 94.2
9.5 4.3 73.9 65.2
0.0 22.7 63.6 54.5
14.4 12.3 32.5 64.9
9.6 85.1 85.1 95.5
6.7 0.0 0.0 65.2
0.7 9.1 29.2 63.0
7B Instruct 7B Low 7B Middle 7B High
59.1 90.3 87.0 85.1
84.4 99.4 99.4 98.1
73.9 100.0 95.7 87.0
49.4 89.6 86.4 83.1
43.1 37.7 78.6 69.5
41.2 39.6 92.2 89.6
27.3 43.5 65.2 39.1
17.6 16.2 72.7 63.6
35.8 29.2 46.1 76.6
44.4 75.3 89.6 92.2
20.0 13.0 21.7 56.5
13.2 18.8 43.5 72.1
Test-OOD-Low
Model
Test-OOD-Middle
Test-OOD-High
Id
Action
Value
Step SR
Id
Action
Value
Step SR
Id
Action
Value
Step SR
3B Instruct 3B Low 3B Middle 3B High
40.5 81.1 70.3 82.4
63.5 98.6 100.0 100.0
100.0 100.0 100.0 100.0
31.1 79.7 70.3 82.4
21.9 37.8 48.6 45.9
20.5 75.7 79.7 81.1
33.3 12.5 12.5 12.5
6.8 35.1 44.6 44.6
16.4 29.7 32.4 39.2
16.4 78.4 78.4 81.1
22.2 0.0 0.0 6.2
0.0 25.7 31.1 39.2
7B Instruct 7B Low 7B Middle 7B High
63.5 89.2 83.8 81.1
91.9 97.3 100.0 95.9
62.5 93.8 93.8 75.0
56.8 85.1 82.4 77.0
46.6 56.8 59.5 58.1
72.6 78.4 82.4 82.4
20.0 31.2 12.5 12.5
30.1 50.0 51.4 54.1
30.1 37.8 37.8 45.9
64.4 79.7 78.4 81.1
20.0 18.8 0.0 6.2
16.4 33.8 35.1 43.2
OOD Multi-level Generalization. To study whether the model can generalize task decomposition ability from in-domain tasks to out-of-domain tasks, we separately use three levels of in-domain tasks as the training set. We use unseen crosswebsite tasks as the test set to evaluate multi-level out-of-domain generalization. For example, we examine how well it applies abilities to unseen tasks. 3.2
Experimental Settings
Settings. All experiments are conducted on Qwen2.5-VL-3B-Instruct and Qwen2.5-VL-7BInstruct for SFT. The batch size is 8, the learning rate is 5.0e-6 and the training epochs are 3, with llama-factory (Zheng et al., 2024b) framework. All experiments are conducted on 4 A800 GPUs. Metric. Following (Deng et al., 2023; Zheng et al., 2024a), we calculate the accuracy between predictions and ground truth, which includes the following four sub-metrics: Id refers to the accuracy of interactive element number in the Set-ofMark (SoM). Action measures the accuracy of action types. Value evaluates the accuracy of action parameters. Step SR represents the accuracy rate of a single-step prediction completely matching the ground truth.
3.3
Results and Analysis
Mastering individual low-level tasks does not necessarily imply mastery of the corresponding high-level task. As shown in Table 2 and Figure 3 in the Step SR in-domain setting, the 3Bmodel trained in low-level training data achieves 80.5% accuracy in low-level test tasks, but only 9.1% accuracy for the corresponding high-level test tasks. Similarly, the 7B-model trained on lowlevel data achieves 89.6% accuracy on low-level tasks, but only 18.8% on high-level ones. This shows that the bottom-up post-training method is not an effective way for enhancing planning ability. Using high-level tasks makes it easier to generalize downwards in-domain with greater overall coverage. As shown in Figure 4 and Figure 6 in the in-domain setting, we define a task where all levels succeed as good generalization, and we refer to this percentage as the coverage percentage (Appendix G for a formal definition). For the 3B model, the coverage percentage is 44.8% when trained on high-level tasks, 22.7% on middle-level, and 3.2% on low-level tasks. The 7B model achieves 51.9% (high-level), 36.4% (middle-level), and 9.1% (lowlevel) coverage. This shows top-down generalization has higher coverage percentage in-domain.
Trained in Low, ID Test 0.6% 3.2% 1.3% 16.9% 17.5% 4.5%
0.6%
Trained in Middle, ID Test 17.5% 1.3% 5.2% 4.5%
22.7%
16.9%
31.2%
55.2%
Trained in Low, OOD Test 18.9% 1.4% 40.5%
0.6%
14.3% 1.3% 5.2% 3.9%
21.6%
14.9%
2.7% 2.7%
5.4%
2.7%
14.9% 29.7%
1.4%
Generalization Types 9.1%4.5%
Trained in High, OOD Test 2.7%
24.3%
44.8%
16.9%
Trained in Middle, OOD Test
18.9%
Trained in High, ID Test
12.2%
2.7%
33.8%
40.5%
5.4%
Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Good Generalization Bad Generalization
2.7%
Figure 4: Generalization distribution pie chart for Qwen2.5-VL-3B. The table shows the distribution of eight types of generalization. Good generalization means successful generalization to other levels, the larger the better. The good generalization area expands from left to right, demonstrating high-level task training yields better generalization. Results for Qwen2.5-VL-7B. The definitions of good/bad generalization are shown in Appendix G.
Using high-level task training can enable the model to acquire stronger generalization capabilities for multi-level tasks in OOD. As shown in Figure 4 and Figure 6 in the out-of-domain setting, for the 3B model, the coverage percentage is 33.8% when trained on high-level tasks, 24.3% on middle-level, and 18.9% on low-level tasks. For the 7B model, the coverage percentage is 37.8% when trained on high-level tasks, 29.7% on middle-level, and 25.7% on low-level tasks. This shows that top-down generalization also has higher coverage percentage out-of-domain.
4
Related Work
DeepResearch Agent. DeepResearch emphasizes broad web searches (Zhang et al., 2025; Li et al., 2025c). Systems like WebSailor (Li et al., 2025a), WebShaper (Tao et al., 2025), and WebWatcher (Geng et al., 2025) focus on information seeking. But experience summarization and compositional generalization analysis (Li et al., 2025b) remain underexplored. AWM (Wang et al., 2024b), Agent KB (Tang et al., 2025b), Memento (Zhou et al., 2025a) and Memp (Fang et al., 2025) construct structured knowledge bases from past explorations using prompt engineering without training. To bridge this gap, we study compositional generalization in task planning and leverage automatically mined experiences to train agents, enabling them to achieve stronger web-based planning capabilities under the same scale of data.
Multimodal Web Navigation Agent. The research on multimodal web agent navigation emphasizes vertical depth navigation on web pages (Wang et al., 2024a; Ning et al., 2025; Zhou et al., 2025b; Tang et al., 2025a; Li et al., 2026). Open-source models need two core abilities: grounding and planning (Wang et al., 2024a; Men et al., 2024; Nguyen et al., 2025). Some works strengthen grounding for more accurate spatial coordinates (Lu et al., 2025; Luo et al., 2025; Zhou et al., 2025c). The SoM representation can reduce the influence of grounding, making it easier to study improvements in planning ability. Prior work often trains on low-level tasks (Gu et al., 2024; Fan et al., 2025) or distills teacher trajectories without fully utilizing experiences (Logeswaran et al., 2025; Trabucco et al., 2025). Some works equip agents with memory to enhance their planning capabilities (Hu et al., 2025; Wang et al., 2024b; Men et al., 2025a; Xia et al., 2026). Additionally, some studies aim to improve trajectory quality. One group of methods uses a reward model to filter trajectories (Men et al., 2025b; Lin et al., 2025; Jin et al., 2025), and another gives the model the ability to adapt to its environment with the trajectories (Su et al., 2025; Zhou et al., 2025b; Sun et al., 2025). Our approach makes high-level tasks more aligned and constrained, and by leveraging the TDHAF framework to quantitatively analyze this capacity from the perspective of planning granularity, thereby providing stronger generalization ability in the same data scale setting.
5
Conclusion
In this work, we propose the Planning Experience Exploration and Utilization (PEEU) method to enhance small MLLMs by leveraging autonomous exploration and hindsight experience. To analyze this, we introduce the Task Decomposition Hierarchical Analysis Framework (TDHAF) to systematically evaluate planning compositional generalization. Experiments show PEEU significantly outperforms larger models on OOD websites, demonstrating training on aligned high-level tasks is effective for planning ability generalization.
Limitations Currently, our evaluation focuses on informationseeking and navigation tasks across diverse realworld websites. Due to privacy and security constraints, we did not include scenarios involving sensitive operations such as user login, CAPTCHA solving, or actual payment transactions. While the proposed high-level task planning is theoretically applicable to these scenarios, extending the agent’s capabilities to handle authenticated sessions and security protocols remains a direction for future research.
Acknowledgements This work was supported by the National Natural Science Foundation of China (No.U24A20335, No.62406321), Beijing Natural Science Foundation (L243006), and the independent research project of the Key Laboratory of Cognition and Decision Intelligence for Complex Systems.
References John R Anderson. 2013. The architecture of cognition. Psychology Press. Anthropic. 2024. Introducing the next generation of claude. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov. 2025. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153. Yuxuan Cai, Yipeng Hao, Jie Zhou, Hang Yan, Zhikai Lei, Rui Zhen, Zhenhua Han, Yutao Yang, Junsong
Li, Qianjun Pan, and 1 others. 2025. Building selfevolving agents via experience-driven lifelong learning: A framework and benchmark. arXiv preprint arXiv:2508.19005. Pengfei Cao, Tianyi Men, Wencan Liu, Jingwen Zhang, Xuzhao Li, Xixun Lin, Dianbo Sui, Yanan Cao, Kang Liu, and Jun Zhao. 2025. Large language models for planning: A comprehensive and systematic survey. arXiv preprint arXiv:2505.19683. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. 2023. Mind2web: Towards a generalist agent for the web. In Thirty-seventh Conference on Neural Information Processing Systems. Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, and Gang Wu. 2025. Guibee: Align gui action grounding to novel environments via autonomous exploration. arXiv preprint arXiv:2501.13896. Runnan Fang, Yuan Liang, Xiaobin Wang, Jialong Wu, Shuofei Qiao, Pengjun Xie, Fei Huang, Huajun Chen, and Ningyu Zhang. 2025. Memp: Exploring agent procedural memory. arXiv preprint arXiv:2508.06433. Xinyu Geng, Peng Xia, Zhen Zhang, Xinyu Wang, Qiuchen Wang, Ruixue Ding, Chenxi Wang, Jialong Wu, Yida Zhao, Kuan Li, and 1 others. 2025. Webwatcher: Breaking new frontiers of visionlanguage deep research agent. arXiv preprint arXiv:2508.05748. Yu Gu, Kai Zhang, Yuting Ning, Boyuan Zheng, Boyu Gou, Tianci Xue, Cheng Chang, Sanjari Srivastava, Yanan Xie, Peng Qi, and 1 others. 2024. Is your llm secretly a world model of the internet? modelbased planning for web agents. arXiv preprint arXiv:2411.06559. Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. 2024. Webvoyager: Building an end-toend web agent with large multimodal models. arXiv preprint arXiv:2401.13919. Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang, Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin Guo, Shihan Dou, Zhiheng Xi, and 1 others. 2025. Memory in the age of ai agents. arXiv preprint arXiv:2512.13564. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Jiexin Xu, Huaijun Li, Xiaojian Jiang, Kang Liu, and Jun Zhao. 2025. Rag-rewardbench: Benchmarking reward models in retrieval augmented generation for preference alignment. In Findings of
the Association for Computational Linguistics: ACL 2025, pages 17061–17090. Jiachun Li, Zhuoran Jin, Tianyi Men, Yupu Hao, Kejian Zhu, Lingshuai Wang, Dongqi Huang, Longxiang Wang, Shengjia Hua, Lu Wang, and 1 others. 2026. Agentic environment engineering for large language models: A survey of environment modeling, synthesis, evaluation, and application. arXiv preprint arXiv:2606.12191. Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, and 1 others. 2025a. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592. Tianle Li, Jihai Zhang, Yongming Rao, and Yu Cheng. 2025b. Unveiling the compositional ability gap in vision-language reasoning model. arXiv preprint arXiv:2505.19406. Yangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, and 1 others. 2025c. Towards agentic rag with deep reasoning: A survey of rag-reasoning systems in llms. arXiv preprint arXiv:2507.09477. Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, and 1 others. 2025d. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921. Haojia Lin, Xiaoyu Tan, Yulei Qin, Zihan Xu, Yuchen Shi, Zongyi Li, Gang Li, Shaofei Cai, Siqi Cai, Chaoyou Fu, and 1 others. 2025. Cuarewardbench: A benchmark for evaluating reward models on computer-using agent. arXiv preprint arXiv:2510.18596. Lajanugen Logeswaran, Jaekyeom Kim, Sungryull Sohn, Creighton Glasscock, and Honglak Lee. 2025. Scaling web agent training through automatic data generation and fine-grained evaluation. In Second Conference on Language Modeling. Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, and Hongsheng Li. 2025. Ui-r1: Enhancing efficient action prediction of gui agents by reinforcement learning. arXiv preprint arXiv:2503.21620. Run Luo, Lu Wang, Wanwei He, and Xiaobo Xia. 2025. Gui-r1: A generalist r1-style vision-language action model for gui agents. arXiv preprint arXiv:2504.10458. Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Unlocking the future: Exploring look-ahead planning mechanistic interpretability in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7713–7724.
Tianyi Men, Pengfei Cao, Zhuoran Jin, Yubo Chen, Kang Liu, and Jun Zhao. 2025a. A troublemaker with contagious jailbreak makes chaos in honest towns. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17561–17587. Tianyi Men, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2025b. Agent-rewardbench: Towards a unified benchmark for reward modeling across perception, planning, and safety in real-world multimodal agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 17521– 17541. Dang Nguyen, Jian Chen, Yu Wang, Gang Wu, Namyong Park, Zhengmian Hu, Hanjia Lyu, Junda Wu, Ryan Aponte, Yu Xia, and 1 others. 2025. Gui agents: A survey. In Findings of the Association for Computational Linguistics: ACL 2025, pages 22522–22538. Liangbo Ning, Ziran Liang, Zhuohang Jiang, Haohao Qu, Yujuan Ding, Wenqi Fan, Xiao-yong Wei, Shanru Lin, Hui Liu, Philip S Yu, and 1 others. 2025. A survey of webagents: Towards next-generation ai agents for web automation with large foundation models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6140–6150. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. Brian H Ross. 1989. Some psychological results on case-based reasoning. In Proceedings: Case-based reasoning workshop, pages 144–147. Morgan Kaufmann. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. David Silver and Richard S Sutton. 2025. Welcome to the era of experience. Google AI, 1. Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan Ö Arık. 2025. Learn-byinteract: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893. Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, and 1 others. 2025. Os-genesis: Automating gui agent trajectory construction via reverse task synthesis. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5555–5579.
Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, and 1 others. 2025a. A survey on (m) llm-based gui agents. arXiv preprint arXiv:2504.13865.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. 2024b. Llamafactory: Unified efficient finetuning of 100+ language models. arXiv preprint arXiv:2403.13372.
Xiangru Tang, Tianrui Qin, Tianhao Peng, Ziyang Zhou, Daniel Shao, Tingting Du, Xinming Wei, Peng Xia, Fang Wu, He Zhu, and 1 others. 2025b. Agent kb: Leveraging cross-domain experience for agentic problem solving. arXiv preprint arXiv:2507.06229.
Huichi Zhou, Yihang Chen, Siyuan Guo, Xue Yan, Kin Hei Lee, Zihan Wang, Ka Yiu Lee, Guchun Zhang, Kun Shao, Linyi Yang, and 1 others. 2025a. Memento: Fine-tuning llm agents without fine-tuning llms. Preprint.
Zhengwei Tao, Jialong Wu, Wenbiao Yin, Junkai Zhang, Baixuan Li, Haiyang Shen, Kuan Li, Liwen Zhang, Xinyu Wang, Yong Jiang, and 1 others. 2025. Webshaper: Agentically data synthesizing via information-seeking formalization. arXiv preprint arXiv:2507.15061.
Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, and Li Erran Li. 2025b. Proposer-agent-evaluator (pae): Autonomous skill discovery for foundation model internet agents. In Forty-second International Conference on Machine Learning.
Brandon Trabucco, Gunnar Sigurdsson, Robinson Piramuthu, and Ruslan Salakhutdinov. 2025. Insta: Towards internet-scale training for agents. arXiv preprint arXiv:2502.06776.
Yuqi Zhou, Sunhao Dai, Shuai Wang, Kaiwen Zhou, Qinglin Jia, and Jun Xu. 2025c. Gui-g1: Understanding r1-zero-like training for visual grounding in gui agents, 2025. URL https://arxiv. org/abs/2505.15810, 3:36–37.
Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, and 1 others. 2024a. Gui agents with foundation models: A comprehensive survey. arXiv preprint arXiv:2411.04890. Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. 2024b. Agent workflow memory. arXiv preprint arXiv:2409.07429. Hui Wei, Zihao Zhang, Shenghua He, Tian Xia, Shijia Pan, and Fei Liu. 2025. Plangenllms: A modern survey of llm planning capabilities. arXiv preprint arXiv:2502.11221. Peng Xia, Jianwen Chen, Hanyang Wang, Jiaqi Liu, Kaide Zeng, Yu Wang, Siwei Han, Yiyang Zhou, Xujiang Zhao, Haifeng Chen, and 1 others. 2026. Skillrl: Evolving agents via recursive skillaugmented reinforcement learning. arXiv preprint arXiv:2602.08234. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR). Weizhi Zhang, Yangning Li, Yuanchen Bei, Junyu Luo, Guancheng Wan, Liangwei Yang, Chenxuan Xie, Yuyao Yang, Wei-Chieh Huang, Chunyu Miao, and 1 others. 2025. From web search towards agentic deep research: Incentivizing search with reasoning agents. arXiv preprint arXiv:2506.18959. Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, and Yu Su. 2024a. Gpt-4v(ision) is a generalist web agent, if grounded. Yaowei Zheng, Junting Lu, Shenzhi Wang, Zhangchi Feng, Dongdong Kuang, and Yuwen Xiong. 2025. Easyr1: An efficient, scalable, multi-modality rl training framework.
A
PEEU Prompt
Inference Prompt for WebVoyager System: Imagine you are a robot browsing the web, just like humans. Now you need to complete a task. In each iteration, you will receive an Observation that includes a screenshot of a webpage and some texts. This screenshot will feature Numerical Labels placed in the TOP LEFT corner of each Web Element. Carefully analyze the visual information to identify the Numerical Label corresponding to the Web Element that requires interaction, then follow the guidelines and choose one of the following actions: 1. Click a Web Element. 2. Delete existing content in a textbox and then type content. 3. Scroll up or down. Multiple scrolls are allowed to browse the webpage. Pay attention!! The default scroll is the whole window. If the scroll widget is located in a certain area of the webpage, then you have to specify a Web Element in that area. I would hover the mouse there and then scroll. 4. Wait. Typically used to wait for unfinished webpage processes, with a duration of 5 seconds. 5. Go back, returning to the previous webpage. 6. Google, directly jump to the Google search page. When you can’t find information in some websites, try starting over with Google. 7. Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly, Action should STRICTLY follow the format: - Click [Numerical Label] - Type [Numerical Label]; [Content] - Scroll [Numerical Label or WINDOW]; [up or down] - Wait GoBack - Google - ANSWER; [content] Key Guidelines You MUST follow: * Action guidelines * 1) To input text, NO need to click textbox first, directly type content. After typing, the system automatically hits ‘ENTER‘ key. Sometimes you should click the search button to apply search filters. Try to use simple language when searching. 2) You must Distinguish between textbox and search button, don’t type content into the button! If no textbox is found, you may need to click the search button first before the textbox is displayed. 3) Execute only one action per iteration. 4) STRICTLY Avoid repeating the same action if the webpage remains unchanged. You may have selected the wrong web element or numerical label. Continuous use of the Wait is also NOT allowed. 5) When a com-
plex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER. * Web Browsing Guidelines * 1) Don’t interact with useless web elements like Login, Sign-in, donation that appear in Webpages. Pay attention to Key Web Elements like search textbox and menu. 2) Vsit video websites like YouTube is allowed BUT you can’t play videos. Clicking to download PDF is allowed and will be analyzed by the Assistant API. 3) Focus on the numerical labels in the TOP LEFT corner of each rectangle (element). Ensure you don’t mix them up with other numbers (e.g. Calendar) on the page. 4) Focus on the date in task, you must look for results that match the date. It may be necessary to find the correct year, month and day at calendar. 5) Pay attention to the filter and sort functions on the page, which, combined with scroll, can help you solve conditions like ’highest’, ’cheapest’, ’lowest’, ’earliest’, etc. Try your best to find the answer that best fits the task. For example: Click [3] Type [3]; [apple] Scroll [WINDOW]; [down] Wait GoBack Google ANSWER; [apple is red] Your reply should strictly follow the format: Thought: Your brief thoughts (briefly summarize the info that will help ANSWER) Action: One Action format you choose Then the User will provide: Observation: A labeled screenshot Given by User User: <image>Now given a task: <task> Please interact with https://www.example.com and get the answer. Observation: please analyze the attached screenshot and give the Thought and Action. I’ve provided the tag name of each element and the text it contains (if text exists). Note that <textarea> or <input> may be textbox, but not exactly. Please focus more on the screenshot and then refer to the textual information. <SoM Observation> Task Setting Prompt <image> Analyze the given webpage screenshot and generate 50 different tasks that users might want to accomplish on this website. You can focus on searching for specific items. The task should be combined with the specific function
of this website. The tasks should be varied, and there should be both difficult and simple tasks. Output only a JSON-formatted list of tasks with no additional commentary or explanation. Example format: "tasks": [ "task 1 description", "task 2 description", ... "task n description" ] Exploration Prompt Imagine you are a robot browsing the web, just like humans. Now you need to complete a task. In each iteration, you will receive an Observation that includes a screenshot of a webpage and some texts. This screenshot will feature Numerical Labels placed in the TOP LEFT corner of each Web Element. Carefully analyze the visual information to identify the Numerical Label corresponding to the Web Element that requires interaction, then follow the guidelines and choose one of the following actions: 1. Click a Web Element. 2. Delete existing content in a textbox and then type content. 3. Scroll up or down. Multiple scrolls are allowed to browse the webpage. Pay attention!! The default scroll is the whole window. If the scroll widget is located in a certain area of the webpage, then you have to specify a Web Element in that area. I would hover the mouse there and then scroll. 4. Wait. Typically used to wait for unfinished webpage processes, with a duration of 5 seconds. 5. Go back, returning to the previous webpage. If you scroll down more than twice and still can’t find the answer, you need to use "Go back" to return. 6. Google, directly jump to the Google search page. When you can’t find information in some websites, try starting over with Google. 7. Answer. This action should only be chosen when all questions in the task have been solved. Correspondingly, Action should STRICTLY follow the format: - Click [Numerical Label] - Type [Numerical Label]; [Content] - Scroll [Numerical Label or WINDOW]; [up or down] - Wait GoBack - Google - ANSWER; [content] Key Guidelines You MUST follow: * Action guidelines * 1) To input text, NO need to click textbox first, directly type content. After typing, the system automatically hits ‘ENTER‘ key. Sometimes you should click the search button to apply search filters. Try to use simple language when searching. 2) You must Distinguish between textbox and search button, don’t type content into the button! If no textbox is found,
you may need to click the search button first before the textbox is displayed. 3) Execute only one action per iteration. 4) STRICTLY Avoid repeating the same action if the webpage remains unchanged. You may have selected the wrong web element or numerical label. Continuous use of the Wait is also NOT allowed. 5) When a complex Task involves multiple questions or steps, select "ANSWER" only at the very end, after addressing all of these questions (steps). Flexibly combine your own abilities with the information in the web page. Double check the formatting requirements in the task when ANSWER. 6) If you feel the current product does not meet the task requirements, you can use GoBack action to return to the previous screen and look for other products. Don’t just scroll down-learn to go back. * Web Browsing Guidelines * 1) Don’t interact with useless web elements like Login, Sign-in, donation that appear in Webpages. Pay attention to Key Web Elements like search textbox and menu. 2) Vsit video websites like YouTube is allowed BUT you can’t play videos. Clicking to download PDF is allowed and will be analyzed by the Assistant API. 3) Focus on the numerical labels in the TOP LEFT corner of each rectangle (element). Ensure you don’t mix them up with other numbers (e.g. Calendar) on the page. 4) Focus on the date in task, you must look for results that match the date. It may be necessary to find the correct year, month and day at calendar. 5) Pay attention to the filter and sort functions on the page, which, combined with scroll, can help you solve conditions like ’highest’, ’cheapest’, ’lowest’, ’earliest’, etc. Try your best to find the answer that best fits the task. Your reply should strictly follow the format: Thought: Your brief thoughts (briefly summarize the info that will help ANSWER) Action: One Action format you choose Then the User will provide: Observation: A labeled screenshot Given by User Experience Extraction Prompt Analyze the user’s intent based on the following: The action performed between these interfaces is <ACTION> Task: The first screenshot shows the interface before interaction, while the second screenshot displays the interface after the click operation. Generate descriptions explaining the purpose of
interaction with the element. Focus on meaningful UI changes (e.g., new elements, transitions, or data updates, Don’t pay attention to the changes in the bbox.). Only output the task descriptions experience. Experience Aggregation Prompt In this task, there are too many details provided. I only want to keep the details specified by the user, and the specific operational details need to be deleted. Please directly output the processed string. The task requirement is a declarative sentence, appearing like a real world user task. The raw task is as follows:<low-level task list>
B
PEEU Algorithm Details
The PEEU algorithm is shown in Algorithm 1. Algorithm 1 Autonomous Planning with Exploration and Experience Utilization Require: Website URL, MLLM M , Environment Env Ensure: Policy π for task-oriented planning Stage 1: Planning Tree Exploration 1: Obtain homepage state s0 from the given URL 2: Generate task list: D = M (s0 , URL) 3: for each task di ∈ D do 4: Execute actions at guided by M 5: Transition: st+1 ∼ P (·|st , at ) 6: Record trajectory τ = (s0 , a0 , s1 , a1 , . . . ) 7: end for 8: Build exploration tree R = Explore(M, T , Env, URL) Stage 2: Planning Experience Utilization 9: for each trajectory τ do 10: Extract atomic experiences ϵt = (st , at , st+1 ) 11: Build µ = (ϵ0 , ϵ1 , . . . , ϵT ) 12: Fuse into PEEU task: d˜ = Φ(µ) 13: end for 14: Train policy π with SFT and GRPO using PEEU dataset 15: return trained policy π For RL training, we set two types of rewards. The first reward is for format, and the second reward is for answer correctness. For the format reward, we align with the action space and action format from WebVoyager. Each reward is 1.0, and if both are correct, the total reward is 2.0.
( 1.0, rformat = 0.0,
if the action follows formats otherwise, (6)
( 1.0, ranswer = 0.0,
if the predicted answer is correct otherwise, (7)
Rrl = rformat + ranswer .
C
(8)
PEEU Experiment Details
This section presents the implementation details of our experiments, including the data processing pipeline on the WebVoyager benchmark and the specific settings for In-Domain (ID) and Out-OfDomain (OOD) evaluations for PEEU. (1) We evaluate the planning capabilities of our models using the WebVoyager benchmark (He et al., 2024), which comprises real-world multimodal tasks across diverse categories such as shopping, research, coding, and travel. To ensure a stable evaluation environment, we exclude websites with strict access frequency limits (e.g., Cambridge Dictionary, Google Search, and Hugging Face). Consequently, our study focuses on the remaining accessible websites, which fully comply with terms of service (He et al., 2024). (2) To rigorously assess cross-site generalization, we structure our dataset into distinct In-Domain (ID) and Out-Of-Domain (OOD) partitions. Allrecipes is utilized as the source for ID exploration and training. Specifically, we curate two datasets for the algorithm: ID Set ( 0.1k tasks): Consists of approximately 100 tasks derived exclusively from Allrecipes. Supplementary OOD Set ( 2k tasks): Consists of approximately 2,000 tasks collected from additional websites. Crucially, the websites used for the Supplementary OOD Set are distinct from the 7 held-out websites reserved strictly for testing. This setup allows us to train/explore on one specific site (and optionally augment with the 2k OOD pool) while testing on 7 completely unseen websites to evaluate zero-shot generalization. We filter out data with incorrect formats prior to usage. Following WebVoyager (He et al., 2024) standard setting, for the experimental hyperparameters, the maximum exploration depth is set to 15 steps. The retrieval module employs all-roberta-
large-v1 (Reimers and Gurevych, 2020) for semantic matching.
D
Definition Details
In this section, we introduce and formalize the definitions of task planning, and then present the three levels of task planning granularity in this work, including low-level tasks, mid-level tasks, and highlevel tasks. As well as the definitions of in-domain, out-of-domain and experience. Task Planning Definition. The task planning is formally defined as a tuple (Li et al., 2025d; Cao et al., 2025; Wei et al., 2025): P = ⟨S, A, T, s0 , G⟩.
(9)
Here, S is a set of environment states, A is a set of actions, T : S × A → S is a state transition function, s0 ∈ S is an initial state, G ⊆ S is a set of goal states. The objective is to find a sequence of actions ⟨a0 , a1 , . . . , an ⟩ that transforms the system from the initial state s0 to a goal state sg ∈ G. In the ReAct paradigm (Yao et al., 2023), the objective is to output the next action given the task description, history, and current observation. This can be formally represented as: at = π(d, H0:t , st ).
(10)
Here, d is the task description, and H0:t = {(s0 , a0 ), (s1 , a1 ), . . . , (st−1 , at−1 )} is the history of state-action pairs up to time t−1, st is the current observation, and π is the planning policy that outputs the action at . Upon task completion, we obtain a trajectory τ = {(s0 , a0 ), (s1 , a1 ), . . . , (sn , an )}. Low-level Task Definition. The low-level task is defined as a single-step task. It is also called the atomic-level task. For step t, the policy π uses only the current low-level task description dlow and the current observation st to determine the next action: at = π(dlow , st ).
(11)
Mid-level Task Definition. The mid-level task is defined as a multi-step subtask. For a subtask spanning steps p to q, the policy π uses the middlelevel task description dmid , the history Hp:t and the current observation st to determine the next action: at = π(dmid , Hp:t , st ).
(12)
High-level Task Definition. The high-level task is defined as a long horizon, composed of a sequence of subtasks. For a long horizon task 0 to n, the policy π uses the high-level task description dhigh , the history H0:t and the current observation st to determine the next action: at = π(dhigh , H0:t , st ).
(13)
In-Domain and Out-of-Domain. For the TDHAF, ID evaluation uses test data from the same trajectories seen during post-training. The task description has been paraphrased, while OOD evaluation uses test data from entirely new websites not encountered during post-training. For the PEEU, ID evaluation uses test data from the same websites seen during post-training, while OOD evaluation uses test data from entirely new websites not encountered during post-training. Experience Definition. As defined in Silver and Sutton (2025), experience is defined as data produced through an agent’s interactions with the environment. Subsequent work (Cai et al., 2025) further categorizes experiences into trajectories, knowledge and skills summarized from these trajectories. In this paper, we mainly refer to what is summarized from the trajectory as experience.
E
TDHAF Prompt
Build Low Level Prompt for TDHAF Your task is to generate task descriptions for CLICK/TYPE/SELECT an on-screen element. Two screenshots are provided: Current UI - Shows a interactive element (labeled "1") with a bounding box. Post-interaction UI - Highlights changes after interaction (excluding bounding box disappearance). Task: Purpose Clarity - Clearly define the purpose of the interaction with the UI element in both descriptions, ensuring they are functionally identical but phrased differently. Ensure the two descriptions serve distinct contexts with no overlapping phrasing. Action Consistency - Use only CLICK, TYPE, or SELECT as action types, with identical parameters in both descriptions (e.g., target element, input text, or selection option). UI Change Focus - Describe only observable UI changes (e.g., new elements appearing, data updates, transitions) resulting from the action-avoid
Inference Prompt for Multimodal-Mind2web for Agent User: <image>You are a web agent. Your task is: <task> The history is: <history>. If you want to complete the task, you should output action CLICK/TYPE/SELECT, id and value in <answer> </answer> tags. Output the one bbox you should interact with in JSON format. Examples: 1. For clicking: <answer>"action": "CLICK", "value": "" ,"id": 3</answer> 2. For typing text: <answer>"action": "TYPE","value": "[email protected]", "id": 5</answer> 3. For selecting an option: <answer>"action": "SELECT", "value": "United States","id": 2</answer>
F
TDHAF Data Construction
Raw data is collected from MultimodalMind2Web (Deng et al., 2023; Zheng et al., 2024a). It is an offline human-expert-annotated
Distribution of Steps 120 100 80 60 40 20 00
Distribution of Tokens
250
Low ID Middle ID High ID Low OOD Middle OOD High OOD
200
History Token
Build High Level Prompt for TDHAF Please make this task more complex, but do not change the parameters in this task. Add more subtasks after this task, and rephrase the original task with synonymous expressions. This task and subsequent tasks can be combined into a more complex task. More complex means that the current task is a subtask in the middle, and then more subtasks are added before and after to merge into a more complex task. But don’t describe the specific tasks in detail. Please output two task descriptions that are paraphrases of each other, in the form of a list of json. The key of the element is the string task, and the value is the task description. The raw task is <task>.
gold trajectory dataset. Employing such a dataset for analysis offers more significant advantages, as it enables fine-grained examination of the model’s behavior at the single-step level, including the target numbers, action types, action parameters. The in-domain test and train data come from the same trajectory, while the out-of-domain test data come from different trajectories of completely different websites. The in-domain training and test data are derived from the same trajectories, but the questions are rewritten. The training set has 616 samples, and the test set has 684 samples. The data statistics are shown in Figure 5. The data split is shown in Table 3. The prompts for generating data are shown in Appendix E, which are the prompts for generating low-level tasks and high-level tasks by GPT-4o.
150
Data size
vague or future-oriented statements. Training vs. Testing Wording - Paraphrase the purpose distinctly for training (instructional) and testing (validation) contexts while keeping functional outcomes identical. Now, generate the two mission-style descriptions adhering to these rules. Only output the lists, nothing else. The raw task is <task>.
100
5
10 15 Planning Steps
20
50 0
0
50
100 150 Task Token
200
Figure 5: Data Distribution for TDHAF.
Table 3: This table shows the TDHAF division of training and test sets for three generalization dimensions. ID indicates that training and test are derived from the same trajectory in the same websites, but the tasks are rewritten. OOD indicates they come from different trajectories across different websites. L denotes low-level tasks, M denotes mid-level tasks, H denotes high-level tasks. Training Set
Test Set
ID Bottom-up Generalization Train-ID-L Train-ID-M Train-ID-H
Test-ID-L, Test-ID-M, Test-ID-H Test-ID-M, Test-ID-H Test-ID-H ID Top-down Generalization
Train-ID-L Train-ID-M Train-ID-H
Test-ID-L Test-ID-L, Test-ID-M Test-ID-L, Test-ID-M, Test-ID-H
OOD Multi-level Generalization Train-ID-L Train-ID-M Train-ID-H
Test-OOD-L, Test-OOD-M, Test-OOD-H Test-OOD-L, Test-OOD-M, Test-OOD-H Test-OOD-L, Test-OOD-M, Test-OOD-H
G
Generalization Distribution and Definition
H
The use of ChatGPT is only limited to grammar checking and linguistic refinement.
Let the set of levels be L = {low, middle, high}.
(14)
For a sample x at level ℓ ∈ L, define an indicator ( 1, if the prediction at level ℓ is correct, I(ℓ, x) = 0, otherwise. (15) Good Generalization. The model is considered to generalize well at some level (low, middle, or high) if it predicts correctly not only at this level but also at the other two levels. That means correct at all three levels. Good generalization means successful generalization to other levels, the larger the better. Good(ℓ, x) = 1 if and only if
′
I(ℓ , x) = 1 ∀ℓ′ ∈ L.
(16)
Bad Generalization. The model is considered to generalize bad at some level if it is correct at this level, but at least one of the other two levels is wrong. Bad generalization means failure to fully generalize to other levels, the smaller the better. Bad(ℓ, x) = 1 ′
if and only if
I(ℓ, x) = 1
′
and ∃ℓ ∈ L, ℓ ̸= ℓ with I(ℓ′ , x) = 0. (17) Coverage Percentage. Among all samples that are predicted correctly at their own level, and these samples that are also correct at all three levels (i.e., that achieve good generalization) is called the coverage percentage. Formally, let Gℓ = {x ∈ Sℓ | Good(ℓ, x) = 1}
(18)
be the set of samples that are correctly predicted at level ℓ and also satisfy the good generalization condition. Here, Sℓ denotes the set of all samples that are predicted correctly at level ℓ, and T denotes the entire test set. The coverage percentage at level ℓ is then defined as Coverage(ℓ) =
|Gℓ | × 100%. |T |
Usage of Chatgpt
(19)
I
Training Reward Details
Trained in Low, ID Test 0.6%
Trained in High, ID Test 1.9% 10.4% 4.5% 10.4%
2.6%
8.4% 9.1% 5.8% 7.8%
1.3%
Trained in Middle, ID Test 2.6%
8.4% 36.4%
14.3% 4.5%
66.9%
1.4% 2.7% 9.5% 1.4%
25.7%
Trained in Middle, OOD Test 4.1% 1.4% 10.8% 1.4%
Generalization Types
13.6% 7.1%
31.2%
Trained in Low, OOD Test
51.9%
29.7%
Trained in High, OOD Test 1.4%
17.6% 37.8%
4.1% 33.8%
20.3% 5.4%
33.8%
18.9%
27.0%
Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Low Mid High Good Generalization Bad Generalization
10.8% 1.4%
Figure 6: Generalization Distribution Pie Chart for Qwen2.5-VL-7B. Good generalization means successful generalization to other levels, and the larger it is, the better. Bad generalization means failure to fully generalize to other levels, and the smaller it is, the better. The good generalization area expands from left to right, demonstrating high-level task training yields better generalization. Results for Qwen2.5-VL-7B.
55
200
50
150
50
100
150
200
Steps 3B coarse - Reward
0
Reward
2.0
50
100
150
55 50
40
200
Steps 3B Atomic - Reward
Reward
2.0
1.4
1.6 1.4
0
90
50
100
150
Steps 7B coarse - Response Length
1.0
200
0
55.0
50
100
150
1.0
200
Steps 7B Atomic - Response Length Response Length
47.5
65
Response Length 0
50
100
150
Steps 7B coarse - Reward
45.0
55
40.0
50
37.5 200
Reward
1.8 1.6 1.4 1.2 1.0
50
100
Steps
150
200
50
100
150
Steps 7B Atomic - Reward
200
0
2.0
2.0
1.8
1.8
1.6 1.4 1.2
0
1.0
50
100
150
200
50
100
150
200
Steps 7B PEEU - Response Length Response Length
45 0
Reward
2.0
0
60
42.5
Reward
50
Length
70
Length
50.0
75
55
Reward
75
80
60
200
1.4
52.5
65
150
1.6
85
70
100
Steps 3B PEEU - Reward
1.2
1.2
1.2
50
1.8
Reward
Reward
1.6
Response Length 0
2.0
1.8
1.8
Reward
45
35
Response Length 0
Length
60
45
50
Reward
3B PEEU - Response Length
40
100
1.0
3B Atomic - Response Length Response Length
Length
250
Length
Length
3B coarse - Response Length
50
Reward
1.6 1.4 1.2
Reward 0
Steps 7B PEEU - Reward
100
Steps
150
Figure 7: RL Training Reward.
200
1.0
0
50
100
Steps
150
200