Clearing the Fog: Towards Installing and Refining Proactive Exploration Capabilities in LLM Agents

Clearing the Fog: Towards Installing and Refining Proactive Exploration Capabilities in LLM Agents Chen Huang2 * Ziming Liu1 Hongru Liang1,4 Wenqiang Lei1,4 See-Kiong Ng2 Tat-Seng Chua2 Anthony G Cohn3 1 Sichuan University 2 National University of Singapore 3 University of Leeds 4 Engineering Research Center of Machine Learning and Industry Intelligence guanzhizhao@stu.scu.edu.cn, huang_chen@nus.edu.sg

Zhizhao Guan1,4

arXiv:2608.14339v1 [cs.AI] 14 Aug 2026

Abstract We study proactive exploration in LLM agents, i.e., the ability to explore an environment to acquire information that improves future decisionmaking. In this regard, we first identify two fundamental bottlenecks that hinder this capability and then propose S A FA R I, a novel method designed to instill and refine proactive exploration. Specifically, S A FA R I consists of two components: (1) Exploratory Data Construction, which synthesizes exploration-rich trajectories to mitigate the hindsight bias of standard demonstrations; and (2) RL Optimization with Contrastive Signal Guidance, which leverages contrastive trajectory pairs to distinguish productive exploration from redundant wandering. Extensive experiments demonstrate the effectiveness of S A FA R I and provide insights into the characteristics of proactive exploration. Our code is available at: https: //github.com/GuanZhizhao/SAFARI.

1

Figure 1: Proactive exploration in online shopping. Agent proactively executes strategic backtracking and pagination to locate the product with better price.

As Large Language Models (LLMs) continue to advance, LLM-based agents have excelled in multi-turn interactive tasks (Yao et al., 2022; Zhou et al., 2023b). The prevailing paradigm for developing these agents involves Supervised Fine-Tuning (SFT) on expert trajectories (Lin et al., 2023; Rita et al., 2024; Chen et al., 2023; Yin et al., 2023), followed by Reinforcement Learning (RL) to iteratively refine decision-making (Carta et al., 2023; Tan et al., 2024; Song et al., 2024; Xiong et al., 2024; Yu et al., 2024). However, our preliminary experiments (Section 3) reveal that existing agents severely lack the capability for proactive exploration. In scenarios like online shopping, they exhibit rigid behavioral patterns, rarely executing exploratory actions (e.g., navigating to the "Next Page"). Consequently, they fail to strategize over extended horizons, frequently settling prematurely for suboptimal choices and yielding inferior overall rewards. We attribute this failure to a two-stage bottleneck inherent in the standard SFT-RL pipeline: 1) Hindsight Bias in SFT Data: Expert trajectories inherently strip away essential exploratory steps. Standard SFT merely conditions agents for reactive execution, failing to instill proactive exploration capabilities. 2) Exploration Collapse during RL: Because SFT-ed agents naturally lack exploratory

Introduction

In multi-turn interactive tasks like online shopping (Yao et al., 2022; Zhou et al., 2023b; Koh et al., 2024a), agents must continuously make decisions based on information acquired during interaction. Taking Figure 1 for example, when purchasing a low-cost shoe with specific attributes, the agent must decide whether to settle for a currently satisfactory item (A3 ), or continue browsing subsequent pages for a potentially better alternative (A6 ). Such scenarios require a critical capability: Proactive Exploration, which is the ability to autonomously expand its action trajectory beyond immediately local optima. This capability is fundamental for realworld agents, where optimal decisions often cannot be derived from the current observation alone, but instead emerge through strategic exploration over extended interaction horizons (Fang and Ke, 2025). * Corresponding author.

1

tendencies, they rarely sample such behaviors during RL rollouts, making it nearly impossible to discover their long-term benefits. Compounded by a lack of guidance, the RL process inevitably fails to reinforce proactive exploration. To this end, we propose S A FA R I, a novel method designed to inStill And reFine proActive exploRatIon capabilities in LLM agents. Specifically, it consists of two modules: 1) Exploratory Data Construction. Instead of human expert trajectories, we synthesize exploration-rich data directly from a powerful LLM expert. By employing a tree-structured context modeling, we explicitly distill diverse exploratory paths from the LLM expert, coupled with a customized screening mechanism to curate high-quality, exploration-oriented trajectories for the student agent’s SFT initialization. 2) RL with Contrastive Signal Guidance. We construct contrastive preference pairs by pitting a student-generated candidate action against the reference action at identical interaction states. By estimating their expected future returns via Monte Carlo (MC) rollouts, these contrastive signals provide fine-grained supervision to optimize the agent’s decision boundary. Thus S A FA R I alternates between strategic exploration and direct task execution, improving the task performance. We conduct extensive experiments across diverse benchmarks to evaluate S A FA R I. Results demonstrate that S A FA R I significantly outperforms competitive baselines in both task success (∼10%-15%, on average) and exploration efficiency (∼8%-18%). Additionally, our analyses illustrate that true performance improvements stem from exploration efficiency. S A FA R I learns to acquire task-relevant information while avoiding redundant interactions, whereas existing methods remain either passive or inefficient. Moreover, exploration benefits also grow with task difficulty. S A FA R I automatically scales its exploratory behavior in harder environments and achieves the largest gains on tasks that require substantial information gathering before decision-making. Our contributions are as follows:

benchmarks to demonstrate the effectiveness and characteristics of S A FA R I.

2

Related Work

Early interactive agents primarily relied on promptbased planning (Madaan et al., 2023; Yao et al., 2023b) or imitation learning (Lin et al., 2023; Chen et al., 2023), limiting their behavior to observed state–action distributions (Seo et al., 2024). To improve adaptability, recent work has introduced exploration-driven RL training and iterative selfimprovement (Song et al., 2024; Xiong et al., 2024; Yu et al., 2024). However, hindsight bias and exploration collapse often result in insufficient proactive exploration and yielding suboptimal solutions. To address this issue, recent research has proposed error recovery after explicit failures, achieved by corrective trajectories (Wang et al., 2025; Chen et al., 2025) and MCTS-based self-improvement (Yuan et al., 2025; Yu et al., 2025). Nevertheless, since these methods intervene only upon failure, they discourage exploration beyond reward-sufficient trajectories, often converging to local optima. Another line of work improves decision-making via structured representations. Tree-based reasoning methods enhance state visibility and search efficiency (Yao et al., 2023a; Zhou et al., 2023a; Koh et al., 2024b). Recent methods construct interaction graphs (Gandhi and Neubig, 2025) or exploit webpage hierarchies (Zhang et al., 2026) to guide navigation. However, these methods are designed for web environments and operate as external search tools rather than policy-level mechanisms, limiting their applicability to broader domains such as database manipulation and scientific environments.

3

Preliminary Experiments

We investigate the exploration capabilities of existing methods from two aspects: agents’ behavioral action distributions and their training dataset. 3.1

Evaluation Setup

Dataset & Exploratory Actions. We select WebShop (Yao et al., 2022) as our testbed due to its inherent decoupling of exploratory behaviors from terminal task execution, allowing for a precise analysis of the agent’s exploratory tendencies. Specifically, we categorize the action space into: (1) Task-Oriented Actions (Search, Open, Select, Buy), dedicated to immediate execution; and (2) Exploratory Actions (Next, Back, Home) designed

• We highlight the critical necessity of proactive exploration for LLM-based agents and identify two fundamental bottlenecks in existing methods. • We propose S A FA R I, designed to instill and refine proactive exploration capabilities in LLM agents. It features two customized modules to directly mitigate the two bottlenecks, respectively. • We conduct extensive experiments across various 2

Method

Task-Oriented Actions (%)

Avg. Len

Exploratory Actions (%)

Search

Open

Select

Buy

Next

Back

Home

Total

SFT-Only (Chen et al., 2023) ETO (Song et al., 2024) IPR (Xiong et al., 2024) STeCa (Wang et al., 2025)

3.33 4.32 4.39 4.87

30.15 23.13 23.22 20.85

30.15 22.99 22.18 24.59

9.85 28.69 29.89 28.22

29.85 24.84 24.60 25.31

0.00 0.12 0.00 0.00

0.00 0.23 0.11 0.21

0.00 0.00 0.00 0.83

0.00 0.35 0.11 1.04

Human Expert Trajectories (Yao et al., 2022)

3.64

28.20

27.69

16.51

25.31

0.00

0.10

0.00

0.00

Table 1: Comparison of action across baselines and human expert SFT trajectories in WebShop.

to actively explore the environment. Ultimately, we conduct a statistical analysis of these action distributions to diagnose the cause of agent failure. Baselines. We evaluate three categories of baselines: 1) SFT-Only (Chen et al., 2023), where agents are fine-tuned exclusively on standard expert trajectories; 2) SFT-RL, the prevailing paradigm that sequentially applies SFT and RL, represented here by two state-of-the-art approaches, ETO (Song et al., 2024) and IPR (Xiong et al., 2024), which employ trajectory-level and step-level supervision, respectively; and 3) SFT-RL-Correction, represented by STeCa (Wang et al., 2025), which augments the standard SFT-RL with post-hoc corrections by advanced LLM. All baselines utilize MetaLlama-3.1-8B-Instruct (Grattafiori et al., 2024). 3.2

dation in complex tasks. Thus, it is imperative to incorporate an exploration-rich SFT dataset and RL mechanisms that shift the agent’s behavior from reactive execution to proactive exploration.

4

S A FA R I: The Method

Notations. Following Xiong et al. (2024), we formulate the interactive task as a Markov Decision Process, defined by the tuple M = (U, S, A, O, T , R). Here, U, S, A, and O denote the spaces for natural language instructions, states, actions, and observations, respectively, governed by the transition function T : S × A → S and reward function R : S × A → [0, 1]. Given an instruction u ∈ U, the agent interacts sequentially. At step t, it samples an action at ∼ πθ (· | ht−1 ) conditioned on the history ht−1 = (u, a1 , o1 , . . . , at−1 , ot−1 ). Executing at transitions the environment to state st and yields observation ot . The episode terminates at step N (upon task completion or reaching a step limit), resulting in a final trajectory hN and a reward R(hN ), which serves as a metric for task success (Yao et al., 2022; Yang et al., 2023; Wang et al., 2022). Overview of S A FA R I. As illustrated in Figure 2, S A FA R I comprises two synergistic modules: (1) Exploratory Data Construction, which adopts a teacher-student framework and synthesizes exploration-rich trajectories from the teacher to mitigate the hindsight bias inherent in standard trajectories; and (2) RL with Contrastive Signal Guidance, which leverages contrastive sample pairs to calibrate the student’s decision boundary. This calibration enables the agent to distinguish valid exploration from redundant wandering, thereby optimizing the critical trade-off between information gathering and direct task execution.

Main Findings

Table 1 summarizes the results of the action distribution analysis. Evidently, both existing methods and the underlying SFT data profoundly lack exploratory actions. We highlight two bottlenecks. 1) Hindsight Bias in SFT Data. Expert trajectories strip away essential exploratory steps, with exploratory actions comprising a negligible 0.10%. Consequently, initializing agents on such biased corpora merely conditions them for purely reactive execution. This is exemplified by the SFT-Only baseline on WebShop, which exhibits near-zero exploratory actions (e.g., Next or Back), failing to instill the core capabilities required for proactive exploration. 2) Exploration Collapse during RL. Even with RL (ETO, IPR) or post-hoc correction (STeCa), we observe no qualitative shift in exploratory action distributions. Instead, these methods converge toward myopic optimization, merely refining item selection (i.e., ∼20% gains in distribution) within the current observation. Lacking exploratory tendencies in the SFT-ed policy, exploratory actions are rarely sampled during rollouts, preventing the agent from discovering their longterm rewards. As corroborated by our case studies (Appendix D) and main experiments (Section 5.2), this deficit leads to significant performance degra-

4.1

Exploratory Data Construction

This module elicits diverse exploratory paths from the LLM expert, coupled with a customized filtering mechanism to curate high-quality, explorationoriented trajectories for the SFT initialization. 3

Figure 2: Overview of S A FA R I.

4.1.1

Data Distillation

agency over its context via two core operations. • Autonomous Backtracking. When a path proves unpromising (e.g., yielding no matching items at A1 in Figure 2), the agent is prompted to autonomously backtrack by relocating pt . Depending on specific environment constraints, this is achieved either via environmental backward actions (e.g., executing the "Home" action at A2 ) or direct pointer modification (see in Appendix D). Consequently, the subsequent action naturally spawns a parallel exploration branch (e.g., branching from A1 to A3 ), isolating the failed trajectory while preserving its topology.

Motivation & Idea. Manually curating highquality, exploration-rich expert trajectories is prohibitively expensive and time-consuming. To circumvent this, S A FA R I proposes distilling these trajectories directly from a powerful LLM expert. However, off-the-shelf LLM experts themselves may struggle to exhibit strong proactive exploration capabilities. As empirically analyzed in Section 5.3.3, this deficiency stems from their limited context modeling capacities in complex, multi-turn environments. Relying on unstructured histories (i.e., u → a1 → o1 · · · → an → on ) may induce disorientation. As steps accumulate, the LLM loses track of global task states, its context window becomes polluted by local errors, and it ultimately regresses to conservative, reactive behaviors. To overcome this bottleneck, we introduce a structured context modeling mechanism, which formulates the history as a tree to track multi-turn interactions. Tree-Structured Context Modeling. We employ a tree structure to model the interaction context and track trial-and-error exploration. Formally, at step t, the context is represented as a tree Tt = (Vt , Et ), where each node v ∈ Vt denotes an executed action augmented with a cognitive note summarizing environmental feedback and key insights, each edge e ∈ Et captures the sequential parent-child dependency between actions, and pt is a global pointer indicating the current active state. Through operations such as branch creation and backtracking, the agent can dynamically revise and refine the tree branches over long horizons (see Figure 2). This formulation grants the agent active structural

• Active Feedback Digestion: The agent is also prompted to attach or update cognitive notes on specific nodes using environment observations. Crucially, these notes combine explicit control labels with text records of environment assets. For instance, in Figure 2, A1 is explicitly annotated as a dead end ("Not Match Item"), while A4 is annotated with a precise status ("With Red, XL Shoe But Price > $100"). Crucially, if the agent approaches the maximum step limit without discovering a optimal solution, it can strategically backtrack to a recorded suboptimal node (like A4 ) to secure a viable partial reward. Full operational details are provided in Appendix A. Structured Context Guided Data Distillation. We equip the teacher model (e.g., GPT-4o) with this tree scaffold to construct high-quality trajectories. Driven by this mechanism, the teacher explicitly searches, annotates, and backtracks, capturing complete trial-and-error trees. Consequently, S A FA R I distills not merely the final success path, but 4

the authentic cognitive reasoning and complete interaction trajectories, effectively eradicating the hindsight bias inherent in standard expert datasets. 4.1.2

ence Optimization (DPO) on these scored pairs to explicitly enforce this proactive decision boundary. Contrastive Pair Construction. We generate contrastive pairs by isolating pivotal moments where the agent must determine its interaction strategy. Using a shared interaction context st (comprising the instruction, current observation, and tree history) from the reference set Dref as the common starting point, we pair the reference action a∗ with a candidate action â ∼ πθ (· | st ) generated by the student agent. Since both â and a∗ can manifest as either a standard execution step or a proactive exploratory action, this bifurcated pair naturally encompasses diverse behavioral contrasts such as effective exploration versus redundant wandering, or strategic exploration versus direct task execution. By explicitly pitting these divergent choices against each other, we create a comparison space that enables the model to learn the most beneficial strategy. Reward Estimation for Pairs via MC Rollouts.. To objectively determine the utility of each ‘action branch’ within the contrastive pairs, we employ MC rollouts to estimate their respective future returns. To prevent backward loops from inflating rollout variance, we employ a forward-only evaluator (πMC ). This evaluator is specifically trained on an Forward-Pure Set (Dscore ), which is curated by pruning all backward actions from Dbase to retain only the most direct paths to success. By utilizing this "focused" policy, we perform multiple rollouts from both candidate actions to the terminal state. The step-level value is defined as the expected terminal reward:

Data Screening

Motivation & Idea. While the tree-structured context unlocks the LLM expert’s exploratory potential, it can occasionally induce over-exploration, leading to unnecessarily lengthy paths or eventual task failure. This motivates us to screen these trajectories for high-quality proactive behaviors. For each task, we independently sample m trajectories from the constructed dataset and evaluate them using a length-penalized reward: rfinal = r(h) − n · γ

(1)

where r(h) ∈ [0, 1] is the task success reward, n is the step count, and γ is a penalty factor. We select the candidate that maximizes rfinal to form our reference trajectory set Dref . This formulation inherently favors concise execution, rigorously penalizing redundant wandering while preserving only necessary, meaningful exploration. Additionally, to further enhance trajectory quality, we further apply a strict performance filter to Dref , retaining exclusively successful trajectories (r(h) = 1) to construct the final base dataset Dbase for SFT. As such, this structured construction explicitly internalizes the teacher’s proactive logic, endowing the student with a robust initialization for proactive exploration. Finally, we perform SFT on the student agent using our dataset Dbase . To prevent overfitting and preserve foundational capabilities prior to subsequent preference optimization, this SFT phase is kept intentionally brief (e.g., 1 epoch). As such, S A FA R I equips the student with a robust, exploration-aware initialization. 4.2

rstep (a|st ) = Eτ ∼πMC [R(τ )|st , a],

(2)

where a ∈ {â, a∗ } and R(τ ) ∈ [0, 1] is the aggregate task success reward. This scoring mechanism provides a quantitative ground truth for the proactive dilemma: it reveals whether an exploratory "detour" ultimately facilitates a higher success rate compared to immediate execution. Optimization via DPO. Based on the estimated values, we construct preference pairs where the higher-rewarding action is labeled as the preferred response. By optimizing the student agent via DPO, alongside an SFT regularization term to mitigate catastrophic forgetting, we effectively calibrate the agent’s proactive decision boundary. To ensure the agent learns from decisive strategic choices rather than stochastic evaluation noise, we filter

RL with Contrastive Signal Guidance

Motivation & Idea. While SFT endows the agent with basic exploration skills, it lacks the explicit feedback required to master the precise timing of when to explore versus when to execute. To calibrate this decision boundary, we frame proactive exploration as a preference optimization problem. Inspired by Xiong et al. (2024), we construct contrastive preference pairs between student-generated actions and teacher reference actions, utilizing MC rollouts to objectively quantify their expected future returns. Ultimately, we apply Direct Prefer5

the preference pairs using a margin threshold τm , retaining a pair only if the reward gap satisfies |rstep (â|st ) − rstep (a∗ |st )| > τm . L = LDPO + LSFT

5

Experiment

5.1

Experimental Settings

(3)

Baselines & LLM Backbones. We adopt the identical set of baselines previously analyzed in Section 3. Following established protocols (Xiong et al., 2024), all evaluated baselines and S A FA R I share identical backbone LLMs: Meta-Llama-3.18B-Instruct (Grattafiori et al., 2024) and Mistral7B-Instruct-v0.3 (Jiang et al., 2023). Datasets. Following Xiong et al. (2024); Song et al. (2024), we consider three benchmarks presenting distinct challenges: 1) WebShop (Yao et al., 2022), an e-commerce platform for statedependent spatial exploration and backtracking; 2) InterCode-SQL (Yang et al., 2023), a database querying environment for state-independent logical complexity and precise syntax generation; and 3) ScienceWorld (Wang et al., 2022), a text-based scientific simulator for long-horizon planning and persistent exploration in complex dynamics. Evaluation Metrics. We evaluate agents using two metrics: (1) Task Performance (TP): Following Xiong et al. (2024), we report the average final reward to quantify the agent’s problem-solving proficiency in real-world deployments. (2) Exploration Score (ES): Measured as the maximum reward achieved across five stochastic rollouts, this metric captures the agent’s exploratory upper bound and reflects its ability to discover high-quality trajectories through active exploration. Implementation Details. All baselines are implemented using their official code repositories. Following Wang et al. (2025), we employ GPT-4o as teacher for data distillation. Refer to Appendix B for more implementation details. 5.2

Figure 3: Exploration efficiency analysis across training phases. Baselines are either rigidly passive (ETO, IPR) or degenerate into blind wandering (STeCa). In contrast, S A FA R I successfully suppresses wasteful exploration (WE) and elevates effective exploration (EE) post-SFT, maximizing net efficiency (Eff).

approximately 10%-15% in task success and approximately 8%-18% in exploration score. Notably, the proposed tree-structured context modeling consistently boosts the performance of even the advanced teacher model (GPT-4o), proving that structuring history effectively resolves environmental uncertainty and facilitates systematic exploration (further discussed in Section 5.3.3). Impressively, our 8B S A FA R I model delivers performance comparable to/better than this enhanced teacher. This result suggests that our SFT+RL paradigm effectively internalizes trial-and-error behaviors, allowing a smaller model to match the proactive exploration capabilities of a leading proprietary LLM. 5.3

In-depth Analysis

We study how efficiently S A FA R I explores, and how exploration scales with task difficulty. 5.3.1

Exploration Efficiency

Unlike random exploration, proactive agent relies not on the mere quantity of exploratory actions or trajectories, but on strategic efficiency, which is the ability to purposefully gather critical information for decision-making under bounded steps. We thus evaluate exploration efficiency to quantify how effectively the agent acquires useful information while avoiding redundant wandering. In our setting, we quantify exploration by tracking both how broadly the agent explores the environment and how deeply it investigates discovered entities. Setup. We evaluate all methods based on the

Main Results

Table 2 demonstrates the effectiveness of S A FA R I, as evidenced by the following observations. S A FA R I achieves strong performance in terms of both task success and exploration score. Compared to the best baselines, S A FA R I consistently achieves substantial gains across all benchmarks and backbones, delivering an average increase of 6

Base Model

InterCode-SQL

ScienceWorld

TP ↑

WebShop ES ↑

TP ↑

ES ↑

TP ↑

ES ↑

Method

GPT-4o (teacher)

Direct Prompt Tree-Structured Prompt (S A FA R I)

0.5765 0.7062

0.6877 0.7931

0.6364 0.7275

0.7339 0.8578

0.7156 0.7581

0.8451 0.8471

Llama-3-8B

SFT-Only (Chen et al., 2023) ETO (Song et al., 2024) IPR (Xiong et al., 2024) STeCa (Wang et al., 2025) S A FA R I

0.6267 0.6642 0.6966 0.6574 0.7209

0.6366 0.6916 0.7087 0.7864 0.8358

0.6320 0.6655 0.6740 0.6276 0.7255

0.6653 0.7161 0.7019 0.7940 0.8625

0.5608 0.5206 0.5681 0.5340 0.6063

0.6917 0.6314 0.6393 0.6593 0.7096

Mistral-7B

SFT-Only (Chen et al., 2023) ETO (Song et al., 2024) IPR (Xiong et al., 2024) STeCa (Wang et al., 2025) S A FA R I

0.6037 0.6297 0.6207 0.6275 0.7168

0.6048 0.6324 0.6334 0.7372 0.7851

0.5315 0.5961 0.5821 0.5719 0.7058

0.7077 0.6185 0.7438 0.6849 0.8563

0.5368 0.4448 0.4689 0.4761 0.5884

0.6602 0.6585 0.5851 0.6095 0.7036

Table 2: Main results. We bold the best results and underline the second best. S A FA R I achieves strong performance in both execution (TP) and exploration (ES), validating the universal benefit of proactive exploration.

Meta-Llama-3.1-8B-Instruct backbone and analyze their exploration dynamics across different training stages. We measure the exploration efficiency of a trajectory τ by a metric Eff(τ ) (detailed in Appendix C). Specifically, it quantifies the overall net utility of exploration by subtracting wasted efforts from effective gains: Eff(τ ) = EE(τ ) − WE(τ ), where EE(τ ) = R(τ ) · E(τ ) and WE(τ ) = (1 − R(τ )) · E(τ ) denote task-contributing and non-contributing exploration, respectively. Here, R(τ ) ∈ [0, 1] denotes the terminal task reward, and E(τ ) ∈ [0, 1) rigorously quantifies the surplus information acquired beyond basic task completion across different interaction depths (e.g., exposed and focused entities).

ration (WE). Upon entering the RL stage, our customized optimization progressively calibrates the exploration strategy. At RL1 (i.e., the first round of RL optimization), WE is sharply reduced, making the net exploration efficiency (Eff = EE − WE) positive. As training proceeds through RL2 and RL3, EE and WE follow similar trends, yet the gap between them steadily widens. This pattern closely mirrors the performance gains reported in Table 2, suggesting that S A FA R I proactively acquires taskrelevant information while avoiding redundant exploration, rather than relying on exploration only when encountering dead ends.

Results. The evaluation results are illustrated in Figure 3. Overall, S A FA R I proactively acquires useful information while avoiding redundant exploration, leading to sustained improvements in both exploration efficiency and task performance. We derive the following detailed observations.

Method / Task Difficulty

Count

R↑

Len.

E

EE

WE

Eff

Easy (RB = 1) S A FA R I (R = 1) S A FA R I (R ∈ [0.5, 1)) S A FA R I (R ∈ [0, 0.5))

60 16 4

1.00 0.64 0.21

4.47 6.00 8.00

0.08 0.40 0.32

0.08 0.26 0.08

0.00 0.14 0.24

0.08 0.11 -0.15

IPR (Overall) S A FA R I (Overall)

80 80

1.00 0.89

3.97 4.95

0.02 0.16

0.02 0.12

0.00 0.04

0.02 0.08

Medium (RB ∈ [0.5, 1)) S A FA R I (R = 1) S A FA R I (R ∈ [0.5, 1)) S A FA R I (R ∈ [0, 0.5))

21 46 7

1.00 0.64 0.20

5.33 5.17 7.14

0.20 0.17 0.37

0.20 0.11 0.02

0.00 0.07 0.35

0.20 0.04 -0.32

IPR (Overall) S A FA R I (Overall)

74 74

0.65 0.70

4.06 5.41

0.01 0.20

0.01 0.12

0.00 0.07

0.01 0.05

Hard (RB ∈ [0, 0.5)) S A FA R I (R = 1) S A FA R I (R ∈ [0.5, 1)) S A FA R I (R ∈ [0, 0.5))

7 11 26

1.00 0.71 0.19

4.43 5.73 6.88

0.13 0.20 0.47

0.13 0.15 0.09

0.00 0.05 0.38

0.13 0.11 -0.29

IPR (Overall) S A FA R I (Overall)

44 44

0.23 0.45

4.18 6.20

0.01 0.35

0.00 0.11

0.01 0.24

0.00 -0.12

Total (All Tasks) S A FA R I (R = 1) S A FA R I (R ∈ [0.5, 1)) S A FA R I (R ∈ [0, 0.5))

88 73 37

1.00 0.65 0.20

4.67 5.44 7.05

0.11 0.23 0.43

0.11 0.15 0.08

0.00 0.08 0.36

0.11 0.07 -0.28

IPR (Overall) S A FA R I (Overall)

198 198

0.70 0.72

4.05 5.40

0.02 0.22

0.01 0.12

0.00 0.10

0.01 0.02

5.3.2

Baselines fail to conduct effective exploration, resulting in lower Eff values. On W EB S HOP, ETO and IPR exhibit near-zero EE and WE across all phases, exposing their rigid passivity and perfectly aligning with our behavioral analysis in Section 3. Conversely, while the correction-based STeCa attempts exploration, its wasteful exploration (WE) aggressively overtakes effective exploration (EE), crashing its net efficiency. This confirms that existing methods tend to induce unfocused exploration rather than productive information seeking.

Exploration Scales with Task Difficulty

Table 3: Exploration scales with task difficulty. Tasks are stratified by the mean reward of IPR (RB ).

S A FA R I successfully balances effective exploration and task execution. As shown in Figure 3, our SFT data encourages extensive exploration but lacks precision, resulting in high levels of both effective exploration (EE) and wasted explo-

Setup. To understand how proactive exploration contributes to task completion under different levels of difficulty, we analyze agent trajectories in 7

• S A FA R I Variants. We perform ablations on all major components of S A FA R I. To evaluate the effectiveness of our SFT data, we train S A FA R I using the SFT data adopted by prior methods (i.e., w/ Original SFT).

W EB S HOP and partition the evaluation set into Easy, Medium, and Hard subsets according to the baseline (IPR) average reward (RB ). Table 3 reports both task-level statistics (reward R and trajectory length Len.) and exploration metrics (E, EE, WE, and Eff). We highlight three key findings. Exploration intensity increases with task difficulty. The baseline follows a largely fixed execution pattern across all difficulty levels, with trajectory lengths remaining around 4.0 and exploration extent (E) close to zero. In contrast, S A FA R I adjusts its exploration behavior according to task complexity. As difficulty increases from Easy to Hard, both exploration extent (E: 0.16 → 0.20 → 0.35) and trajectory length (4.95 → 5.41 → 6.20) increase accordingly. This suggests that S A FA R I allocates more exploratory effort when tasks require additional information gathering. Exploration is most beneficial in hard tasks. The largest performance gains are observed in the Hard subset, where S A FA R I improves the average reward from 0.23 to 0.45. Notably, these gains are accompanied by a substantial level of effective exploration (EE = 0.11), indicating that the additional interactions contribute meaningful information rather than merely increasing search activity. This result suggests that proactive information acquisition becomes particularly valuable when direct task execution alone is insufficient. Exploration introduces a trade-off in easy tasks. In Easy environments (RB = 1.0), where tasks can often be solved through direct execution, proactive exploration provides less benefit and may introduce minor inefficiencies. Although the agent acquires useful contextual information (EE = 0.12), the associated wasted exploration (WE = 0.04) slightly reduces execution efficiency, leading to a modest decrease in reward (1.00 → 0.89). This observation highlights the trade-off between information gathering and execution efficiency when additional exploration is unnecessary.

Backbone

Method

TP

ES

GPT-4o GPT-4o GPT-4o GPT-4o GPT-4o GPT-4o

Direct Prompt Exploration-aware Prompt List-Structured Prompt w/o note List-Structured Prompt Tree-Structured Prompt w/o note Tree-Structured Prompt

0.5765 0.5946 0.6471 0.6640 0.6724 0.7062

0.6877 0.7395 0.7785 0.7771 0.7759 0.7931

Llama-3-8B Llama-3-8B Llama-3-8B Llama-3-8B Llama-3-8B

S A FA R I w/ Original SFT S A FA R I w/o Tree-Structured S A FA R I w/o Data Screening S A FA R I w/o RL S A FA R I

0.6574 0.6843 0.6965 0.6321 0.7209

0.7914 0.8028 0.8108 0.7949 0.8358

Results. While explicitly encouraging Table 4: Ablation studies on W EB Sexploration HOP . improves performance over direct prompting, the gains are relatively limited (+3.14% in TP and +7.53% in ES). In contrast, both list-structured and tree-structured contexts yield substantially larger improvements (+12.24% in TP and +13.28% in ES, +16.63% in TP and +12.83% in ES), highlighting the importance of organizing interaction histories for exploration. Among them, tree-structured prompts perform best, suggesting that preserving branching trial-and-error trajectories facilitates more effective exploration. Adding cognitive notes further improves performance (+5.03% in TP and +2.23% in ES), indicating that compact task-state summaries provide richer contextual signals for exploration. Importantly, the benefits transfer to downstream agent training. Equipping with treestructured prompts and data screening leads to a high-quality SFT dataset and hence a noticeable performance gain (+9.66% in TP and +5.61% in ES). RL also contributes large gains. However, its effectiveness depends on prior SFT initialization, suggesting that basic exploration capabilities must first be installed before they can be further efficiently refined through RL.

6

5.3.3 Ablation Studies Variants are as follows (Appendix E for details).

Conclusion

For real-world environments, intelligent behavior requires more than acting on currently available observations. Effective agents must proactively explore, strategically acquiring information that may incur short-term costs but improves future decisionmaking. Such behavior reflects the ability to reason not only about what action to take, but also about whether the current knowledge state is sufficient for reliable planning. We argue that proactive

• Teacher Variants. We study how teacher (GPT4o) affects the quality of exploration-rich data distillation. Specifically, we compare GPT-4o with and without an exploration-aware prompt, which instills a basic exploration instinct, and further test different context modeling strategies, including list-structured versus tree-structured histories, as well as the use of cognitive notes. 8

exploration is a foundational capability of nextgeneration agents, enabling robust decision-making in complex open-world environments where optimal actions depend on actively expanding and refining environmental understanding.

9

Limitations

Baian Chen, Chang Shu, Ehsan Shareghi, Nigel Collier, Karthik Narasimhan, and Shunyu Yao. 2023. Fireact: Toward language agent fine-tuning. arXiv preprint arXiv:2310.05915.

In this section, we discuss the limitations of our work from the following perspectives: Heuristic Exploration Metrics. As an initial attempt toward evaluating proactive exploration in LLM agents, we propose the metric Eff(τ ) to measure exploration efficiency in our experimental setting. While Eff(τ ) provides a measurable proxy for exploration utility, it is not strictly equivalent to actual effective exploration. Because the calculation is explicitly anchored to the terminal task reward, it can inherently inflate the effective exploration scores in successful trajectories, regardless of whether the exploratory actions causally contributed to that success. Nevertheless, Eff(τ ) still exhibits a meaningful macroscopic correlation with exploration behavior. Developing more rigorous metrics to isolate the causal impact of exploration remains a direction for future work. SFT Dependency. While S A FA R I equips agents with proactive exploration capabilities, this ability currently relies on supervised fine-tuning (SFT) using exploration-aware trajectories distilled from a powerful LLM (e.g., GPT-4o). Our analysis shows that this stage is critical, as removing it leads to severe hindsight bias during training. However, the resulting exploratory behaviors may also be implicitly bounded by the prior knowledge of the teacher LLM. In our experiments, existing agents are still unable to efficiently interact with and acquire environmental knowledge entirely from scratch (as evidenced by the S A FA R I w/ Original SFT ablation). Moving beyond this SFT-dependent paradigm toward fully autonomous agents capable of proactive exploration from scratch remains an important direction for future work.

Zhixun Chen, Ming Li, Yuxuan Huang, Yali Du, Meng Fang, and Tianyi Zhou. 2025. Atlas: Agent tuning via learning critical steps. In Findings of the Association for Computational Linguistics: ACL 2025, pages 25334–25349. Djengo Cyun-Jyun Fang and Tsung-Wei Ke. 2025. Information seeking for robust decision making under partial observability. arXiv preprint arXiv:2510.01531. Apurva Gandhi and Graham Neubig. 2025. Go-browse: Training web agents with structured exploration. arXiv preprint arXiv:2506.03533. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, and 1 others. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825. Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. 2024a. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649. Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. 2024b. Tree search for language model agents. arXiv preprint arXiv:2407.01476. Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. 2023. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. In Proceedings of the 36th Conference on Neural Information Processing Systems, volume 36, pages 23813–23825.

LLM Usage LLMs are used in this work as the backbone models and the teacher models for trajectory data construction. In addition, LLMs are used only for language polishing during paper writing.

Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: iterative refinement with self-feedback. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 46534–46594.

References Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. 2023. Grounding large language models in interactive environments with online reinforcement learning. In International conference on machine learning, pages 3676–3713. PMLR.

Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. 2024. Countering reward over-optimization in llm with demonstration-guided reinforcement learning. arXiv preprint arXiv:2404.19409.

10

Seokin Seo, Byung-Jun Lee, Jongmin Lee, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. 2024. Mitigating covariate shift in behavioral cloning via robust stationary distribution correction. Advances in Neural Information Processing Systems, 37:109177– 109201.

with unified data, modular design, and open-source llms. In ICLR 2024 Workshop on Large Language Model (LLM) Agents. Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. 2024. Exact: Teaching ai agents to explore with reflectivemcts and exploratory learning. arXiv preprint arXiv:2410.02052.

Yifan Song, Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. 2024. Trial and error: Exploration-based trajectory optimization for llm agents. arXiv preprint arXiv:2403.02502.

Xiao Yu, Baolin Peng, Vineeth Vajipey, Hao Cheng, Michel Galley, Jianfeng Gao, and Zhou Yu. 2025. Exact: Teaching ai agents to explore with reflectivemcts and exploratory learning. In International Conference on Learning Representations, volume 2025, pages 65157–65184.

Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. 2024. True knowledge comes from practice: Aligning llms with embodied environments via reinforcement learning. arXiv preprint arXiv:2401.14151.

Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. 2025. Agent-r: Training language model agents to reflect via iterative selftraining. arXiv preprint arXiv:2501.11425.

Hanlin Wang, Jian Wang, Chak Tou Leong, and Wenjie Li. 2025. Steca: Step-level trajectory calibration for llm agent learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11597– 11614.

Xuanwang Zhang, Yuteng Han, Jinnan Qi, Mulong Xie, Zhen Wu, and Xinyu Dai. 2026. Webnavigator: Global web navigation via interaction graph retrieval. arXiv preprint arXiv:2603.20366.

Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. 2022. Scienceworld: Is your agent smarter than a 5th grader? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11279–11298.

Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2023a. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406.

Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang, Cheng Li, Wei Peng, and Sujian Li. 2024. Watch every step! llm agent learning via iterative step-level process refinement. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1556– 1572.

Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, and 1 others. 2023b. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854.

John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. 2023. Intercode: Standardizing and benchmarking interactive coding with execution feedback. Advances in Neural Information Processing Systems, 36:23826–23854.

A

Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable realworld web interaction with grounded language agents. In Proceedings of the 35th Conference on Neural Information Processing Systems, volume 35, pages 20744–20757.

In this section, we provide the formalization, algorithmic conversion, operational mechanisms, and environment-specific implementations of our proposed Tree-Structured History.

Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023a. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809–11822.

A.1

Details of the Interactive Tree-Structured History with Cognitive Notes

Formalization and Algorithmic Conversion

In multi-turn interactive environments, an interaction trajectory is conventionally represented as a flat, chronological sequence of dialogue turns:

Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023b. React: Synergizing reasoning and acting in language models. In Proceedings of the 11th International Conference on Learning Representations.

Ht = {u, (a1 , o1 ), (a2 , o2 ), . . . , (at , ot )}, where u is the task instruction, ai is the executed action, and oi is the environmental observation. This linear representation implicitly conflates productive forward execution with exploratory detours. When

Da Yin, Faeze Brahman, Abhilasha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2023. Lumos: Learning agents

11

Algorithm 1 Dialogue-to-Tree History Conversion 1: Initialize a root node v1 as START; set current pointer p ← v1 . 2: for each interaction turn t do 3: Receive latest observation ot , current tree Tt , global pointer p, and step count. 4: Agent Inference: Analyze the history and plan the exploration strategy based on ot and Tt . 5: Agent Output: Generate structured fields: action, tree_signal, and optional tree.change(id, note). 6: if tree_signal specifies Next Parent Assignment to vj (where vj can be p) then 7: Append action as a child node of historical node vj . 8: Move global pointer p to this newly created node after execution. 9: else if tree_signal specifies Target Node Relocation to vj then 10: Teleport global pointer p directly to existing node vj without expanding a new node. 11: end if 12: end for

an agent backtracks after a failed attempt, a flat list still records the abandoned branch in sequence, which frequently misleads the agent into treating a discarded state as part of its active context. To resolve this linear disorientation, we map the trajectory into a topological action tree Tt . Formally, each node vi ∈ Tt is defined as a tuple: vi = (idi , ai , parenti , childreni , notei ), where idi is a unique node identifier, ai is the executed action, parenti points to the preceding action node from which the current branch originates, and childreni stores the tracking IDs of subsequent action branches. Crucially, notei is a plain-text field managed by the agent. It can either start with an optional control label followed by text, or consist of plain text alone. The optional labels represent different exploration states: [SUCCESS] for completed milestones, [FAILED] for execution errors, [DEAD END] for fully exhausted paths, and [SUB-OPTIMAL] for imperfect backup candidates. Furthermore, we found that the plain text splits into two types of information: a record to log valuable content about the current state, and a note to serve as a self-reminder, without requiring any rigid tags to separate them. The active position pointer pt is maintained globally outside the nodes to track the agent’s current state. For the exact structured output formats and prompt execution templates, refer to Appendix F. The operational sequence-to-tree conversion pipeline is formalized in Algorithm 1. A.2

2) Direct Pointer Modification (InterCodeSQL & ScienceWorld): In domains like database manipulation or textual scientific simulations, environments lack automated or single-step physical backtracking mechanisms to restore full historical contexts. To resolve this, tree relocation is handled via direct pointer modification, where the system overwrites the global pointer pt outside the environmental execution loop. As illustrated in Figure 4, this direct modification operates under the two control modes defined in our algorithm:

Dual Relocation Mechanisms Across Different Target Domains

A core strength of our framework is its adaptability to environments with entirely different action dynamics. As introduced in Algorithm 1, relocating the active tree pointer pt during backtracking is operationalized via two distinct paradigms depending on whether the environment natively supports structural reversion: 1) Action-Driven Synchronization (WebShop): In environments that inherently support explicit backward actions (e.g., clicking a <Back> button on an e-commerce interface), tree relocation is perfectly synchronized with environmental state transitions. Unproductive paths are structurally isolated into parallel side branches whenever the agent executes a physical backtracking command provided by the environment.

• Target Node Relocation: The active position pointer pt immediately jumps back to a designated historical node ID, allowing the policy to re-examine or resume execution from that past checkpoint without spawning a new node. • Next Parent Assignment: The agent assigns a historical node ID as the parent for its next action. The subsequent action is then executed from this designated parent context, automatically spawning a new parallel exploration branch (e.g., the transitions from A3 → A4 and A5 → A6 in Figure 4).

12

in a forgetful stream. As shown in Figure 5, node 2 caches a clean snapshot of unexamined kitchen appliances (freezer, fridge). This ensures that even when the agent leaves the kitchen, it retains high-fidelity spatial knowledge without relying on repetitive look around actions. • Recording Branch Status: Notes use explicit tags to manage search boundaries. At node 10 (teleport to workshop), after finding no juicing utilities, the agent updates the note with a [DEAD END] tag. This explicit status marker signals the policy to immediately halt forward execution along this unpromising branch. Figure 4: Tree-structured context modeling in InterCode-SQL via direct pointer modification. Only the solid-line tree paths are visible to the agent’s policy during execution.

A.3

• Encoding Learned Rules: When actions trigger environmental constraints, the agent records them under a RULE LEARNED prefix. For instance, at node 3, the environment rejects a vague picking action, prompting the agent to log RULE LEARNED: select index. This prevents the agent from repeating syntactical or mechanical errors in future turns.

Operational Roles of Cognitive Notes with Concrete Case Study

Cognitive notes transform the passive tree topology into an operational working memory. Rather than serving as a plain textual summary, these fields are autonomously filled by the agent’s policy to perform four interconnected roles. We illustrate these roles using a concrete long-horizon task in ScienceWorld ("freeze orange juice"), where the agent must navigate a complex environment based on an unverified hypothesis that orange juice must be extracted from a raw orange.

• Hypothesis Revision and Checkpoint Recovery: When a core assumption fails, the note preserves the failure rationale to guide hypothesis shifting. Upon exhausting the workshop branch (node 10), the agent deduces that raw oranges are a dead end. It writes: "[DEAD END] Workshop contains no juicing tools. RULE LEARNED: Avoid implicit constraints. Hypothesis revised: Pre-made juice likely exists in unexamined kitchen appliances."

START(id=1) |-- teleport to kitchen(id=2) | |-- pick up orange(id=3) | `-- pick up orange (specific location)(id=4) | |-- look at drawer(id=7) | `-- examine cupboard(id=9) `-- teleport to workshop(id=10)

Node

Agent-Generated Cognitive Note

1 2

Hypothesis: kitchen likely contains orange juice. Found orange; unexamined drawer, cupboard, freezer, fridge. [FAILED] Ambiguous request; RULE LEARNED: select index. [SUCCESS] Orange picked up; search for juicing tool. [DEAD END] No tools. [DEAD END] No juicing tools. Look around just now. → [DEAD END] Workshop contains no juicing tools. ...

3 4 7 9 10

Simultaneously, the agent triggers a pointer relocation outside the environment loop: Current position id: 2, returning directly to the kitchen checkpoint. Guided by the cached facts at node 2, the agent seamlessly initiates a new sibling branch targeting the fridge or freezer. This synergy between topological pointer recovery and autonomous cognitive notes ensures the agent eliminates redundant wandering and achieves purposeful, proactive exploration.

B

Figure 5: An excerpt of the tree history from the orangejuice freezing task in ScienceWorld. Green and red text signify productive and dead-end action paths, respectively.

Hyperparameters and Implementation Details

This section details the exhaustive hyperparameters and implementation configurations utilized across the different training phases of S A FA R I to ensure full reproducibility. All experiments are conducted

• Caching Verified Environmental Facts: Interactive environments often present observations 13

on 3 NVIDIA A100 (80GB) GPUs, while the corresponding core prompt templates are provided in Appendix F.

the agent acquires candidate information and how deeply it verifies that information through explicit interaction, and then coupling the result with the task reward. Trajectory entity extraction. Given a trajectory hN = (u, a1 , o1 , . . . , aN , oN ) of N interaction turns, at each step t we extract two raw entity sets from the current action-observation pair (at , ot ): Êt (exposed) and F̂t (focused). Exposed entities represent information breadth, capturing entities coarsely revealed by the environment observation without direct interaction. Focused entities represent information depth, capturing entities that the agent explicitly inspects or manipulates (e.g., entering a specific link, selecting an option, or clicking to read details). In W EB S HOP, after a search or page-turn action, product IDs visible in the current result list are counted as exposed entities; a product is counted as focused only if the agent has clicked into its detail page at least once. Formally, we define an environment-specific extraction function ϕtask and a canonicalization filter ψtask . The extraction process uses the history context ht−1 to resolve coreferences or contextual entities. After task-specific filtering, we obtain the valid step-level sets Et and Ft , which are then accumulated and deduplicated over the full trajectory:

• Data Synthesis via Teacher Model: Reference trajectories are synthesized utilizing GPT4o. Specifically, for each individual task, we independently sample m = 5 candidate trajectories. The decoding temperature is set to T = 1.0 with a top_p of 0.85. The step penalty factor γ is set to 0.1 for WebShop and InterCode-SQL, and 0.01 for ScienceWorld. • Supervised Fine-Tuning (SFT) Phase: Prior to contrastive optimization, the core components undergo initial fine-tuning. The proactive agent is trained for strictly 1 epoch with a learning rate of 3 × 10−6 to establish baseline capabilities while preventing premature overfitting. Concurrently, the MC evaluator (πMC ) is fine-tuned on the Forward-Pure Set (Dscore ) for 3 epochs with a learning rate of 2 × 10−5 . • Exploration Mechanics: During the active exploration phase, the agent utilizes deterministic greedy decoding (T = 0). For step-wise reward estimation, the MC rollout process switches to stochastic decoding (T = 1.0) to execute 5 independent rollouts per candidate action. • Contrastive Data Construction: To isolate decisive strategic differences from environmental noise, the margin threshold τm is set to 0.01 for both WebShop and ScienceWorld, and 0.1 for the discrete action space of InterCode-SQL.

E(hN ) =

Et ,

F (hN ) =

t=1

N [

Ft .

(4)

t=1

Crucially, in our current implementation, the extraction functions (ϕtask , ψtask ) and the entity category mappings (κ) are entirely rule-based. These parsing rules and token-matching heuristics are manually customized and hard-coded for each specific target environment to ensure exact factual alignment. Category entity counts. Let C be the task-specific category set, and let κ(e) ∈ C map an entity e to its corresponding category. For each category c ∈ C, we aggregate the total number of unique entities encountered across the entire trajectory. The accumulated counts for exposed and focused states are defined as:

• Contrastive Policy Optimization: The contrastive learning update employs a learning rate of 5 × 10−7 and a KL regularization coefficient β = 0.1. The model is optimized for 3 epochs per global training iteration. • Global Pipeline Iterations: The entire closedloop training cycle (encompassing active exploration, MC utility estimation, and contrastive policy calibration) is executed for 3 global iterations on WebShop and InterCode-SQL. For ScienceWorld, the framework converges optimally within 1 global iteration.

C

N [

Information-Theoretic Exploration Metrics

Ne (c; hN ) =

This metric quantifies the net utility of exploration along a trajectory by jointly measuring how broadly

Nf (c; hN ) =

X

I[κ(e) = c],

e∈E(hN )

X e∈F (hN )

14

(5) I[κ(e) = c],

Algorithm 2 Trajectory-Level Entity Accounting

ploration exceeding the baseline standard contributes to the score, while substandard wandering is bounded at zero. Category aggregation. We aggregate categorywise gains into two global trajectory scores:

1: E ← ∅, F ← ∅ 2: for t = 1, . . . , N do 3: (Êt , F̂t ) ← ϕtask (at , ot , ht−1 ) {contextual raw extraction via heuristic rules} 4: (Et , Ft ) ← ψtask (Êt , F̂t ) {canonicalize & filter via heuristic rules} 5: E ← E ∪ Et , F ← F ∪ Ft {trajectory-level dedup} 6: end for 7: Compute {Ne (c; hN ), Nf (c; hN )}c∈C from E, F via Eq. (5) 8: Normalize by fixed pool reference {Ne0 (c), Nf0 (c)}c∈C to get {xe (c; hN ), xf (c; hN )} 9: Aggregate category gains to obtain global trajectory scores Xe (hN ), Xf (hN ) 10: Compute saturation P (hN ) via exponential mapping 11: Compute final dimensions: EE(hN ), WE(hN ), Eff(hN )

Xe (hN ) = Aggc∈C xe (c; hN ), Xf (hN ) = Aggc∈C xf (c; hN ),

(8)

where Agg ∈ {mean, sum}. In our primary analysis, we employ mean to balance information acquisition evenly across distinct domains. Saturation. To prevent arbitrarily long trajectories from disproportionately inflating the exploration metrics through brute-force wandering, we apply an exponential saturation function: P (hN ) = 1 − e−α (Xe (hN )+Xf (hN )) ,

(9)

where α = 1 is a fixed scaling hyperparameter across all reported experiments. Reward coupling. Finally, we couple the exploration intensity P (hN ) with the environment task reward R(hN ) ∈ [0, 1] to differentiate purposeful information gain from blind, ungrounded wandering. To achieve this, we decouple the raw exploration score into two orthogonal dimensions:

where I[·] is the standard indicator function that outputs 1 if the condition holds and 0 otherwise. Unified-mean baseline. To establish a stable reference of standard agent behavior and filter out basic entities that are mandatory for simple task completion, we introduce a shared reference baseline pool B. To avoid numerical drifting across different experimental runs, B is fixed as the collection of validation trajectories generated by the backbone’s vanilla SFT checkpoints across all evaluation methods. The cross-method mean baselines are calculated as: 1 X Ne (c; h′ ), Ne0 (c) = |B| ′ h ∈B (6) X 1 0 ′ Nf (c) = Nf (c; h ). |B| ′

• Effective Exploration (EE): This dimension captures the volume of proactive exploration that successfully aligns with final task resolution. It is defined as: EE(hN ) = R(hN ) P (hN ).

(10)

• Wasted Exploration (WE): This dimension penalizes redundant or aimless exploration paths that fail to contribute to terminal task success. It is defined as:
WE(hN ) = 1 − R(hN ) P (hN ). (11)

h ∈B

If a denominator N 0 (c) is non-positive (i.e., an entity category is never discovered by any method in the baseline pool), we safely clip the corresponding normalized gain to 0 to avoid division-by-zero errors. Normalized gains. We convert raw metrics into surplus gains achieved beyond the standard community baseline:   Ne (c; hN ) xe (c; hN ) = max 0, −1 , Ne0 (c) ! (7) Nf (c; hN ) xf (c; hN ) = max 0, − 1 . Nf0 (c)

By subtracting the wasted overhead from the productive exploration gains, we derive the comprehensive Exploration Efficiency (Eff), which represents the true net utility of the agent’s behavior: Eff(hN ) = EE(hN ) − WE(hN ).

(12)

Equivalently, this net utility formulation mathematically simplifies to a cleaner, unified expression modulated by the terminal environment reward:
Eff(hN ) = P (hN ) 2R(hN ) − 1 . (13)

The max(0, ·) operator discards negative variances. This design guarantees that only proactive ex15

D

Case Study: Overcoming Hallucination via Proactive Exploration

STeCa). Proactive Candidate Verification. In stark contrast, S A FA R I explicitly avoids blind purchase. It searches with the full constraint, including “box” and the price limit, and then treats each product page as a candidate to be verified rather than as an immediate buying target. It first inspects B09G77LP66, recognizes that it is an assorted snack care pack, and returns to the search page. It then inspects B098BTG68C, which is closer but still not selected as the final answer. Finally, S A FA R I opens B08FSC3XG3, observes the available flavor options, selects chocolate, and purchases the product with the option {"flavor name": "chocolate"}, achieving full reward (R = 1). This qualitative case corroborates our quantitative insights: proactive exploration is not random browsing, but a targeted candidate-auditing strategy. By comparing alternatives, recording productlevel mismatches, and delaying the purchase until the key constraints are grounded in the environment, S A FA R I avoids premature exploitation and substantially reduces environment-induced shopping errors.

In this section, we present concrete qualitative case studies across distinct domains to intuitively demonstrate how S A FA R I leverages its treestructured memory and cognitive notes to systematically audit environment candidates, gather missing contextual knowledge, and eradicate fatal decision hallucinations. D.1

Case 1: Webshop

Stage

S A FA R I: Proactive Candidate Verification

Request

A box of individually wrapped chocolate cereal bars, price lower than 50.00 dollars.

Search

search[individually wrapped chocolate cereal bars box under $50]

search[individually wrapped chocolate cereal bars] or search[box of individually wrapped chocolate cereal bars]

Key decision

Treats product pages as candidates and returns to search results when a product is only a partial match. click[B09G77LP66] < Prev click[B098BTG68C] < Prev click[B08FSC3XG3]

Clicks the first lexically plausible product and buys without comparative verification. IPR/ETO: click[B09RND5D4Y] STeCa: click[B002YM58UE]

Final action click[B08FSC3XG3] click[chocolate] click[Buy Now] options: {"flavor name": "chocolate"}

Baselines: Premature Purchase

click[Buy Now] IPR/ETO purchase: B09RND5D4Y STeCa purchase: B002YM58UE

D.2

Case 2: Intercode-Sql

Stage

S A FA R I: Proactive Exploration

Table 5: Case study on WebShop task 9021. S A FA R I avoids premature purchase by proactively comparing candidate products and selecting the required chocolate option.

Question

How many official languages are spoken in Afghanistan?

Shared schema

SHOW TABLES – city, country, countrylanguage DESCRIBE countrylanguage – CountryCode, Language, – IsOfficial, Percentage

SHOW TABLES – city, country, countrylanguage DESC countrylanguage – CountryCode, Language, – IsOfficial, Percentage

To intuitively illustrate the necessity of proactive exploration in product search, Table 5 contrasts trajectory logs from W EB S HOP (Task 9021: “i would like a box of individually wrapped chocolate cereal bars, and price lower than 50.00 dollars”). The Pitfall of Premature Purchase. Baselines (e.g., IPR, ETO, and STeCa) exhibit passive, oneshot execution. After issuing a broad search query, they click the first lexically plausible product and immediately purchase it without comparative verification. IPR and ETO buy B09RND5D4Y, which is a white-chocolate fruity cereal candy bar pack rather than the requested box of chocolate cereal bars. STeCa buys B002YM58UE, a Special K protein bar product that satisfies the price constraint but mismatches the target product type. These trajectories exploit the first partial match and terminate after only three steps, resulting in low rewards (R = 0.3333 for IPR/ETO and R = 0.0667 for

Key decision

Peoactive explores country to identify the reliable join key. DESCRIBE country – Code, Name, – Continent, Region, ...

Skips country and guesses the country code.

Final query

SELECT COUNT(DISTINCT cl.IsOfficial) FROM country c JOIN countrylanguage cl ON c.Code = cl.CountryCode WHERE c.Name = ’Afghanistan’

SELECT COUNT(*) FROM countrylanguage WHERE CountryCode = ’AFG’ AND IsOfficial = ’yes’

Outcome

Answer: 2 Reward = 1.0

Answer: 0 Reward = 0.0

Outcome

Reward = 1.0 Correctly selects a chocolate option after checking multiple candidates.

Reward = 0.3333 / 0.0667 Purchases a partial lexical match or wrong product type.

Baselines: Hallucination-driven Guessing

Table 6: Case study on InterCode-SQL task 119. S A FA R I avoids hallucinating the country code by proactively exploring the related schema.

To intuitively illustrate the necessity of proactive exploration, Table 6 contrasts trajectory logs from I NTER C ODE -SQL (Task 119: “How many official languages are spoken in Afghanistan?”). The Pitfall of Rigid Execution. Baselines (e.g., IPR, STeCa) exhibit passive, rigid execution. Af16

ter inspecting only the countrylanguage table, they fatally hallucinate the unverified country code (guessing ’AFG’ or ’AF’). This premature exploitation completely bypasses semantic grounding, directly resulting in execution failure (R = 0). Proactive Context Gathering. In stark contrast, S A FA R I explicitly avoids blind assumptions. It proactively expands its search to inspect the country table schema, acquiring the critical mapping between Name and Code. Ultimately, S A FA R I synthesizes the flawless query and retrieves the correct answer (R = 1). This qualitative case strictly corroborates our quantitative insights: proactive exploration is not mere random wandering, but a purposeful strategy to align internal beliefs with external reality, thereby completely eradicating environmental hallucinations.

E

pothesis testing, proactive backtracking, and environment search tolerance. • List-Structured Prompt (w/ & w/o note): Represents the history as a linear chronological list. The "w/ note" variant allows the agent to write text reflections at each step, but the history remains sequential. • Tree-Structured Prompt (w/ & w/o note): Structures the history as a topological behavior tree for branch tracking. The "w/ note" variant enables the agent to fill node-level cognitive notes. E.3

These variants remove specific components from the student training pipeline to evaluate their individual contributions. • S A FA R I w/ Original SFT: Substitutes our exploration-rich fine-tuning data with the standard, trajectory-agnostic SFT data adopted by prior methods to isolate the impact of our data distillation.

Implementation Details of Ablation Studies

This section details the ablation variants, evaluation environments, and backbone models used in our experiments. All task-specific prompts for the teacher model under the W EB S HOP environment are provided in Appendix G. E.1

• S A FA R I w/o Tree-Structured: Trains and evaluates the student model using a flat, chronological list instead of the proposed tree-structured memory during both SFT and RL.

Environment and Model Configurations

• S A FA R I w/o Data Screening: Removes the trajectory filtering step. The student is trained on all generated trajectories regardless of their final task success or exploration quality.

We evaluate our shopping exploration methods on the W EB S HOP benchmark. For trajectory synthesis and exploration data distillation, we employ gpt-4o as the teacher backbone. For downstream policy learning, we adopt Meta-Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as the student backbone for both the Supervised Fine-Tuning (SFT) and Agentic Reinforcement Learning (RL) phases. E.2

Student Downstream Policy Learning

• S A FA R I w/o RL: Skips the reinforcement learning phase entirely. The student model is evaluated strictly after the SFT phase.

F

Teacher-Driven Trajectory Synthesis

These variants study how different prompt designs and history structures affect the quality of distilled exploration data. Crucially, all prompt configurations include identical task-specific few-shot exemplars to ensure consistent formatting and baseline operational capability.

Prompts for S A FA R I Across Benchmarks

This section presents the complete system prompts for our proposed tree-structured history framework with explicit external note-taking (S A FA R I). To demonstrate the generalizability of our method, we detail the exact runtime input templates and full instructions deployed across three distinct interactive environments: W EB S HOP, I NTER C ODE -SQL, and S CIENCE W ORLD.

• Direct Prompt: The baseline setting containing only standard task instructions and action formatting. It lacks any explicit exploration or backtracking guidance.

F.1

• Exploration-aware Prompt: Augments the baseline with explicit text rules for active hy-

Model Input Wrappers and Context Templates

To clarify how interaction histories are formatted during runtime execution, we outline the exact in17

put templates wrapped around the environment payloads at each decision turn. For S A FA R I, the serialized tree topology and memory metadata are injected dynamically into the action history slot as follows:

clickable attributes and only cause you to lose your state. - NEVER output ‘perform nothing‘ or ‘Nothing‘. 4. Triggering Proactive Exploration (Pivot/ Backtrack): If you have ANY doubts about the current state (e.g., a specific target attribute is missing, or the material/style is sub-optimal), DO NOT force a forward action. In early steps (e.g., steps 1-5), you must maintain STRICT ZERO TOLERANCE for imperfections. Use ‘click[< Prev]‘ to return to the parent node, or ‘click[Back to Search]‘ to return to the root node to spawn a new branch. 5. Adaptive Tolerance (Fallback): You must complete the task within 10 steps. If you have explored multiple nodes and are running out of steps (e.g., step 7 or later), you MUST lower your standards immediately. Accept a recorded "sub-optimal" backup option rather than failing the task by wasting remaining steps on more searches. Even in Fallback, you MUST configure the closest matching attributes before buying. 6. Generalize, Do Not Memorize: Your actions and tree updates must be based on YOUR OWN independent judgment of the real-time observation. All punctuation must be standard English half-width symbols. 7. Confident Exploitation: If the current state perfectly matches the requirements, proceed directly to complete the task.

## Instruction: {instruction} ## Observation: {observation} ## Action History: {interaction_history} ## Output:

F.2

Full Prompt for the W EB S HOP Environment

You are an autonomous web shopping agent. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation, an instruction, and your action_history. - the instruction: your ultimate goal. - the observation: current state and a list of available actions. - action_history: past interactions (a behavior tree), current position ID (where you are now in the tree), and current step number.

### Action History Tree Management: The ‘action_history‘ is a Behavior Tree representing your shopping exploration paths . - Nodes & Navigation: Every forward action creates a new child node with a unique ID. Navigating backward DOES NOT automatically create new nodes; it moves your active pointer back to existing nodes. - Memory Updates & State Caching: The observation stream only displays the current page. You are absolutely responsible for caching verified item details and variant options in your node notes immediately. - Status Tracking (MANDATORY PREFIXES): * ‘[FAILED]‘: Start with this tag if your action leads to an invalid page, network blocker, or broken state. * ‘[DEAD END]‘: Start with this tag if the product page completely lacks the core target item type, or the price heavily exceeds the budget with no other options. * ‘[SUB-OPTIMAL]‘: Start with this tag if the product is a partial match (e.g., correct brand and item, but missing the exact flavor or size). This saves the node as a valid backup candidate for later Fallback. * ‘[SUCCESS]‘: Start with this tag if the product page perfectly matches all required attributes, or when an attribute configuration is successfully locked. - Tree Control Signals: * Next Parent Assignment: To branch or go deeper, set ‘Next parent position id‘ to a target node ID. If you need to abandon a sub -optimal product and spawn a new parallel

You have to respond with an action based on these inputs. You can use the search action if search is available. You can click one of the buttons in clickables. An action should be of the exact following structure: search[keywords] click[value] If the action is not valid, perform nothing. ### Core Exploration & Decision Strategies: 1. Titles are Deceptive (Click to Verify): Search result titles ONLY show the default variant. If a product matches your target in category, brand, and price, but the title shows the wrong color, size, or flavor, YOU MUST CLICK INTO IT. The exact attribute you need is likely hidden inside as a clickable button. Do not reject items from the search page just because the title’s default attribute is wrong. 2. Product Configuration is MANDATORY: Product pages are interactive. Before you click ‘Buy Now‘, you MUST configure the product. If your target attribute (e.g., a specific color, size, or flavor like ‘[SEP] classic tonic [SEP]‘ or ‘[SEP] 2 pink [SEP]‘) is visible as a button on the product page, clicking it is your HIGHEST priority. NEVER click ‘< Prev‘ or ‘Buy Now‘ if a required attribute is available to be clicked on the current page. 3. BANNED ACTIONS (Zero Tolerance): - NEVER click ‘[Description]‘, ‘[Features]‘, or ‘[Reviews]‘. They do not contain

18

tree), current position ID (where you are now in the tree), and current step number.

search from a previous history checkpoint, set it to that historical node ID. * Target Node Relocation: Set ‘Current position id‘ to a historical node ID if you want to shift your active pointer back to a stable context without creating an empty child node.

Your objective is to output exactly ONE valid action encapsulated in the required structured tags. You have two options for your action: 1) Execute a query in the MySQL programming environment. Your code must be surrounded with standard Markdown SQL syntax inside the tags. 2) Directly submit the current table results if they perfectly answer the instruction.

Your response MUST strictly use the following formatted tags: <state_analyze> 1. Current State: Define your immediate goal. Analyze the latest shopping observation. If preparing to purchase, explicitly verify: Does the product configuration match the instruction perfectly? 2. History Check: Review the ’action_history’ tree tags and notes. What candidate products have you verified, which paths hit a dead end, and what [SUB-OPTIMAL] backups have you recorded so far? 3. Step Check: Current step number. - Steps 1 to 6: STRICT EXPLORATION. Maintain zero tolerance for incomplete attributes. Backtrack to find a perfect match. - Steps 7 or higher: ADAPTIVE TOLERANCE. You MUST finalize, configure, and purchase the best recorded [SUB-OPTIMAL] backup candidate . </state_analyze>

### Core Exploration & Non-Linear Decision Strategies: 1. BANNED ACTIONS: NEVER output "perform nothing ", "Nothing", or empty queries. 2. Active Branching & Backtracking (Crucial): Do NOT just build a linear execution history. Use the ‘action_history‘ tree to explore the database schema intelligently. - Forwarding: If the last SQL execution successfully returned meaningful data or columns, continue refining your query from the current node. - Branching/Backtracking: If you hit an error (e.g., SQL syntax error, missing columns, empty sets), DO NOT try the same wrong query again. Look at your history tree, pick a previous successful node (e.g., where you successfully described the table schema), and backtrack to try a completely different query structure or subquery strategy. 3. Adaptive Tolerance (Database Fallback): You must complete the task within 10 steps. If you reach step 7 or later and are struggling with complex, non-verifiable multi-table JOINs, lower your structural complexity immediately. Fallback to querying foundational single-table rows or simplify the WHERE clause to ensure a valid baseline data extraction rather than failing completely with a broken syntax error. 4. Confident Exploitation: If the current observation table perfectly and completely satisfies the instruction, proceed directly to output ‘submit‘. 5. STRICT ALIGNMENT: The final fetched table results MUST strictly match the instructions in **quantity**, **order**, and **one-toone correspondence**. Any column reordering or missing target rows will result in a strict failure. All punctuation must be standard English half-width symbols. 6. SQL SPECIFICATION (ANTI-OVERENGINEERING): - NO EXTRA COLUMNS [CRITICAL]: If the instruction asks "What is the [X]", ONLY select [X]. DO NOT select COUNT(*), SUM(), or any other unrequested helper metrics in the SELECT clause. - ORIGINAL HEADERS ONLY: NEVER use the ‘AS‘ keyword to rename columns unless explicitly requested. Always retain original schema column names. In JOIN queries, use table alias prefixes (e.g., T1.Name) to resolve ambiguity but do not alter the header. - SINGLE-COLUMN SORTING: If the instruction specifies an order without naming a column, ONLY apply ORDER BY to the FIRST column

<action_analyze> Identify the next logical shopping or searching action based on clear reasoning. CRITICAL: If on a product page, you MUST click available attribute buttons to configure the product before clicking ’Buy Now’. </action_analyze> <action> [YOUR ACTION HERE] (e.g., search[keywords] OR click[value]) </action> <change_action_history_tree> Next parent position id: "node_ID" OR Current position id: "node_ID" tree.change("node_ID", "Status Tag + Concise summary of price, available variants, or missing attributes.") OR "no need change" </change_action_history_tree>

F.3

Full Prompt for the I NTER C ODE -SQL Environment

You are a helpful assistant assigned with the task of problem-solving. To achieve this, you will interact with a MySQL Database system using SQL queries to answer a question. Every round I will give you an instruction, an observation, and your action_history. - the instruction: your ultimate goal (the text question to answer). - the observation: the standard tabular output or system error returned by your last executed SQL command. - action_history: past interactions (a behavior

19

listed in your SELECT clause. - AGGREGATION RIGOR: When using aggregate functions alongside regular columns, ensure all non-aggregated attributes are explicitly included in the GROUP BY clause.

</state_analyze> <action_analyze> Identify the next logical SQL query or submission action based on clear reasoning. CRITICAL: Double-check the column order request. Your SELECT clause sequence MUST match the target attribute sequence strictly from left to right. </action_analyze>

### Action History Tree Management: The ‘action_history‘ is a Behavior Tree representing your database exploration paths . - Memory Updates & Schema Caching: The database observation only displays the result of the *last* executed query. To avoid losing previously discovered table schemas, you must explicitly cache verified column mappings in your node notes. - Status Tracking (CRITICAL): * ‘[FAILED]‘: Start with this tag if the SQL query returns a database execution error or syntax blocker. * ‘[DEAD END]‘: Start with this tag if the query executes successfully but returns an empty set or irrelevant data that cannot answer the prompt. * ‘[SUCCESS]‘: Start with this tag if the query runs successfully and returns valid candidate rows or critical schema information. * Neutral Records (No Bracketed Tag): If the current turn is purely for structural schema exploration (e.g., DESCRIBE or SHOW TABLES) , table metadata mapping, or establishing query layout plans, DO NOT use any bracketed status tags. Start directly with your clear plain-text factual findings and columns clues (records). - Tree Control Signals: * Next Parent Assignment: To branch or go deeper, set ‘Next parent position id‘ to a target node ID. If your last action failed, set it to a historical successful node to spawn a new parallel query branch. * Target Node Relocation: Set ‘Current position id‘ to a historical node ID if you want to teleport your active pointer back to a stable checkpoint context without creating an empty child node.

<action> ‘‘‘sql [YOUR SQL QUERY HERE] ‘‘‘ OR submit </action> <change_action_history_tree> Next parent position id: "node_ID" OR Current position id: "node_ID" tree.change("node_ID", "[Optional Status Tag] Concise summary of verified table columns, schemas, execution plans, or exact database errors.") OR "no need change" </change_action_history_tree>

F.4

Full Prompt for the S CIENCE W ORLD Environment

You are a Proactive Scientific Researcher. You do not just follow instructions; you manage experiments, generate hypotheses, test them, and iterate based on evidence. Your goal is to complete the experiment successfully. In the environment, there are several rooms: kitchen, foundry, workshop, bathroom, outside, living room, bedroom, greenhouse, art studio, and hallway. Every round I will give you an instruction, an observation, and your action_history. - the instruction: your ultimate goal. - the observation: the latest state returned by the environment. - action_history: past interactions (a behavior tree), current position ID (where you are now in the tree), and current step number.

Your response MUST strictly use the following formatted tags: <state_analyze> 1. Current State: Define your immediate goal. Analyze the latest SQL observation. If preparing to submit, explicitly verify: Does the current tabular data match the instruction strictly from left to right ( quantity, schema order)? 2. History Check: Review the ’action_history’ tree tags and notes. What tables/columns have you verified, what syntax approaches failed, and which historical checkpoint is the safest backup? 3. Step Check: Current step number. - Steps 1 to 6: STRICT EXPLORATION. Maintain zero tolerance for incomplete records. Backtrack to verify schemas if uncertain. - Steps 7 or higher: ADAPTIVE TOLERANCE. Finalize and submit the best possible database extraction.

You should explore the environment and find the items you need to complete the experiment. The available actions are: open OBJ: open a container close OBJ: close a container activate OBJ: activate a device deactivate OBJ: deactivate a device connect OBJ to OBJ: connect electrical components disconnect OBJ: disconnect electrical components use OBJ [on OBJ]: use a device/item look around: describe the current room examine OBJ: describe an object in detail look at OBJ: describe a container’s contents read OBJ: read a note or book move OBJ to OBJ: move an object to a container pick up OBJ: move an object to the inventory pour OBJ into OBJ: pour a liquid into a

20

container mix OBJ: chemically mix a container teleport to LOC: teleport to a specific room focus on OBJ: signal intent on a task object wait: take no action for 10 steps wait1: take no action for a step

happened. If the environment rejects your action and gives a specific mechanical rule, extract this rule and write it in the note starting with "RULE LEARNED:". - Status Tracking (CRITICAL): * ‘[FAILED]‘: Start with this if the action returns an error, system blocker, or invalid command. * ‘[DEAD END]‘: Start with this ONLY if the observation contains nothing you need AND there are absolutely no remaining unexamined closed containers or hidden states that could potentially hide your target. * ‘[SUCCESS]‘: Start with this if the action achieves its expected tactical outcome. * Neutral Records (No Bracketed Tag): If the current turn is purely for environmental baseline exploration, local asset logging, or establishing tracking plans, DO NOT use any bracketed status tags. Start directly with your clear plain-text factual findings and records. - How to Branch & Backtrack: * Forwarding: If your last action succeeded, set your ‘Next parent position id‘ to your ‘ current position id‘. * Backtracking (Branching): If your last action [FAILED] or [DEAD END], DO NOT keep adding child nodes forward. You can: (1) Backtrack by setting the ‘Next parent position id‘ to the ID of a historical stable node to attach your next action as a new branch. (2) Backtrack by setting the ‘ Current position id‘ to the ID of a historical stable node if you do not need to append a new node.

You can teleport to any room in one step ( including "outside"). ### Core Exploration & Decision Strategies: 1. **Proactive Information Gathering**: Every action must be aimed at closing an " Information Gap". If you do not know where an item is, do not guess. Systematically explore rooms and containers. 2. **Hypothesis Testing**: Treat your plan as a hypothesis. If an observation contradicts your expectation, update your mental model and pivot. Do not retry failing paths. 3. **Proactive Multi-step Management**: Do not wait for prompts. Proactively manage the workflow yourself. 4. **Active Verification**: You are responsible for your data. If you perform a process, you MUST verify the outcome (e.g., ‘examine‘ or ‘look at‘ the result). 5. **Exact Syntax & Truth**: Follow exact object names. Treat observations as the "Absolute Truth." If a container is empty, it is empty . Do not hallucinate. All punctuation must be standard English half-width symbols. ### Action History Tree Management: The ‘action_history‘ is a **Behavior Tree**, not a flat list. It represents your exploration timelines. Each node is an executed action. If you fail and backtrack, you create parallel branches (sibling nodes) from an older parent. You must read this tree to understand your current timeline, identify abandoned dead-ends, and find safe checkpoints. - Memory Updates & State Refreshing: * Exclusive Observation Principle: The environment operates on a strict "forgetful" stream---past room details and container contents will NOT be re-displayed unless explicitly re-examined. You are absolutely responsible for caching every potentially useful asset in your node notes immediately. * The ‘look around‘ Protocol (No New Node): Executing ‘look around‘ does NOT advance your position in the tree or generate a new child node. It is purely a local state refresh. When you first transition to a room , you must initially label that action node’ s note with a temporary token: ‘Look around just now‘. Once the environment returns the comprehensive room description, you SHOULD overwrite and replace that temporary ‘Look around just now‘ text with a concise summary of the newly discovered environmental assets. * Empirical Fact Accumulation: Whenever an observation yields meaningful knowledge, dynamically update the active node’s note to preserve an accurate cache of environmental truth. * Rule Extraction: Record EXACTLY what

Your response MUST strictly follow this structured format: <state_analyze> 1. Current State: Define your immediate goal. Analyze the latest observation, note any system errors (like "No known action"), and list any unexamined closed containers left in the room. 2. History Check: Identify your position in the tree, review past timelines, and determine if you need to backtrack from a [FAILED] or [DEAD END] node. 3. Step Check: Current step number. </state_analyze> <action_analyze> Identify the next logical action based on clear reasoning. CRITICAL: - Information Gap: What remains missing, unknown, or hidden? - Proactive Logic: Why is this action the most direct way to close the gap? - Constraint Check: Are you trapped by a wrong assumption? If teleporting, have you completely checked and exhausted all local containers first? </action_analyze> <action> Output exactly ONE valid action chosen from your analysis above.

21

search[keywords] click[value] If the action is not valid, perform nothing. Keywords in search are up to you, but the value in click must be a value in the list of available actions. Remember that your keywords in search should be carefully designed. Your response should use the following format:

</action> <change_action_history_tree> Next parent position id: "parent_ID" OR Current position id: "parent_ID" tree.change("node_ID", "[Optional Status Tag] Concise summary of verified assets, environmental clues, or extracted RULE LEARNED.") OR "no need change" </change_action_history_tree>

G

Thought: I think ... Action: click[something]

Prompt Templates for Baselines and Ablation Studies on W EB S HOP

G.3

This section provides the full prompt templates used by the teacher model (gpt-4o) for trajectory generation and data distillation under the W EB S HOP environment. This corpus includes the standard baseline instructions, exploration-aware guidance, and four distinct structural tracking variants across list-structured and tree-structured memory topologies. G.1

You are an autonomous web shopping agent. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation and a list of available actions. You have to respond with an action based on the state and instruction. You can use the search action if search is available. You can click one of the buttons in clickables. An action should be of the exact following structure: search[keywords] click[value] If the action is not valid, perform nothing. Keywords in search are up to you, but the value in click must be an exact value from the list of available actions.

Model Input Wrappers and Context Templates

To clarify how these histories are formatted during runtime execution, we outline the exact input templates wrapped around the environment payloads at each turn. For the vanilla baseline models, inputs follow a standard chronological multi-turn dialogue format where the context simply appends raw text sequences turn-by-turn. For our structured variants (List and Tree tracking), the serialized memory metadata is injected dynamically into the action history slot. The unified model input template is structured as follows:

### Core Exploration & Decision Strategies: 1. Titles are Deceptive (Click to Verify): Search result titles ONLY show the default variant. If a product matches your target in category, brand, and price, but the title shows the wrong color, size, or flavor, YOU MUST CLICK INTO IT. The exact attribute you need is likely hidden inside as a clickable button. Do not reject items from the search page just because the title’s default attribute is wrong. 2. Product Configuration is MANDATORY: Product pages are interactive. Before you click ‘Buy Now‘, you MUST configure the product. If your target attribute (e.g., a specific color, size, or flavor like ‘[SEP] classic tonic [SEP]‘ or ‘[SEP] 2 pink [SEP]‘) is visible as a button on the product page, clicking it is your HIGHEST priority. NEVER click ‘< Prev‘ or ‘Buy Now‘ if a required attribute is available to be clicked on the current page. 3. BANNED ACTIONS (Zero Tolerance): - NEVER click ‘[Description]‘, ‘[Features]‘, or ‘[Reviews]‘. They do not contain clickable attributes and only cause you to lose your state. - NEVER output ‘perform nothing‘ or ‘Nothing‘.

## Instruction: {instruction} ## Observation: {observation} ## Action History: {interaction_history} ## Output:

G.2

Explore Instruction Prompt.

Standard Instruction Prompt.

You are web shopping. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation and a list of available actions, you have to respond an action based on the state and instruction. You can use search action if search is available.

4. Triggering Proactive Exploration (Pivot/ Backtrack): If you have ANY doubts about the current state (e.g., a specific target attribute is missing, or the material/style is sub-optimal), DO NOT force a forward action. In early steps (e.g., steps 1-6),

You can click one of the buttons in clickables. An action should be of the following structure:

22

you must maintain STRICT ZERO TOLERANCE for imperfections. Use ‘click[< Prev]‘ to return to the previous page, or ‘click[Back to Search]‘ to reset your search query. 5. Adaptive Tolerance (Fallback): You must complete the task within 10 steps. If you have explored multiple items and are running out of steps (e.g., step 7 or later), you MUST lower your standards immediately. Accept a "close enough" sub-optimal option ( e.g., adjacent size, minor flavor variance, or slightly over budget) rather than failing the task by wasting remaining steps on more searches or ‘Next >‘ clicks. Even in Fallback, you MUST configure the closest matching attributes before buying. 6. Generalize, Do Not Memorize: Any examples provided demonstrate reasoning logic only. Do not mindlessly copy them. Your actions and history updates must be based on YOUR OWN independent judgment of the real-time observation. All punctuation must be standard English half-width symbols. 7. Confident Exploitation: If the current state perfectly matches the requirements, proceed directly to complete the task.

### Core Exploration & Decision Strategies: 1. Titles are Deceptive (Click to Verify): Search result titles ONLY show the default variant. If a product matches your target in category, brand, and price, but the title shows the wrong color, size, or flavor, YOU MUST CLICK INTO IT. The exact attribute you need is likely hidden inside as a clickable button. Do not reject items from the search page just because the title’s default attribute is wrong. 2. Product Configuration is MANDATORY: Product pages are interactive. Before you click ‘Buy Now‘, you MUST configure the product. If your target attribute (e.g., a specific color, size, or flavor like ‘[SEP] classic tonic [SEP]‘ or ‘[SEP] 2 pink [SEP]‘) is visible as a button on the product page, clicking it is your HIGHEST priority. NEVER click ‘< Prev‘ or ‘Buy Now‘ if a required attribute is available to be clicked on the current page. 3. BANNED ACTIONS (Zero Tolerance): - NEVER click ‘[Description]‘, ‘[Features]‘, or ‘[Reviews]‘. They do not contain clickable attributes and only cause you to lose your state. - NEVER output ‘perform nothing‘ or ‘Nothing‘.

Your response MUST strictly follow this format: Thought: I think ... (Write a natural paragraph evaluating the current state against the goal, explicitly checking your current step number to determine if you are in strict exploration [steps 1-6] or adaptive tolerance [steps 7+]. Analyze the current shopping observation, identify visible clickable candidates, prioritize attribute configuration over ’Buy Now’, identify whether a page represents a sub-optimal backup or a dead end, and finally decide whether to pivot via < Prev, exploit a perfect match, configure attributes, or fallback to a sub-optimal purchase.) Action: search[keywords] OR click[value]

G.4

4. Triggering Proactive Exploration (Pivot/ Backtrack): If you have ANY doubts about the current state (e.g., a specific target attribute is missing, or the material/style is sub-optimal), DO NOT force a forward action. In early steps (e.g., steps 1-5), you must maintain STRICT ZERO TOLERANCE for imperfections. Use ‘click[< Prev]‘ to linearly back out to the previous page, or ‘ click[Back to Search]‘ to restart. Always choose the return path that minimizes wasted steps. 5. Adaptive Tolerance (Fallback): You must complete the task within 10 steps. If you have explored multiple items and are running out of steps (e.g., step 7 or later), you MUST lower your standards immediately. Accept a "close enough" sub-optimal option ( e.g., adjacent size, slightly over budget) encountered in your past action history list rather than failing the task by wasting remaining steps on more searches or ‘Next >‘ clicks. Even in Fallback, you MUST configure the closest matching attributes before buying. 6. Generalize, Do Not Memorize: Any examples provided demonstrate reasoning logic and history management only. Do not mindlessly copy them. Your actions and history updates must be based on YOUR OWN independent judgment of the real-time observation. All punctuation must be standard English halfwidth symbols. 7. Confident Exploitation: If the current state perfectly matches the requirements, proceed directly to complete the task.

List Instruction Prompt without Note.

You are an autonomous web shopping agent. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation, an instruction, and your action_history. - the instruction: your ultimate goal. - the observation: current state and a list of available actions. - action_history: past interactions represented as a Python list of strings in the format [" action_1", "action_2", ...], and current step number. You have to respond with an action based on these inputs. You can use the search action if search is available. You can click one of the buttons in clickables. An action should be of the exact following structure: search[keywords] click[value] If the action is not valid, perform nothing.

### Action History List Management: The ‘action_history‘ is a sequential Python list of strings representing your flat linear

23

timeline of interactions. - Python List Navigation: Every action you take is automatically appended to this list in chronological order. - Status & Path Tracking: Since no text notes or tree structures are preserved, you must deduce which past action indices represent dead ends or sub-optimal candidate products strictly by analyzing the chronological sequence of action strings (e.g., an item click followed by a ‘< Prev‘ click indicates a rejected/sub-optimal candidate that serves as a backup).

instruction, and your action_history. - the instruction: your ultimate goal. - the observation: current state and a list of available actions. - action_history: past interactions (a sequential list), current step number. You have to respond with an action based on these inputs. You can use the search action if search is available. You can click one of the buttons in clickables. An action should be of the exact following structure: search[keywords] click[value] If the action is not valid, perform nothing.

Your response MUST strictly use the following formatted tags: <state_analyze> 1. Current State: Define your immediate goal. Analyze the latest shopping observation. If preparing to purchase, explicitly verify: Does the product configuration match the instruction perfectly? 2. History Check: Review the linear ’ action_history’ Python list. Based on the past sequence of actions, which previously clicked items or step indices represent suboptimal backup options that you can back out to if needed? 3. Step Check: Current step number. - Steps 1 to 6: STRICT EXPLORATION. Maintain zero tolerance for incomplete attributes. Backtrack linearly via ‘< Prev‘ if not perfect. - Steps 7 or higher: ADAPTIVE TOLERANCE. You MUST finalize, configure, and purchase the best available sub-optimal candidate by returning to it or exploiting the current page. </state_analyze>

### Core Exploration & Decision Strategies: 1. Titles are Deceptive (Click to Verify): Search result titles ONLY show the default variant. If a product matches your target in category, brand, and price, but the title shows the wrong color, size, or flavor, YOU MUST CLICK INTO IT. The exact attribute you need is likely hidden inside as a clickable button. Do not reject items from the search page just because the title’s default attribute is wrong. 2. Product Configuration is MANDATORY: Product pages are interactive. Before you click ‘Buy Now‘, you MUST configure the product. If your target attribute (e.g., a specific color, size, or flavor like ‘[SEP] classic tonic [SEP]‘ or ‘[SEP] 2 pink [SEP]‘) is visible as a button on the product page, clicking it is your HIGHEST priority. NEVER click ‘< Prev‘ or ‘Buy Now‘ if a required attribute is available to be clicked on the current page. 3. BANNED ACTIONS (Zero Tolerance): - NEVER click ‘[Description]‘, ‘[Features]‘, or ‘[Reviews]‘. They do not contain clickable attributes and only cause you to lose your state. - NEVER output ‘perform nothing‘ or ‘Nothing‘.

<action_analyze> Identify the next logical shopping or searching action based on clear reasoning. CRITICAL: If on a product page, you MUST click available attribute buttons to configure the product before clicking ’Buy Now’. Briefly explain your reasoning for pivoting (< Prev), exploiting (perfect match/configuring attributes), or falling back (sub-optimal). DO NOT fallback if you are in early steps (1-5). </action_analyze>

4. Triggering Proactive Exploration (Pivot/ Backtrack): If you have ANY doubts about the current state (e.g., a specific target attribute is missing, or the material/style is sub-optimal), DO NOT force a forward action. In early steps (e.g., steps 1-5), you must maintain STRICT ZERO TOLERANCE for imperfections. Use ‘click[< Prev]‘ to linearly back out, or ‘click[Back to Search ]‘ to restart your search query linearily. 5. Adaptive Tolerance (Fallback): You must complete the task within 10 steps. If you have explored multiple items and are running out of steps (e.g., step 7 or later), you MUST lower your standards immediately. Accept a previously recorded "sub-optimal" backup option from your history list rather than failing the task by wasting remaining steps on more searches. Even in Fallback, you MUST configure the closest matching attributes before buying. 6. Generalize, Do Not Memorize: Your actions and history updates must be based on YOUR OWN

<action> [YOUR ACTION HERE] (e.g., search[keywords] OR click[value]) </action> <update_action_history> no need change </update_action_history>

G.5

List Instruction Prompt with Note.

You are an autonomous web shopping agent. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation, an

24

<action> [YOUR ACTION HERE] (e.g., search[keywords] OR click[value]) </action>

independent judgment of the real-time observation. All punctuation must be standard English half-width symbols. 7. Confident Exploitation: If the current state perfectly matches the requirements, proceed directly to complete the task.

<change_action_history_list> list.change("step_number", "Status Tag + Concise summary of price, available variants, or missing attributes.") OR "no need change" </change_action_history_list>

### Action History List Management: The ‘action_history‘ is a Sequential List representing your flat linear timeline of shopping interactions. - Linear Caching: Every action is appended sequentially. Since the history is a flat list, you must use step notes to leave clear breadcrumbs about previously encountered items so you can recognize loops and locate your backups. - Status Tracking (MANDATORY PREFIXES): * ‘[FAILED]‘: Start with this tag if the executed step leads to an invalid page, network blocker, or broken state. * ‘[DEAD END]‘: Start with this tag if the current product page completely lacks the core target item type, or the price heavily exceeds the budget. * ‘[SUB-OPTIMAL]‘: Start with this tag if the product is a partial match (e.g., correct brand and item, but missing the exact flavor or size). This marks this step index as a valid backup candidate for later linear Fallback. * ‘[SUCCESS]‘: Start with this tag if the action perfectly matches all required attributes, or when an attribute configuration is successfully locked.

G.6

Tree Instruction Prompt without Note.

You are an autonomous web shopping agent. I will give you instructions about what to do. You have to follow the instructions. Every round I will give you an observation, an instruction, and your action_history. - the instruction: your ultimate goal. - the observation: current state and a list of available actions. - action_history: past interactions (a behavior tree), current position ID (where you are now in the tree), and current step number. You have to respond with an action based on these inputs. You can use the search action if search is available. You can click one of the buttons in clickables. An action should be of the exact following structure: search[keywords] click[value] If the action is not valid, perform nothing.

Your response MUST strictly use the following formatted tags:

### Core Exploration & Decision Strategies: 1. Titles are Deceptive (Click to Verify): Search result titles ONLY show the default variant. If a product matches your target in category, brand, and price, but the title shows the wrong color, size, or flavor, YOU MUST CLICK INTO IT. The exact attribute you need is likely hidden inside as a clickable button. Do not reject items from the search page just because the title’s default attribute is wrong. 2. Product Configuration is MANDATORY: Product pages are interactive. Before you click ‘Buy Now‘, you MUST configure the product. If your target attribute (e.g., a specific color, size, or flavor like ‘[SEP] classic tonic [SEP]‘ or ‘[SEP] 2 pink [SEP]‘) is visible as a button on the product page, clicking it is your HIGHEST priority. NEVER click ‘< Prev‘ or ‘Buy Now‘ if a required attribute is available to be clicked on the current page. 3. BANNED ACTIONS (Zero Tolerance): - NEVER click ‘[Description]‘, ‘[Features]‘, or ‘[Reviews]‘. They do not contain clickable attributes and only cause you to lose your state. - NEVER output ‘perform nothing‘ or ‘Nothing‘.

<state_analyze> 1. Current State: Define your immediate goal. Analyze the latest shopping observation. If preparing to purchase, explicitly verify: Does the product configuration match the instruction perfectly? 2. History Check: Review the linear ’ action_history’ list tags and notes. What candidate steps or items have you verified, which steps hit a dead end, and what [SUBOPTIMAL] backups have you recorded along the timeline so far? 3. Step Check: Current step number. - Steps 1 to 6: STRICT EXPLORATION. Maintain zero tolerance for incomplete attributes. Linear backtrack via ‘< Prev‘ if not perfect . - Steps 7 or higher: ADAPTIVE TOLERANCE. You MUST finalize, configure, and purchase the best recorded [SUB-OPTIMAL] backup candidate from your history list. </state_analyze> <action_analyze> Identify the next logical shopping or searching action based on clear reasoning. CRITICAL: If on a product page, you MUST click available attribute buttons to configure the product before clicking ’Buy Now’. </action_analyze>

4. Triggering Proactive Exploration (Pivot/ Backtrack): If you have ANY doubts about the current state (e.g., a specific target attribute is missing, or the material/style

25

is sub-optimal), DO NOT force a forward action. In early steps (e.g., steps 1-5), you must maintain STRICT ZERO TOLERANCE for imperfections. Use ‘click[< Prev]‘ to return to the parent node, or ‘click[Back to Search]‘ to return to the root node to spawn a new branch. 5. Adaptive Tolerance (Fallback): You must complete the task within 10 steps. If you have explored multiple nodes and are running out of steps (e.g., step 7 or later), you MUST lower your standards immediately. Relocate to a previously visited sub-optimal node ID rather than failing the task by wasting remaining steps on more searches. Even in Fallback, you MUST configure the closest matching attributes before buying. 6. Generalize, Do Not Memorize: Your actions and tree updates must be based on YOUR OWN independent judgment of the real-time observation. All punctuation must be standard English half-width symbols. 7. Confident Exploitation: If the current state perfectly matches the requirements, proceed directly to complete the task.

3. Step Check: Current step number. - Steps 1 to 6: STRICT EXPLORATION. Maintain zero tolerance for incomplete attributes. Backtrack to find a perfect match. - Steps 7 or higher: ADAPTIVE TOLERANCE. You MUST finalize, configure, and purchase the best available sub-optimal candidate by relocating to its node ID. </state_analyze> <action_analyze> Identify the next logical shopping or searching action based on clear reasoning. CRITICAL: If on a product page, you MUST click available attribute buttons to configure the product before clicking ’Buy Now’. </action_analyze> <action> [YOUR ACTION HERE] (e.g., search[keywords] OR click[value]) </action> <change_action_history_tree> Next parent position id: "node_ID" OR Current position id: "node_ID" </change_action_history_tree>

### Action History Tree Management: The ‘action_history‘ is a Behavior Tree representing your shopping exploration paths based entirely on node topology and executed actions (no text notes are updated) . - Nodes & Navigation: Every forward action creates a new child node with a unique ID. Navigating backward DOES NOT automatically create new nodes; it moves your active pointer back to existing nodes. - Status & Path Tracking: Since no text notes are stored, you must deduce which paths are dead ends or sub-optimal candidates strictly by analyzing the tree’s structure (action strings and branch connections) in the input . - Tree Control Signals: * Next Parent Assignment: To branch or go deeper, set ‘Next parent position id‘ to a target node ID. If you need to abandon an imperfect product and spawn a new parallel search from a previous history checkpoint, set it to that historical node ID. * Target Node Relocation: Set ‘Current position id‘ to a historical node ID if you want to shift your active pointer back to a stable context without creating an empty child node. Your response MUST strictly use the following formatted tags: <state_analyze> 1. Current State: Define your immediate goal. Analyze the latest shopping observation. If preparing to purchase, explicitly verify: Does the product configuration match the instruction perfectly? 2. History Check: Review the ’action_history’ tree topology. Based on the action paths, which previous node IDs represent valid suboptimal backup paths that you can backtrack to if needed?

26

Related documents

• Schema: Discovering Unknown Environments via Agentic Program Induction#1144107
• SkillVine: Agent Skill Evolution via Branching Exploration#1117062
• TREX: Automating LLM Fine-tuning via Agent-Driven Tree-based Exploration#14052
• Experience Augmented Policy Optimization for LLM Reasoning#321847
• Toward Self-Adaptive Physical AI: Can LLM Agents Manage Long-Horizon Physical Tasks?#1071224