arXiv CSopen access

What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

W HAT YOU T HINK IS W HAT YOU S EE: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Haoxi Li 1 Qinglin Hou 2 Jianfei Ma 3 Jinxiang Lai 1 Tao Han 1 Sikai Bai 1 Jingcai Guo 4 Jie Zhang 1 Song Guo 1

arXiv:2605.03782v1 [cs.AI] 5 May 2026

Abstract To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities into their policies via explicit CoT reasoning, enabling them to mentally simulate futures before acting. However, relying solely on passive reasoning over visited states is insufficient for sparse-reward tasks, as it lacks the epistemic drive to actively uncover the “known unknown” required for robust generalization. We ask: Can VLM agents actively find signals that challenge and refine their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent’s linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning “what the agent thinks” with “what the agent sees” is key to solving complex or sparse agentic tasks.

(a) Sokoban Observation (b) PrimitiveSkill Observation Figure 1. Prediction error from different environments. (a) Prediction: the target and box will be at the same row. (b) Prediction: the purple cube will be stacked on top of the green cube.

observable visual environments. To navigate these complex dynamics, recent VLM agents increasingly internalize world modeling (Xing et al., 2025) directly into their policies via reinforcement learning (RL), employing explicit Chain-ofThought (CoT) reasoning to maintain belief states and predict world dynamics. However, current paradigms (Wang et al., 2025; Chen et al., 2025; Shu et al., 2025; He et al., 2025) largely restrict this capability to passive exploitation of visited states, refining the agent’s reasoning accuracy on the current data distribution rather than driving active discovery. This reliance on passive interpretation often leads to spurious success in sparse-reward settings: an agent may learn to perfectly describe a “dead end” in a puzzle game without ever realizing it should have explored a different path. We thus ask: Can VLM agents actively find signals that challenge and refine their internal world model through curiosity-driven exploration?

1. Introduction The rapid evolution of Vision-Language Model (VLM) agents has fundamentally shifted the control paradigm from text-based, fully observable settings to complex, partially

To address this, curiosity learning (Schmidhuber, 1991; Sun et al., 2011; Schmidhuber, 2010b; Houthooft et al., 2016; Azar et al., 2019) offers a natural solution to this challenge. Typically, it involves training an auxiliary world model to predict future observations, using the prediction error as an intrinsic reward. However, applying this paradigm directly to VLM agents overlooks a critical architectural shift: the world model is no longer an external auxiliary module but is internalized within the policy itself via linguistic CoT reasoning. Consequently, standard curiosity methods (Pathak et al., 2017b; Ermolov & Sebe, 2020; Groth et al., 2021;

1

Department of Computer Science and Engineering, Hong Kong University of Science and Technology (HKUST), Hong Kong, China. 2 Department of Computer Science, University of Southern California, Los Angeles, USA 3 School of Computing, National University of Singapore, Singapore. 4 Department of Computing, Hong Kong Polytechnic University, Hong Kong, China.. Correspondence to: Jie Zhang <[email protected]>. Preprint. May 6, 2026.

1

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Guo et al., 2022), which focus solely on predicting latent visual futures from visual pasts, are insufficient, as they decouple exploration from the agent’s linguistic reasoning process. For instance, a VLM agent might learn robust visual representations to satisfy the curiosity objective, yet its linguistic reasoning could remain detached from physical reality, leading to hallucinations. To bridge this modality gap, we argue for a unified objective grounded in the principle that “what the VLM agent thinks should predict what it sees”. Effective exploration, therefore, must drive the VLM agent towards interactions where its linguistic hypothesis of the future fails to align with the visual reality, forcing the internalized world model to ground itself through active falsification.

quently, exploration becomes a self-paced curriculum: as the visual encoder captures increasingly complex semantics, the rejuvenated projector continuously reveals new layers of known unknowns, sustaining the curiosity drive throughout long-horizon learning. We empirically evaluate GLANCE across a diverse suite of five agentic tasks: Grid Puzzles, 3D Navigation, Object Manipulation, and Geometric Reconstruction. Our results demonstrate that GLANCE consistently outperforms current exploitation-based RL methods (Wang et al., 2025; Chen et al., 2025) in VLM agents. Notably, we show that our cross-modal curiosity empowers VLM agents to learn effective exploration policies even in the absence of extrinsic rewards. Furthermore, ablation studies on Curriculum Exploration confirm that periodic rejuvenation of the alignment projector is crucial to preventing curiosity collapse and sustaining long-term epistemic drive. Remarkably, GLANCE achieves these results using a lightweight architecture where the VLM agent simultaneously serves as the world model and the policy, concurrently trained across all tasks without human demonstrations. These findings suggest that the synergetic grounding of thinking and seeing is a fundamental prerequisite for building autonomous, curious, and physically-grounded agents.

In this work, we present GLANCE (Grounding Linguistic Alignment for Curiousity Exploration), a unified framework that operationalizes this principle by strictly aligning the agent’s explicit reasoning with visual reality. Specifically, GLANCE projects the latent representation of the VLM agent’s linguistic prediction into the visual representation space of a momentum-updated target network, which encodes the actual observation. We leverage the discrepancy between these two representations as a unified self-supervised objective that solves three problems simultaneously: (i) shapes the visual encoder to capture semantically actionable features, (ii) grounds the internalized world model in physical dynamics, and (iii) provides a curiositydriven intrinsic reward to train the policy. This joint optimization of world modeling and exploration transforms the latter from stochastic search into active falsification, steering the agent towards states where its current reasoning fails to explain the visual reality. Unlike standard visual bootstrapping methods (Grill et al., 2020; Schwarzer et al., 2020; Guo et al., 2022) that rely solely on visual-to-visual prediction, GLANCE drives world modeling via natural language reasoning, ensuring exploration is guided by the agent’s semantic understanding rather than mere visual novelty.

2. Preliminaries In this section, we formalize the reinforcement learning framework for VLM agents and describe the internalized world modeling paradigm that serves as our foundation. 2.1. Problem Formulation We formulate the multi-turn VLM agentic task as a Partially Observable Markov Decision Process (POMDP) (Åström, 1965; Kaelbling et al., 1998), defined by a tuple (S, A, O, T, R, Z, γ), where S, A, and O correspond to the state, action, and observation spaces. At each turn t, the agent takes an action at ∼ πθ (at |bt ) in a sequence of natural language tokens, where bt is a sufficient statistic computed from the history ht = (o0 , a0 , . . . , ot−1 , at−1 , ot ). Then, the environment transitions to a new state st+1 ∼ T (st+1 |st , at ), which is hidden from the agent’s view, and emits a visual observation ot+1 ∼ Z(ot+1 |st+1 , at ) and a reward R(st , at ) to the agent. For brevity, we will refer to the reward using the shorthand rt whenever applicable. The VLM agent’s objective is to learn a policy πθ that maximizes the expected cumulative discounted return: "∞ # X t J (θ) = Eπθ ,T,Z γ Ebt [R(st , at )] b0 (1)

However, a critical challenge arises from the mismatch in learning dynamics between the VLM agents’ thinking and seeing. Specifically, since the pre-trained LLM backbone is already semantically rich, the proposed lightweight alignment projector tends to quickly overfit to superficial visual features. This leads to an early vanishing in intrinsic rewards, a phenomenon we term curiosity drain, where the agent stops exploring because it mistakenly believes it has mastered the environment. To address this, we further introduce a Curriculum Exploration mechanism. By periodically re-initializing the projector weights while preserving the evolving visual encoder, we enforce a successive refinement process. This rejuvenation compels the agent to re-examine familiar states using its enhanced visual perception, uncovering fine-grained discrepancies that were previously masked by the projector’s overfitting. Conse-

t=0

where γ ∈ [0, 1) is the discount factor, b0 is the initial belief, and bt is updated deterministically as bt = 2

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

τ (bt−1 , at−1 , ot ), where τ is the belief update function.

3. Method

Since solving the belief-space POMDP is computationally intractable (Madani et al., 1999), history-based POMDP methods (Silver & Veness, 2010; Guez et al., 2012) provide a tractable alternative by summarizing past observations. Analogously, RNN- or Transformer-based architectures can be seen as maintaining a hidden state that encodes this history, and, in this view, a VLM can be interpreted as updating its internal representation with each new observation to predict the next token, akin to approximate belief tracking in a POMDP.

In this section, we propose GLANCE, a unified framework designed to transform the VLM agent’s internalized world modeling into a structured source of epistemic drive. As illustrated in Fig. 2, the architecture consists of two parallel streams: an Online VLM Agent that generates reasoning and visual-grounded predictions, and a Target Network that provides stable evolving regression targets. 3.1. The GLANCE Framework The core design of GLANCE is built upon the interaction between a thinking policy and a witnessing encoder. We formalize the two primary components as follows.

2.2. Internal World Models in VLM Agents Reasoning as World Modeling To address the demands of partial observability, recent frameworks (Wang et al., 2025) require VLMs to explicitly internalize world modeling into their reasoning processes. Specifically, at turn t, upon receiving an observation ot , the model infers the underlying state st that generates it. From this inferred state, a reasoning path zt is constructed, and a prediction of the next state transition st+1 is made, which serves as a prior before the next observation is received. Together, the inferred state, reasoning path, and predicted next state constitute the agent’s internal world-modeling tokens Φ, structured as the following sequence:

Online VLM Agent The online stream is parameterized by θ = (v, ℓ), comprising a visual encoder fv and a LLM backbone Λℓ . At each turn t, given the history ht and the current observation ot , the agent internalizes the world modeling with Φt , followed by an action at . To account for the prediction error, we define the linguistic hypothesis state ht+1 ∈ Rd as the Transformer’s final layer hidden state corresponding to the position of the last prediction token. This vector ht+1 serves as a semantic compression of the agent’s prediction about the future state st+1 . To bridge the modality gap between the linguistic latent space and the visual feature space, we augment the online agent with a lightweight projector gψ .

<Obs>st </Obs><Res>zt </Res><Pred>st+1 </Pred>.

By combining the inferred state st and the reasoning trace zt , the model produces an action at (<Ans>at </Ans>) that is executable in the environment. Notably, the predicted next state reflects the agent’s belief about future outcomes. The knowledge gap is often characterized by the discrepancy between this prediction and the actual observation, highlighting potential weaknesses in the agent’s world modeling capability.

Momentum Encoder To provide stable targets for the self-supervised alignment task, we instantiate a momentum target network parameterized by ϕ. This stream consists of a visual encoder fϕ , which is structurally identical to the online visual encoder fv . Following the principle of bootstrap latent methods (Grill et al., 2020; Guo et al., 2022), the parameters ϕ are not updated via gradient descent. Instead, the target network acts as a slowly evolving teacher that maintains a moving average of the online encoder’s weights. At the end of each training iteration, the parameters are updated as:

Optimization with Extrinsic Rewards Following the existing VLM-RL paradigm, the policy is typically optimized using the Proximal Policy Optimization (PPO) algorithm (Schulman et al., 2017). Such VLM agent is supervised by a composite reward signal provided by the environment or external evaluators (e.g., LLM-as-a-judge). We denote this aggregate signal as the extrinsic reward rte : rte = rttask + rtreason + rtformat

ϕ ← αϕ + (1 − α)v,

(3)

where α ∈ [0, 1] is the target decay rate. Upon executing the executable action aet and observing the next turn ot+1 , the target network encodes the future observation into a visual reality representation:

(2)

where rttask is the sparse task reward, rtreason evaluates the quality of world model reasoning, and rtformat ensures struc-

yt+1 = sg(fϕ (ot+1 )),

tural adherence. To handle the hierarchical credit assignment between turn-level outcomes and token-level generation, a Bi-Level Generalized Advantage Estimation (GAE) mechanism is employed, which propagates advantages from the final turn-level reward back to individual reasoning tokens within each action sequence.

(4)

where sg(·) denotes the stop-gradient operator. This setup enables GLANCE to align the agent’s current linguistic hypothesis ht+1 with the grounded visual reality yt+1 in a temporally consistent manner. 3

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

GLANCE Framework

Grounding

Projector

𝑳𝒆𝒙𝒑𝒍𝒐𝒓𝒆 Env.

World Modeling: <pred> 𝒔ො 𝒕+𝟏 </pred> Answer:ഥ 𝒂𝒕

𝒓𝒕 = 𝒓𝒆𝒕 + 𝜷 ∙ 𝑳𝒆𝒙𝒑𝒍𝒐𝒓𝒆 𝑳𝒆𝒙𝒑𝒍𝒐𝒓𝒆

Large Language Model

𝒓𝒆𝒕

Env.

VLM agent

Vison Encoder

Target Vison Encoder

Text Prompt

𝒐𝒕

Prediction Loss Source

Back propagation

𝒐𝒕+𝟏

EMA Updating

Rollout

Target Vison Encoder Vison Encoder with same architecture

Figure 2. Overview of the GLA N CE framework. GLA N CE unifies world modeling and exploration into a self-supervised cross-modal loop. Left: The VLM agent generates an explicit reasoning trajectory containing a future state prediction st+1 . The latent representation corresponding to the final </pred> token is mapped via a lightweight projector to align with the visual reality encoded by a Momentum Target Vision Encoder. The resulting prediction loss, Lexplore , serves as the source of curiosity. Crucially, gradients from Lexplore backpropagate through the frozen LLM to update the Online Vision Encoder, forcing it to capture semantically actionable features. Right: The VLM agent interacts with the environment via rollouts, guided by a composite reward rt = rte + β · Lexplore . This incentivizes the agent to proactively visit states where its internal reasoning fails to explain visual outcomes. The target network is updated via EMA to provide stable regression anchors. Fire icons indicate trainable modules, while snowflake icons denote frozen parameters.

3.2. Grounding Reasoning via Visual Alignment

To ensure that the internalized world model is effectively grounded while maintaining the LLM’s semantic priors, we optimize Eq. (6) through a selective gradient routing: we freeze the LLM parameters to prevent computational overhead and language drift (Lazaridou & Baroni, 2020), but allow gradients to back-propagate through the backbone. This procedure ensures that ∇Lexplore explicitly updates the projector gψ and, crucially, the online visual encoder fv . By doing so, the visual encoder is shaped to extract features that are semantically consistent with the agent’s logical reasoning, effectively “learning to see what the agent thinks”.

To endow the agent with a physically grounded world model, we require that its linguistic hypothesis of the future dynamics (manifested in the reasoning tokens) effectively predicts the visual reality (encoded by the target network). We operationalize this principle through a self-supervised crossmodal prediction objective. Cross-modal Prediction Given the extracted linguistic hypothesis state ht+1 from the online VLM’s last prediction token, we first map it from the semantic language space to the visual latent space using the projector gψ . Formally, the online prediction ybt+1 is computed as: ybt+1 = gψ (ht+1 ).

3.3. Curiosity as Active Exploration To drive active exploration, we interpret the prediction error not merely as a representational loss, but as a proxy for epistemic uncertainty. The intrinsic reward rti at turn t is formulated as follows:

(5)

Simultaneously, the momentum target network processes the next-turn visual observation ot+1 to yield the target representation yt+1 = fϕ (ot+1 ).

rti = β · Lexplore (v, ψ, t).

Prediction Objective The prediction loss Lexplore to minimize is defined as the mean squared error between the normalized predictions and targets:   2 yt+1 ybt+1 − sg Lexplore (v, ψ, t) = , ∥b yt+1 ∥2 ∥yt+1 ∥2 2 (6) where sg(·) denotes the stop-gradient operator. This operator is crucial for preventing representational collapse, ensuring that the target network provides a stable anchor for the online agent to regress towards.

(7)

The total reward signal rt utilized for policy optimization is defined as a composite sum: rt = rte + rti ,

(8)

where rte is the extrinsic reward defined in Sec. 2.2. 3.4. Curriculum Exploration with Rejuvenation In GLANCE, a critical challenge arises from the mismatch in learning dynamics: the lightweight projector gψ tends 4

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity Table 1. Main results on the general agentic benchmarks. We report the average success rate for puzzle and embodied control tasks, and perceptual similarity (Average of DINO and DreamSim) for the SVG reconstruction task. Trained models utilize Q WEN 2.5-VL-3B as the backbone. Bold indicates the best performance among trained models. Model/Method

Navigation

Sokoban FrozenLake

PrimitiveSkill

SVG

Overall

Base Common Average Place Stack Drawer Align Average Dino DreamSim Average Open-Source Models Qwen2.5-VL-72B (Bai et al., 2023)

0.20

0.44

0.70

0.77

0.74

1.00 0.50

0.00

1.00

0.44

0.84

0.62

0.73

0.51

Qwen2.5-VL-7B (Bai et al., 2023)

0.14

0.14

0.33

0.38

0.35

0.00 0.00

0.00

0.75

0.19

0.84

0.27

0.56

0.28

Qwen2.5-VL-3B (Bai et al., 2023)

0.13

0.14

0.20

0.26

0.23

0.00 0.00

0.00

0.00

0.00

0.79

0.30

0.54

0.21

VLM-R1-3B (Shen et al., 2025)

0.16

0.15

0.33

0.34

0.34

0.00 0.00

0.00

0.00

0.00

0.79

0.27

0.54

0.24

Dense Extrinsic Rewards RL with World Model Reasoning for Visual States (Backbone: Qwen2.5-VL-3B) VAGEN-Full

0.79

0.72

0.80

0.81

0.81

1.00 0.88

1.00

1.00

0.97

0.90

0.66

0.78

0.81

GLANCE-Full

0.85

0.78

0.86

0.88

0.87

1.00 0.88

1.00

1.00

0.97

0.92

0.70

0.81

0.86

0.90 0.92

0.65 0.69

0.78 0.80

0.76 0.80

Sparse Extrinsic Reward RL with World Model Reasoning Strategy (Backbone: Qwen2.5-VL-3B) VAGEN-Base GLANCE-Base

0.61 0.74

0.71 0.73

0.78 0.81

0.80 0.81

0.79 0.81

1.00 0.88 1.00 0.88

0.88 1.00

0.88 0.88

0.91 0.94

Turn-level PPO with World Model Reasoning Strategy (Backbone: Qwen2.5-VL-3B) Turn-PPO w/ Mask

0.38

0.68

0.78

0.84

0.81

0.00 0.00

0.00

1.00

0.25

0.89

0.64

0.77

0.55

GLANCE w/ Turn-PPO

0.52

0.70

0.78

0.80

0.79

1.00 0.63

0.00

1.00

0.66

0.90

0.66

0.78

0.69

RL Baselines with World Model Reasoning Strategy (Backbone: Qwen2.5-VL-3B) Vanilla-PPO

0.18

0.21

0.32

0.25

0.29

0.00 0.00

0.00

0.00

0.00

0.83

0.44

0.64

0.26

GRPO w/ Mask

0.20

0.57

0.88

0.81

0.85

0.00 0.00

0.00

1.00

0.25

0.92

0.66

0.79

0.54

Proprietary Models o4-mini (OpenAI, 2025)

0.44

0.82

0.75

0.75

0.75

1.00 0.50

0.00

0.75

0.56

0.90

0.66

0.78

0.67

GPT-4o (Hurst et al., 2024)

0.43

0.54

0.75

0.69

0.72

0.50 0.63

0.00

0.88

0.50

0.91

0.69

0.80

0.60

Gemini 2.5 Pro (Google, 2025)

0.58

0.78

0.63

0.63

0.63

0.63 0.63

0.00

0.75

0.50

0.93

0.78

0.86

0.67

Claude 4.5 Sonnet (Anthropic, 2025)

0.31

0.80

0.67

0.67

0.67

0.63 0.50

0.00

1.00

0.53

0.95

0.81

0.88

0.64

Claude 3.7 Sonnet (Anthropic, 2024)

0.25

0.69

0.48

0.47

0.47

0.63 0.13

0.00

1.00

0.44

0.94

0.77

0.85

0.54

4. Experiment

to converge rapidly to the pre-trained LLM’s rich semantic space, finding trivial mappings to the visual targets long before the visual encoder fv has fully adapted. This leads to an early vanishing of Lexplore , a phenomenon we term curiosity drain. To counteract this, we introduce an adaptive Curriculum Exploration mechanism.

In this section, we evaluate GLANCE against state-of-theart VLM-RL baselines across five diverse agentic tasks. We aim to answer: (i) Can GLANCE facilitate effective exploration in sparse-reward environments? (ii) Does reasoningdriven curiosity improve the grounding of internalized world models? and (iii) How does the adaptive curriculum rejuvenation sustain long-term exploration?

Formally, we monitor the instantaneous rate of change in prediction loss, defined as

4.1. Experimental Setup (t)

(t−1)

δt = Lexplore − Lexplore .

(9)

Environment We benchmark GLANCE on a comprehensive evaluation suite featuring five distinct tasks that span 2D grid puzzles, 3D embodied navigation, and open-ended generative reasoning.

When δt remains below a convergence threshold ϵ for a consecutive duration of K steps, we randomly re-initialize the projector parameters, while preserving the evolved parameters of the online visual encoder v. Upon rejuvenation, the prediction error spikes. This compels the agent to revisit familiar states to ground its reasoning in increasingly finegrained visual details, thereby sustaining the epistemic drive and revealing new layers of “known unknowns” throughout the learning process.

Cognitive Grid Puzzles (Sokoban & FrozenLake): These classic reasoning domains serve as testbeds for multi-step logical planning. In Sokoban (Schrader, 2018), the agent must execute a precise sequence of pushes to move boxes to targets while avoiding irreversible deadlocks. FrozenLake (Towers et al., 2024) requires navigating a hazardous grid where safe paths are non-trivial to discover. Both tasks utilize discrete action spaces and demand accurate state 5

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

estimation to handle the lack of a global view.

Sokoban

Success Rate

0.8

Success Rate

Embodied 3D Control (Navigation & PrimitiveSkill): To probe performance in high-dimensional visual settings, we employ Navigation (Yang et al., 2025; Kolve et al., 2017) and PrimitiveSkill (Tao et al., 2024; Nasiriany et al., 2022). Navigation is a first-person 3D task that requires the agent to locate specific objects in indoor scenes, challenging its ability to synthesize spatial memory from transient visual observations. PrimitiveSkill involves controlling a Panda robotic arm for complex manipulation using a hybrid action space (e.g., pick(x,y,z)). This task requires finegrained visual grounding to map 3D coordinate reasoning to physical interactions.

Navigation-Average 1.00

GLANCE GLANCE w/o projector rejuvenation GLANCE w/o EMA update

0.6

0.4

0.2

0.80 0.60 0.40 0.20 0.00

GLANCE GLANCE w/o projector rejuvenation GLANCE w/o EMA update

-0.20 0

50

100

150

Steps

200

250

0

20

40

60

80

Steps

Figure 3. Ablation study on GLANCE components.

Generative Reasoning (SVG Reconstruction): Representing an open-ended creativity task (Rodriguez et al., 2025), the agent must generate SVG code to replicate a target geometry. This task features an unbounded text-based action space and requires the agent to align abstract linguistic commands with dense, pixel-level visual outcomes, serving as a stress test for cross-modal consistency.

loss optimization and the RL actor, while the critic is optimized with 1 × 10−5 , The global training batch size of 128. To counter the non-stationarity and high variance of the training process, we employ the same reward normalization scheme as in (Schulman et al., 2017; Burda et al., 2019), dividing intrinsic rewards by the running standard deviation of their corresponding returns. To balance exploration and exploitation, we set the intrinsic reward scaling factor β = 0.3 for the Navigation task and β = 0.1 for all other environments. During inference and rollout, we set the VLM generation temperature to 0.7 to balance diversity and precision. The detailed hyper-parameter settings are shown in Appendix B.7.

Baselines We utilize Qwen2.5-VL-3B (Bai et al., 2023) as the unified VLM backbone. We evaluate GLANCE as a plug-and-play module across three RL baselines with world model reasoning strategy to demonstrate its robustness: (i) VAGEN (Wang et al., 2025), featuring extrinsic rewards rte and Bi-Level GAE; (ii) VAGEN-Base (Wang et al., 2025), which utilizes more sparse rewards rtformat + rttask and standard token-level GAE; and (iii) Turn-level PPO with masking (Zhai et al., 2024). These are also compared against standard RL baselines including GRPO (Shao et al., 2024) and Vanilla PPO (Schulman et al., 2017). Furthermore, we compare our results against zero-shot performance from seven existing large vision-language models. The details of the baseline algorithm are provided in the Appendix B.1.

4.2. Main Results As shown in Tab. 1, our proposed GLANCE framework achieves consistently superior test-time performance across all five tasks compared to baselines relying solely on extrinsic rewards. In the sparse extrinsic reward RL setting, GLANCE achieves an overall score of 0.86, a significant improvement over VAGEN-Full. This performance gap is particularly evident in challenging logic-heavy tasks like Sokoban, where our method improves the success rate from 0.79 to 0.85. These results validate our core hypothesis: aligning the agent’s linguistic world model with visual reality provides a more robust grounding signal than passive exploitation of visited states.

Metrics Performance in puzzle and manipulation tasks is measured by the average Success Rate (SR), where a nonzero reward is provided only upon goal completion. For the SVG task, we employ a composite similarity metric aggregating DreamSim and DINO scores to assess generative quality. Detailed reward strategy and metrics definition are provided in Appendix B.2.

Moreover, the effectiveness of our curiosity drive is most prominent in the Turn-level PPO setting, especially within the PrimitiveSkill environment. In PrimitiveSkill, where precise coordinate reasoning is required but task rewards are extremely sparse, purely exploitation-based agents often suffer from reasoning stagnation. GLANCE overcomes this by incentivizing the agent to actively visit and reason about unfamiliar object-hand interactions. This confirms that for VLM agents, curiosity is not merely an auxiliary bonus but a fundamental driver for discovering the complex transition dynamics necessary for long-horizon control.

Projector Architecture We instantiate the projector gψ as a lightweight 2-layer Multi-Layer Perceptron (MLP). To prevent representational collapse, it employs a bottleneck architecture, projecting the VLM’s high-dimensional hidden state to a bottleneck layer (half the visual latent size) before mapping it to the visual feature space. Implementation Details During training, we employ decoupled learning rates: 1 × 10−6 for both the prediction 6

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

0.6 0.4 0.2 0.0

0.80

0.60

0.40

β = 0.1 β = 0.3 β = 0.5

0.20 0

50

100

150

200

Timing (s/step)

Success Rate

Success Rate

0.8

Sokoban

Navigation-Average 1.00

β = 0.05 β = 0.1 β = 0.5

250

0

20

Steps

40

60

SVG GLANCE VAGEN

600

400

0

100

200

Steps

Figure 4. Sensitivity analysis of the exploration weight β.

GLANCE VAGEN

800

600

400

200

200

80

Steps

Timing (s/step)

Sokoban 1.0

300

0

20

40

60

80

Steps

Figure 5. Comparison of per-step training time

Table 2. Ablation analysis on extrinsic rewards rtreason . Method VAGEN-Full (w/ rtreason ) VAGEN-Full (w/o rtreason ) GLANCE-Full (w/o rtreason )

Sokoban 0.79 0.63 0.67

FrozenLake 0.72 0.70 0.71

Sensitivity to Exploration Weight (β) Finally, we analyze the sensitivity of the exploration coefficient β. As shown in Fig. 4, we observe that the VLM performance peaks at moderate values, exhibiting a non-monotonic relationship. A moderate weight (β ∈ [0.1, 0.3]) strikes the optimal balance, providing sufficient epistemic drive without overshadowing the task reward. Conversely, an excessively high β (0.5) distracts the agent, leading to noisy exploration, while a too-low β fails to provide enough impetus to overcome exploration barriers.

Navigation (Avg) 0.81 0.79 0.81

4.3. Ablation Study In this section, we conduct component-wise ablations to verify the efficacy of our architectural choices and the robustness of the intrinsic reward.

4.4. Further Analysis Curriculum Exploration and Momentum Targets We first conduct an ablation study on the Sokoban and Navigation tasks. As shown in Fig. 3, while the agent without adaptive curriculum reset enables decent initial learning, our approach achieves superior performance, particularly in the later stages of training. This indicates that the reset mechanism successfully prevents the intrinsic reward from diminishing too early. Furthermore, replacing the momentum target encoder with a direct online copy results in significant instability and divergence. This underscores that the slowly evolving teacher provides a critical, consistent anchor for grounding the agent’s linguistic hypothesis.

Computational Efficiency A potential concern with incorporating self-supervised objectives is the added computational overhead. In Fig. 5, we compare the per-step training time of GLANCE against VAGEN-Full across two representative tasks. Although GLANCE introduces an additional forward pass for the momentum target network and a backward pass for the prediction loss, the computational overhead remains marginal. This efficiency is primarily attributed to our lightweight projector design and the fact that the target network requires no gradient computation. Given the significant performance gains, this trade-off is highly favorable for scaling VLM agents.

Intrinsic Curiosity vs. External Supervision A central question is whether our self-supervised intrinsic reward can substitute for expensive dense rewards. Tab. 2 compares the impact of removing the dense reasoning reward rtreason provided by the LLM-as-a-judge. As expected, removing rtreason leads to a decrease in performance for both frameworks, confirming that external feedback is a strong training signal. However, in this challenging setting lacking reasoning supervision, GLANCE-Full consistently outperforms VAGEN-Full. This suggests that our curiosity-driven intrinsic reward effectively acts as a proxy for reasoning supervision. Even in the absence of an external judge, the drive to align “thinking” with “seeing” implicitly encourages the agent to maintain high-quality reasoning.

Visualization of Curriculum Dynamics To empirically validate the effectiveness of our Curriculum Exploration mechanism, we visualize the temporal correlation between the prediction loss and task success rate in Fig. 6. The results exhibit a distinctive sawtooth style pattern: upon each projector reset, the prediction loss intentionally spikes, reflecting a sudden rejuvenation of the intrinsic curiosity signal. Crucially, each loss spike is followed by a subsequent increase in the success rate, allowing the VLM agent to break through performance plateaus that frequently trap passive models. This provides evidence that our curriculum mechanism effectively sustains the epistemic drive, allowing the visual encoder to ground reasoning in increasingly fine-grained environment details without stagnating. 7

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

0.8

Success Rate

Prediction Loss

reasoning. While promising, these methods predominantly rely on passive supervision (e.g., LLM-as-a-judge) to refine the world model on visited states. In contrast, GLANCE integrates world modeling with curiosity-driven exploration, transforming the reasoning process from a passive inference task into an active exploration process that proactively seeks to falsify and refine the agent’s internal belief state.

0.9

projector rejuvenation

2.0

1.5

1.0

0.5

0.7 0.6 0.5 0.4 0.3 0.2 projector rejuvenation

0.1 0.0

0

100

200

Steps

300

0

100

200

300

Steps

6. Discussion and Limitations

Figure 6. Visualization of the curriculum exploration dynamics.

In this work, we demonstrate that grounding linguistic reasoning in visual reality creates a robust epistemic drive. Below, we discuss critical design choices, current limitations, and future directions.

5. Related Work Curiosity-driven Exploration Prediction error has been shown to be an effective indicator of novelty across different domains, including neuroscience (Oudeyer & Kaplan, 2007) and machine learning. In particular, it manifests itself in decision-making systems to encourage active exploration, for example, through competency maps (Thrun & Möller, 1991). The comparison between prediction and actual observation or reward leads to subjective surprise (Schmidhuber, 2010a). This process interleaves with the world model, consistently treating prediction as an expectation of reality and, in turn, refining the world model itself (Schmidhuber, 1991). Similar ideas have been adopted in deep RL in combination with neural network–based modules (Stadie et al., 2015; Pathak et al., 2017a; Burda et al., 2019), showcasing the capability to solve sparse-reward problems and to sufficiently explore complex environments that would otherwise be impossible for standard RL. Nonetheless, these methods are confined to either pixel- or proprioception-based domains. Adapting them to the semantic reasoning space of VLM agents—where uncertainty stems from linguisticvisual misalignment—remains an open frontier that our work addresses.

Backbone Update and Language Drift A central design choice in GLANCE is to freeze the LLM backbone while back-propagating gradients solely to the visual encoder and projector. While updating the LLM could allow the agent to refine its reasoning logic to better match visual dynamics, it introduces the risk of language drift. Furthermore, fine-tuning large-scale backbones via RL is computationally intensive. We hypothesize that a balanced approach like Low-Rank Adapters (LoRA) could modulate reasoning without catastrophic forgetting, representing a promising avenue for future efficiency-performance trade-offs. Projector Expressivity and Information Bottleneck We currently employ a lightweight MLP projector to bridge the semantic-visual gap. This creates a tight information bottleneck that forces the visual encoder to capture high-level actionable semantics. However, this simple architecture might limit the granularity of the alignment. Future work could explore more sophisticated cross-modal mechanisms, such as Transformer-based projectors or cross-attention layers, to capture complex spatial-temporal correlations between the CoT and the future environment state.

World Models in Autonomous Agents. World models, which learn to simulate environmental dynamics to support planning, have been central to model-based RL (Sutton, 1991; Ha & Schmidhuber, 2018). Traditional approaches like Dreamer (Hafner et al., 2019) and MuZero (Schrittwieser et al., 2020) learn the transition model in a latent space to perform lookahead search or generate synthetic experiences. With the advent of LLMs, a new paradigm has emerged where the pre-trained transformer itself acts as a world simulator. Methods such as RAP (Hao et al., 2023) and WebDreamer (Gu et al., 2024) leverage LLMs to predict action outcomes and score candidate plans in text-based or web environments. However, these approaches typically treat the world model as an external module or rely on frozen knowledge. Most recently, frameworks like VAGEN (Wang et al., 2025) have proposed internalizing world modeling directly into the VLM’s policy via explicit Chain-of-Thought

7. Conclusion In this paper, we argue that for VLM agents to master partially observable, sparse-reward tasks, explicit reasoning must be coupled with an active exploration of internal beliefs against visual reality. We introduce GLANCE, a unified framework that transforms the agent’s linguistic predictions into a source of curiosity-driven exploration. By optimizing a self-supervised alignment objective, GLANCE simultaneously shapes a semantically-rich visual encoder and provides granular intrinsic rewards that prevent reasoning stagnation. Our empirical results demonstrate that GLANCE consistently achieves great performance across diverse domains without relying on expensive external supervision. Furthermore, the introduced Adaptive Curriculum Reset ensures a sustained epistemic drive throughout long-horizon 8

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

training. In the end, our work shifts the perspective of VLM agents from passive inference makers to active hypothesis testers, providing a scalable and principled path toward autonomous embodied discovery.

Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P., Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271–21284, 2020.

Impact Statement This paper presents work whose goal is to advance the field of machine learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Groth, O., Wulfmeier, M., Vezzani, G., Dasagi, V., Hertweck, T., Hafner, R., Heess, N., and Riedmiller, M. Is curiosity all you need? on the utility of emergent behaviours from curious exploration. arXiv preprint arXiv:2109.08603, 2021.

References

Gu, Y., Zhang, K., Ning, Y., Zheng, B., Gou, B., Xue, T., Chang, C., Srivastava, S., Xie, Y., Qi, P., et al. Is your llm secretly a world model of the internet? model-based planning for web agents. arXiv preprint arXiv:2411.06559, 2024.

Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://api.semanticscholar.org/ CorpusID:268232499, 2024. Anthropic. Introducing claude sonnet 4.5, 2025.

Guez, A., Silver, D., and Dayan, P. Efficient bayes-adaptive reinforcement learning using sample-based search. Advances in neural information processing systems, 25, 2012.

Åström, K. J. Optimal control of markov processes with incomplete state information i. Journal of mathematical analysis and applications, 10:174–205, 1965.

Guo, Z., Thakoor, S., Pı̂slar, M., Avila Pires, B., Altché, F., Tallec, C., Saade, A., Calandriello, D., Grill, J.-B., Tang, Y., et al. Byol-explore: Exploration by bootstrapped prediction. Advances in neural information processing systems, 35:31855–31870, 2022.

Azar, M. G., Piot, B., Pires, B. A., Grill, J.-B., Altché, F., and Munos, R. World discovery models. arXiv preprint arXiv:1902.07685, 2019. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A versatile visionlanguage model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023.

Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122, 2(3), 2018. Hafner, D., Lillicrap, T., Fischer, I., Villegas, R., Ha, D., Lee, H., and Davidson, J. Learning latent dynamics for planning from pixels. In International conference on machine learning, pp. 2555–2565. PMLR, 2019.

Burda, Y., Edwards, H., Storkey, A. J., and Klimov, O. Exploration by random network distillation. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreview.net/ forum?id=H1lJJnR5Ym. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. April 2021.

Hao, S., Gu, Y., Ma, H., Hong, J., Wang, Z., Wang, D., and Hu, Z. Reasoning with language model is planning with world model. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8154–8173, 2023.

Chen, S., Zhu, T., Wang, Z., Zhang, J., Wang, K., Gao, S., Xiao, T., Teh, Y. W., He, J., and Li, M. Internalizing world models via self-play finetuning for agentic rl. arXiv preprint arXiv:2510.15047, 2025.

He, X., Yang, T., Cao, K., Wu, R., Meng, C., Zhang, Y., Kang, Z., Wei, X., and Chen, Q. Active intelligence in video avatars via closed-loop world modeling. arXiv preprint arXiv:2512.20615, 2025.

Ermolov, A. and Sebe, N. Latent world models for intrinsically motivated exploration. Advances in Neural Information Processing Systems, 33:5565–5575, 2020.

Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.

Fu, S., Tamir, N., Sundaram, S., Chai, L., Zhang, R., Dekel, T., and Isola, P. DreamSim: Learning new dimensions of human visual similarity using synthetic data. June 2023.

Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.

Google. Gemini 2.5: Our most intelligent ai model, 2025. 9

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Kaelbling, L. P., Littman, M. L., and Cassandra, A. R. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998.

Rodriguez, J. A., Puri, A., Agarwal, S., Laradji, I. H., Rodriguez, P., Rajeswar, S., Vazquez, D., Pal, C., and Pedersoli, M. Starvector: Generating scalable vector graphics code from images and text. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 16175–16186, 2025.

Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Deitke, M., Ehsani, K., Gordon, D., Zhu, Y., et al. Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv:1712.05474, 2017.

Schmidhuber, J. Curious model-building control systems. In Proc. international joint conference on neural networks, pp. 1458–1463, 1991.

Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stanford, CA, 2000. Morgan Kaufmann.

Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation (1990-2010). IEEE Trans. Auton. Ment. Dev., 2(3):230–247, 2010a. doi: 10.1109/ TAMD.2010.2056368. URL https://doi.org/10. 1109/TAMD.2010.2056368.

Lazaridou, A. and Baroni, M. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419, 2020.

Schmidhuber, J. Formal theory of creativity, fun, and intrinsic motivation (1990–2010). IEEE transactions on autonomous mental development, 2(3):230–247, 2010b.

Madani, O., Hanks, S., and Condon, A. On the undecidability of probabilistic planning and infinite-horizon partially observable markov decision problems. In Hendler, J. and Subramanian, D. (eds.), Proceedings of the Sixteenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, July 18-22, 1999, Orlando, Florida, USA, pp. 541–548. AAAI Press / The MIT Press, 1999. URL http://www.aaai.org/Library/ AAAI/1999/aaai99-077.php.

Schrader, M.-P. B. gym-sokoban. https://github. com/mpSchrader/gym-sokoban, 2018. Schrittwieser, J., Antonoglou, I., Hubert, T., Simonyan, K., Sifre, L., Schmitt, S., Guez, A., Lockhart, E., Hassabis, D., Graepel, T., et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839): 604–609, 2020.

Nasiriany, S., Liu, H., and Zhu, Y. Augmenting reinforcement learning with behavior primitives for diverse manipulation tasks. In 2022 International Conference on Robotics and Automation (ICRA), pp. 7477–7484. IEEE, 2022.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/ 1707.06347.

OpenAI. Introducing openai o3 and o4-mini, 2025. Oudeyer, P. and Kaplan, F. What is intrinsic motivation? A typology of computational approaches. Frontiers Neurorobotics, 1:6, 2007. doi: 10.3389/NEURO.12.006. 2007. URL https://doi.org/10.3389/neuro. 12.006.2007. Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In Precup, D. and Teh, Y. W. (eds.), Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 2778–2787. PMLR, 2017a. URL http://proceedings.mlr.press/ v70/pathak17a.html.

Schwarzer, M., Anand, A., Goel, R., Hjelm, R. D., Courville, A., and Bachman, P. Data-efficient reinforcement learning with self-predictive representations. arXiv preprint arXiv:2007.05929, 2020. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv. org/abs/2402.03300, 2 (3):5, 2024. Shen, H., Liu, P., Li, J., Fang, C., Ma, Y., Liao, J., Shen, Q., Zhang, Z., Zhao, K., Zhang, Q., et al. Vlm-r1: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504.07615, 2025. Shu, B., Cai, Y., Sun, J., Han, C., Yu, E., Zhao, L., Hu, J., Zhang, Y., Lv, H., Peng, Y., et al. Thinking by doing: Building efficient world model reasoning in llms via multi-turn interaction. arXiv preprint arXiv:2511.23476, 2025.

Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778–2787. PMLR, 2017b. 10

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Silver, D. and Veness, J. Monte-carlo planning in large pomdps. Advances in neural information processing systems, 23, 2010. Stadie, B. C., Levine, S., and Abbeel, P. Incentivizing exploration in reinforcement learning with deep predictive models. CoRR, abs/1507.00814, 2015. URL http://arxiv.org/abs/1507.00814. Sun, Y., Gomez, F., and Schmidhuber, J. Planning to be surprised: Optimal bayesian exploration in dynamic environments. In International conference on artificial general intelligence, pp. 41–51. Springer, 2011. Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM Sigart Bulletin, 2(4):160– 163, 1991. Tao, S., Xiang, F., Shukla, A., Qin, Y., Hinrichsen, X., Yuan, X., Bao, C., Lin, X., Liu, Y., Chan, T.-k., et al. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. arXiv preprint arXiv:2410.00425, 2024. Thrun, S. B. and Möller, K. Active exploration in dynamic environments. Advances in neural information processing systems, 4, 1991. Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., De Cola, G., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., et al. Gymnasium: A standard interface for reinforcement learning environments. arXiv preprint arXiv:2407.17032, 2024. Wang, K., Zhang, P., Wang, Z., Gao, Y., Li, L., Wang, Q., Chen, H., Wan, C., Lu, Y., Yang, Z., et al. Vagen: Reinforcing world model reasoning for multi-turn vlm agents. arXiv preprint arXiv:2510.16907, 2025. Xing, E., Deng, M., Hou, J., and Hu, Z. Critiques of world models. arXiv preprint arXiv:2507.05169, 2025. Yang, R., Chen, H., Zhang, J., Zhao, M., Qian, C., Wang, K., Wang, Q., Koripella, T. V., Movahedi, M., Li, M., et al. Embodiedbench: Comprehensive benchmarking multimodal large language models for vision-driven embodied agents. arXiv preprint arXiv:2502.09560, 2025. Zhai, S., Bai, H., Lin, Z., Pan, J., Tong, P., Zhou, Y., Suhr, A., Xie, S., LeCun, Y., Ma, Y., et al. Fine-tuning large vision-language models as decision-making agents via reinforcement learning. Advances in neural information processing systems, 37:110935–110971, 2024.

11

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

A. Algorithm Algorithm 1 GLANCE Input: Learning rate for RL η and visual-linguistic alignment κ, EMA decay α, intrinsic reward scale β, curiosity drain parameters ϵ and K. Initialize: VLM parameters θ = {v, ℓ}, projector parameters ψ, momentum encoder ϕ ← v, and replay buffer B ← ∅. 1: for iteration i = 1, 2, . . . do 2: #1: Trajectory Rollout 3: for step t = 1, . . . , T do 4: Observe ot and generate response at ∼ πθ (·|ot , ht ) with explicit prediction st+1 . 5: Execute action at in environment, receive ot+1 and task reward rttask . 6: Extract online latent ht+1 at </prediction> and target latent yt+1 = sg(fψ (ot+1 )). 7: Compute prediction error Lexplore (ht+1 , yt+1 ) and intrinsic reward rti = β · Normalize(Lexplore ). 8: Store transition and latent pair in B. 9: end for 10: #2: Optimization 11: Step A: Joint Representation Learning 12: Freeze ℓ to preserve linguistic reasoning. 13: Sample latent pairs from B, compute ∇Lexplore w.r.t. {ψ, v}. 14: Update projector and visual encoder: 15: ψ ← ψ − κ∇ψ Lexplore 16: v ← v − κ∇v Lexplore 17: Step B: Policy and Critic Update 18: Compute Advantages At using Bi-level GAE on rttotal = rttask + rti . 19: Unfreeze ℓ and update full parameters θ via PPO objective: 20: θ ← θ − η∇θ JPPO (θ) 21: Step C: Momentum Encoder Update 22: ϕ ← αϕ + (1 − α)v 23: #3: Rejuvenation (Optional) 24: if curiosity drain then 25: ψ ← Randomly Initialize(ψ) 26: end if 27: Clear the buffer B ← ∅. 28: end for

B. Experimental Details. B.1. Baseline To evaluate the robustness and plug-and-play nature of GLANCE, we benchmark against several RL baselines specifically designed for VLM agents or multi-turn reasoning. All trained baselines share the same Qwen2.5-VL-3B backbone and the internalized world-modeling action structure at = ⟨zt , aet ⟩ defined in Section 2.2. The primary distinctions between these baselines lie in the composition of the extrinsic reward rte and the granularity of the credit assignment mechanism (i.e., advantage estimation). Table 3 summarizes these differences. Table 3. Key differences between RL baselines. We categorize baselines by their extrinsic reward density and the level at which advantages are calculated and propagated. Baseline

Extrinsic Reward (rte )

Advantage Estimation

VAGEN-Full (Wang et al., 2025) VAGEN-Base (Wang et al., 2025) Turn-level PPO (Zhai et al., 2024)

Dense: rttask + rtreason + rtformat Sparse: rttask + rtformat Sparse: rttask + rtformat

Bi-Level GAE Token-Level GAE Turn-Level GAE (Uniform)

12

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

VAGEN-Full (Wang et al., 2025) This baseline represents the upper bound of dense supervision. It utilizes a comprehensive reward signal where rtreason is derived from an LLM-as-a-judge to provide direct feedback on the accuracy of state estimation and transition modeling tokens. To effectively bridge the gap between high-level reasoning turns and low-level token generation, it employs Bi-Level GAE. This mechanism first computes a turn-level advantage Aturn and then uses it as t a terminal target to compute a secondary, inner-GAE for individual reasoning tokens within zt . This ensures that the credit for a successful turn is explicitly assigned to the reasoning steps that justified the action. VAGEN-Base (Wang et al., 2025) As a more challenging baseline for exploration, VAGEN-Base removes the dense reasoning supervision (rtreason = 0), relying only on task success and formatting adherence. Furthermore, it simplifies the credit assignment by using standard Token-Level GAE, where the sparse turn reward is assigned to the final token of the sequence. Advantages are then propagated backward through the temporal-difference (TD) error across all action tokens solely based on the value function’s estimates. This setup tests the agent’s ability to learn internal world models without explicit semantic guidance. Turn-level PPO with Masking (Zhai et al., 2024) This approach treats each turn as the atomic unit of optimization. While it uses the same sparse reward as VAGEN-Base, it simplifies the advantage estimation by computing a single scalar advantage for the entire turn (Aturn t ). This scalar is then assigned uniformly to every non-masked action token in the sequence. This coarse-grained assignment serves as a baseline to demonstrate the necessity of token-specific credit assignment for generating complex, multi-step reasoning chains. Standard RL Baselines In addition to the VLM-specific methods, we evaluate Vanilla PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024). Vanilla PPO is implemented without observation token masking, highlighting the pitfalls of standard RL in long-context language modeling. GRPO is utilized as a reference for group-based relative policy optimization, which omits the critic network but often requires significantly larger sample sizes to resolve high trajectory diversity in visual environments. B.2. Reward and Evaluation Metrics We follow the evaluation protocol of VAGEN (Wang et al., 2025) and use the same core metrics for comparability. We evaluate agents using trajectory-level metrics. Let τ denote a rollout trajectory, and let D denote the test set of trajectories. For evaluation, we define: • f (τ ) ∈ {0, 1}: binary success indicator, equal to 1 if the task objective is achieved by the end of τ and 0 otherwise; • g(τ ) ∈ [0, 1]: DreamSim similarity between the final rendered image and the target image; • h(τ ) ∈ [0, 1]: DINO-based similarity between the final rendered image and the target image. Puzzle and manipulation environments. For Sokoban, FrozenLake, Navigation, and PrimitiveSkill, a trajectory is considered successful if it meets the environment-defined success condition. We report the average success rate: Success Rate = Eτ ∼D [f (τ )] .

(10)

Consistent with these tasks, the environment reward is sparse and becomes non-zero only upon goal completion. SVG reconstruction. For SVG Reconstruction, we assess generation quality using perceptual similarity. We report the test-set averages of DreamSim and DINO similarities: DreamSim Score = Eτ ∼D [g(τ )] ,

(11)

DINO Score = Eτ ∼D [h(τ )] .

(12)

When a single summary metric is needed, we additionally report Avg = (DreamSim Score+DINO Score)/2. DreamSim (Fu et al., 2023) measures perceptual similarity via diffusion-model representations, while DINO (Caron et al., 2021) computes similarity in a self-supervised visual feature space; together they offer complementary views of reconstruction quality. 13

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

B.3. Environment Settings Sokoban. Tab. 4 lists the available actions, and Tab. 5 summarizes the environment configuration used in our experiments.

Table 4. Action space for the Sokoban environment. Name Up Left Right Down

Description Shift the agent by one grid cell upward. Shift the agent by one grid cell to the left. Shift the agent by one grid cell to the right. Shift the agent by one grid cell downward.

Table 5. Hyperparameters for the Sokoban environment. Name dim room max steps num boxes min actions to succeed max actions per step max turns

FrozenLake.

Value (6, 6) 100 1 5 3 3

Description Grid size for the Sokoban room. Maximum number of steps per episode. Number of boxes that must be pushed onto targets. Minimum action count required for a successful episode. Maximum number of actions allowed per turn. Maximum number of interaction turns per episode.

We use a four-direction grid controller (Tab. 6); key environment settings are shown in Tab. 7.

Table 6. Action space for the FrozenLake environment. Name Up Left Right Down

Description Move one cell toward the upper neighbor. Move one cell toward the left neighbor. Move one cell toward the right neighbor. Move one cell toward the lower neighbor.

Table 7. Hyperparameters for the FrozenLake environment. Name desc is slippery size max actions per step min actions to succeed max turns

Navigation.

Value None False 4 3 5 3

Description Map layout (None indicates a randomly generated map). Whether transitions include slipping dynamics. Side length of the square grid. Maximum number of actions allowed per turn. Minimum action count required to count as success. Maximum number of interaction turns per episode.

Tab. 8 enumerates movement and camera controls, while Tab. 9 reports the rendering and control settings. 14

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity Table 8. Action space for the Navigation environment. Name MoveAhead MoveBack MoveRight MoveLeft RotateRight RotateLeft LookUp LookDown

Description Translate forward by one movement step. Translate backward by one movement step. Strafe right by one movement step. Strafe left by one movement step. Yaw right by 90◦ . Yaw left by 90◦ . Pitch the camera upward by 30◦ . Pitch the camera downward by 30◦ .

Table 9. Hyperparameters for the Navigation environment. Name resolution down sample ratio fov multiview max actions per step success threshold step length max turns

Value 255 1.0 100 False 5 1.5 0.5 4

Description Rendered image resolution. Down-sampling ratio for observations. Camera field-of-view in degrees. Whether multiple camera views are enabled. Maximum number of actions allowed per turn. Distance threshold for success. Translation distance per movement action. Maximum number of interaction turns per episode.

PrimitiveSkill. The agent issues parameterized manipulation commands (Tab. 10). We also list the per-turn action budget and horizon in Tab. 11. Table 10. Action space for the PrimitiveSkill environment. Name pick(x, y, z) place(x, y, z) push(x1, y1, z1, x2, y2, z2)

Description Grasp an object at position (x, y, z) in the workspace. Place the grasped object at position (x, y, z). Push an object from (x1, y1, z1) to (x2, y2, z2).

Table 11. Hyperparameters for the PrimitiveSkill environment. Name max actions per step max turns

Value 2 3

Description Maximum number of actions allowed per turn. Maximum number of interaction turns per episode.

SVG Reconstruction. The policy outputs free-form SVG markup as its action (Tab. 12); dataset and horizon details are provided in Tab. 13. Table 12. Action space for the SVG Reconstruction environment. Name SVG Code

Description A text action specifying SVG markup.

15

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity Table 13. Hyperparameters for the SVG Reconstruction environment. Name dataset name

Value starvector/svg-iconssimple 2

max turns

Description Dataset used to sample target SVG examples. Maximum number of interaction turns per episode.

B.4. Reward Assignment Sokoban.

Tab. 14 details the reward components used in Sokoban.

Table 14. Reward structure for the Sokoban environment. Reward Type Success reward Failure penalty Box placement reward Format reward Grounding reward weight World modeling reward weight

FrozenLake.

Value 10 -0.1 1 0.5 0.5 0.5

Description Granted when all boxes are placed on target locations. Applied each step until completion. Added for each box pushed onto a target. Per-turn reward encouraging structured visual state reasoning. Weight applied to the task-state reward. Weight applied to the task-transition reward.

The reward decomposition for FrozenLake is summarized in Tab. 15.

Table 15. Reward structure for the FrozenLake environment. Reward Type Success reward Failure penalty Format reward Grounding reward weight World modeling reward weight

Navigation.

Value 10 -0.1 0.5 0.5 0.5

Description Granted when the agent reaches the goal position. Applied each step until completion. Per-turn reward encouraging structured visual state reasoning. Weight applied to the task-state reward. Weight applied to the task-transition reward.

The reward components for Navigation are listed in Tab. 16.

Table 16. Reward structure for the Navigation environment. Reward Type Success reward Failure penalty Format reward Grounding reward weight World modeling reward weight

PrimitiveSkill.

Value 10 -0.1 0.5 0.5 0.5

Description Granted when the agent reaches the goal location. Applied each step until completion. Per-turn reward encouraging structured visual state reasoning. Weight applied to the task-state reward. Weight applied to the task-transition reward.

Tab. 17 reports the reward design for PrimitiveSkill. 16

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

Figure 7. Examples of visual states from four environments used in our study

Table 17. Reward structure for the PrimitiveSkill environment. Reward Type Success reward Failure penalty Stage-based reward

Value 10 -0.1 (stage + 1) × 2

Format reward Grounding reward weight World modeling reward weight

0.5 0.5 0.5

SVG Reconstruction.

Description Granted when the manipulation task is completed. Applied each step until completion. Granted upon reaching key subgoals (stage is the highest completed stage). Per-turn reward encouraging structured visual state reasoning. Weight applied to the task-state reward. Weight applied to the task-transition reward.

The SVG reward is based on visual similarity; Tab. 18 lists the full specification. 17

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity Table 18. Reward structure for the SVG Reconstruction environment. Reward Type Image similarity

Value Variable

Format reward Grounding reward weight World modeling reward weight DreamSim weight Dino weight

0.5 0.5 0.5 5.0 0.0001

Description Weighted DreamSim (Fu et al., 2023) and DINO (Caron et al., 2021) similarity between the generated and target images. Per-turn reward encouraging structured visual state reasoning. Weight applied to the task-state reward. Weight applied to the task-transition reward. Scale factor applied to DreamSim similarity. We only use DreamSim score for reward.

B.5. Case Study In this section, we show some cases from our four environments (Fig. 7). B.6. Prompt Collection This section summarizes the prompt set used in our framework. We directly adopt the prompt templates from VAGEN (Wang et al., 2025) as our default prompts. For clarity and conciseness, we only present the prompts corresponding to Sokoban, Frozenlake, and Navigation in this section. Sokoban Training Prompt for World Model Strategy You are a Sokoban solver. Sokoban Quick Guide Goal: Push all boxes onto targets. Symbols (If image is provided there are no symbols): # Wall | _ Floor | O Target | X Box | P You | * Box on Target | S You on Target Rules: 1. Push boxes (can’t pull). 2. Avoid walls. Actions you can take: Left, Down, Right, Up. You can take up to 3 action(s) at a time, separated by ,. You should first give the description of your observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> e.g. <think><observation>The box is below the player and the target is below the box</ observation><reasoning>I need to go down then push the box down to the target</ reasoning><prediction>The player will be above the box, the target and box will be at the same place.</prediction></think><answer>Down,Down</answer> [Initial Observation]: <image> Decide your next action(s). You can take up to 3 action(s) at a time, separated by ,. You should first give the description of your observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> <think><observation>The player is at the bottom of the screen, and there is a box to the right of the player. The target is to the left of the box.</observation>< reasoning>The player needs to push the box to the target to complete the goal.</ reasoning><prediction>The player will push the box to the target, moving up, down, and to the left.</prediction></think><answer>Up, Down, Left</answer>

18

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

FrozenLake Training Prompt for World Model Strategy You are a FrozenLake solver. FrozenLake Quick Guide Goal: Reach the goal (G). Symbols (If image is provided there are no symbols): _ Frozen | O Hole | G Goal | P Player | X Player fell into hole | * Player on goal Rules: 1. Avoid falling into holes. 2. Frozen tiles are slippery, you may move perpendicular to your intended direction. Actions you can take: Left, Down, Right, Up. You can take up to 3 action(s) at a time, separated by ,. You should first describe the observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> e.g. <think><observation>The player is on the above the target</observation><reasoning >I should go down then left to reach the target</reasoning><prediction>The player will reach the target</prediction></think><answer>Down,Left</answer> [Initial Observation]: <image> Decide your next action(s). You can take up to 3 action(s) at a time, separated by ,. You should first describe the observation, then your reasoning, then predict the next state, and finally your answer. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> <think><observation>The player is on the right side of the grid.</observation>< reasoning>The player is on the right side of the grid, which is indicated by the position on the grid.</reasoning><prediction>The player will move to the left or down.</prediction></think><answer>Left, Left, Down</answer>

Navigation Training Prompt for World Model Strategy You are a home robot and perform navigation tasks according to instructions. Actions you can take: moveahead, moveback, moveright, moveleft, rotateright, rotateleft, lookup, lookdown. moveahead: Move forward by some distance moveback: Move backward by some distance moveright: Move rightward by some distance moveleft: Move leftward by some distance rotateright: Rotate to the right by 90 degrees rotateleft: Rotate to the left by 90 degrees lookup: Tilt the camera upward by 30 degrees lookdown: Tilt the camera downward by 30 degrees Rewards: Format correct: +0.5 Achieve the human instruction: +10.0 The instruction will be provided with each observation. Look at the image carefully and navigate to complete the instruction. Hints: 1. You can take multiple actions at a time, in most cases, if you find the target object is far away from you, you can call moveahead, moveleft and move right multiple times. 2. If you find yourself seems to be stuck, you can lookdown to see if there’s any object above or below you, you can also rotate to see if there’s any object behind you. Example: Round 1:

19

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity image_1 <think><observation>There is a garbage can in the upper left corner of the image, next to the kitchen sink. To move there, we can go forward-left, but since there’s a kitchen counter directly ahead, we should go left first.</observation><reasoning> Following the strategy, I can go by first moving leftward.</reasoning><prediction>I will be infront of the garbage</prediction></think> <answer>moveleft, moveleft</answer> Round 2: Env_feedback: Last action is executed successfully. image_2 <think><observation>From the secene, I see that by moving leftward, we are getting closer to the garbage can. Now, the garbage can is in front of me, slightly to the left. And there’s a large area ahead of us.</observation><reasoning>Following the strategy, I can go by first moving forward then moving leftward.</reasoning>< prediction>I will be closer to the garbage</prediction></think> <answer>moveahead, moveahead,moveahead,moveleft</answer> Round 3: Env_feedback: Last action is executed successfully. image_3 <think><observation>From the image we can see the garbage can is very close to us, still to our front-left. Moving leftward might be blocked but i can see that there is still space in front of me to get closer to the garbage can.</observation>< reasoning>Following the strategy, we can take about two steps forward then one step left to reach the garbage can.</reasoning><prediction>I will reach the garbage</ prediction></think> <answer>moveahead, moveahead,moveleft</answer> Round 4: Env_feedback: Success You can take up to 5 action(s) at a time, separated by ’,’. You should first give your thought process with the your observation, reasoning, and prediction of next state, then your answer. Both the observation and prediction should describe what you see or expect to see in the environment. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> e.g. <think><observation>I am at the entrance of a bedroom. There is a bed to the left, a desk with a lamp on the right, and a closet straight ahead. The target object, a book, appears to be on the desk.</observation><reasoning>I need to move toward the desk to reach the book. I’ll turn right and move forward.</reasoning><prediction>I am now standing in front of the desk. The desk has a lamp, a computer, and several books on it. The target book is within reach on the right side of the desk.</ prediction></think><answer>rotateright,moveahead,moveahead</answer> [Initial Observation]: <image> Human Instruction: I am looking for a luminous device on my desk to read books at night. Can you navigate to that object and stay close? Decide your next action(s). You can take up to 5 action(s) at a time, separated by ’,’. You should first give your thought process with the your observation, reasoning, and prediction of next state, then your answer. Both the observation and prediction should describe what you see or expect to see in the environment. Your response should be in the format of: <think><observation>...</observation><reasoning>...</reasoning> <prediction>...</prediction></think><answer>...</answer> <think><observation>There is a luminous device on the desk to the right of the chair. To reach it, I should move forward and then turn right.</observation><reasoning> Following the strategy, I can go by first moving forward then turning right to reach the luminous device.</reasoning><prediction>I will be close to the luminous device</prediction></think><answer>moveahead, moveahead, moveright</answer>

We use an LLM-as-a-Judge to evaluate the agent’s reasoning by checking whether the agent’s descriptions and predictions 20

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

match the ground-truth state information. We use task-specific prompts for both and StateEstimation (current-state grounding) and TransitionModeling (next-state prediction). Each prompt asks the judge to review the evidence and then output a YES/NO decision. Sokoban S TATE E STIMATION Evaluation Evaluate whether the description accurately captures the key position relationships in the Sokoban game state. Answer YES if the directional relationships are correct, or NO if they contain directional errors. # Context You are evaluating whether the description correctly identifies the directional relationships between: 1. The player and the box(es) 2. The box(es) and the target(s) The description doesn’t need to be perfectly precise or mention every detail - it just needs to have the correct directional relationships (Up, Down, Left, Right). Example: Groundtruth Current State Information: [’box0 is at the same row and to the left of the player’, ’target0 is above and on the left side of the player’, ’target0 is above and on the left side of box0’] State Description: The player is below the box and the target is below the box. - <think>The state description contains spatial relationship information, do further analysis. According to the ground truth data, box0 is at the same row and to the left of the player, target0 is above and on the left side of the player, target0 is above and on the left side of box0. The description states ’The player is below the box and the target is below the box.’ The player is actually at the same row as the box (not below), and the target is actually above the box (not below). Both directional relationships are incorrectly identified.</think><answer>NO</answer> Example: Groundtruth Current State Information: [’box0 is above and on the right side of the player’, ’target0 is above and at the same column as the player’, ’target0 is above and on the left side of box0’] State Description: The box is above the player and the target is to the left of the box - <think>The state description contains spatial relationship information, do further analysis. According to the ground truth data, box0 is above and on the right side of the player, target0 is above and at the same column as the player, target0 is above and on the left side of box0. The description states ’The box is above the player and the target is to the left of the box.’ It correctly identifies that the box is above the player (box0 is above and on the right side of the player). It also correctly identifies that the target is to the left of the box (target0 is above and on the left side of box0). Both key directional relationships are accurately described.</think><answer>YES</answer> # Groundtruth Current State Information: {state_information_dict} # State Description: {natural_language_description} Think step by step: 1. Relative Relationship Requirements: - Must describe at least one relationships BETWEEN entities (player-box, playertarget, box-target) - Absolute positions like "player is on the left side" are insufficient - Need relational descriptions like "player is left of target" 2. Essential Relationships to Check

21

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity - Player-Target relationship (highest priority) - Player-Box relationship - Box-Target relationship 3. Equivalent Expression Recognition - "box is above player" = "player is below box" - "target is left of box" = "box is right of target" - Must recognize these as identical spatial relationships. Absolute position is not allowed Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

Sokoban T RANSITION M ODELING Evaluation Evaluate whether the prediction correctly anticipates the key position relationships that will exist in the next Sokoban state. Answer YES if the predicted directional relationships are correct, or NO if they contain directional errors. # Context You are evaluating whether the prediction correctly identifies the directional relationships that will exist after the move: 1. The future position of the player relative to the box(es) 2. The future position of the box(es) relative to the target(s) # Important: The Prediction Comes First Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task is to check if the prediction correctly anticipated what actually happened. If the box and target are at same position, this prediciton is seen as success immediately (YES) Example: Groundtruth Next State Information: [’box0 is above and on the right side of the player’, ’target0 is above and on the left side of the player’, ’target0 is above and on the left side of box0’] Next State Prediction: The player will be to the left of the box, and the box will be to the right of the target. - <think>The prediction state contains spatial relationship between player and target, do further analysis. According to the ground truth data, box0 is above and on the right side of the player, target0 is above and on the left side of the player, target0 is above and on the left side of box0. The description states ’The player will be to the left of the box, and the box will be to the right of the target.’ It correctly identifies that the player is to the left of the box (since box0 is on the right side of the player). It also correctly identifies that the box is to the right of the target (since target0 is on the left side of box0). Therefore, this description correctly identifies the key directional relationships.</think><answer> YES</answer> # Groundtruth Next State Information: {state_information_dict} # Next State Prediction: {natural_language_description} Think step by step: 1. Relative Relationship Requirements: - Must describe at least one relationships BETWEEN entities (player-box, playertarget, box-target) - Absolute positions like "player is on the left side" are insufficient - Need relational descriptions like "player is left of target"

22

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

2. Essential Relationships to Check - Player-Target relationship (highest priority) - Player-Box relationship - Box-Target relationship 3. Equivalent Expression Recognition - "box is above player" = "player is below box" - "target is left of box" = "box is right of target" - Must recognize these as identical spatial relationships. Absolute position is not allowed Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

FrozenLake S TATE E STIMATION Evaluation Evaluate whether the description accurately captures the key position relationships in the FrozenLake game state. Answer YES if the directional relationships are correct, or NO if there are errors. # Context You are evaluating whether the description correctly identifies: 1. The directional relationship between the player and the goal (MUST Have) 2. The directional relationship between the player and the hole (if present) The description doesn’t need to be perfectly precise - it just needs to have the correct directional relationships between the player and target (Up, Down, Left, Right), and between the player and hole if applicable. # Groundtruth Current State Information: {state_information_dict} # State Description: {natural_language_description} Think step by step: 1. Player relationship with Goal - Goal (Target) MUST include in state description, without target the description is automatically wrong (NO) - If there is no direction between player and goal, like "player is right to the target", the description is automatically wrong (NO) - This takes highest priority over all other considerations 2. Equivalent Expression Recognition - "goal is above player" = "player is below goal" - "target is left of box" = "box is right of target" - Must recognize these as identical spatial relationships. Absolute position is not allowed 3. Simple Judgment Rule - If player at goal -> YES - If direction aligns with needed movement -> YES - Otherwise -> NO Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

23

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

FrozenLake T RANSITION M ODELING Evaluation Evaluate whether the prediction correctly anticipates the key aspects of the next FrozenLake state. Answer YES if the prediction accounts for directional relationships and potential holes, or NO if it contains errors. # Context You are evaluating whether the prediction correctly identifies: 1. The position relationship between the player and the goal after the prediction # Important: The Prediction Comes First Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task is to check if the prediction correctly anticipated what actually happened. The prediction doesn’t need to perfectly describe every aspect of the next state - it just needs to correctly anticipate the directional relationships (Up, Down, Left, Right) or address any dangers from holes. # Groundtruth Next State Information: {state_information_dict} # Next State Prediction: {natural_language_description} Think step by step: 1. Player relationship with Goal - If player is already at the goal position, the prediction is automatically correct (YES) - Goal (Target) MUST include in prediction state, without target the prediction is automatically wrong (NO) - If there is no direction between player and goal, like "player is right to the target", the prediction is automatically wrong (NO) - This takes highest priority over all other considerations 2. Directional Correctness - Evaluate if the predicted movement direction aligns with the relative position between player and goal - For example, if player is left of goal, moving right is correct - **CRITICAL: Recognize equivalent expressions of the same spatial relationship** * "player is above target" = "target is below player" * "player is left of target" = "target is right of player" * These are the SAME relationship expressed from different perspectives 3. Simple Judgment Rule - If player at goal -> YES - If direction aligns with needed movement -> YES - Otherwise -> NO Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

Navigation S TATE E STIMATION Evaluation Evaluate whether the description effectively communicates the spatial relationship between the agent and target object, even if the exact directional terms differ. Answer YES if the overall spatial understanding is correct, or NO if it fundamentally misunderstands the spatial layout. # Context You are evaluating whether the description effectively conveys where the target object is located relative to the agent. The exact directional terminology (left, right,

24

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity ahead, etc.) may differ between the state information and the description, but the important factor is whether the description would lead to correct navigation. # Groundtruth Current State Information: {state_information_dict} # State Description: {natural_language_description} Think step by step: 1. Check if the description contains spatial relationship between agent and target object - If no spatial relationship is mentioned, answer NO 2. If spatial relationship exists, check if the predicted direction is consistent with the target direction - "ahead/forward" = "ahead" - "left" = "left" - "right" = "right" - Combined directions like "forward-left", "forward-right" are acceptable if they include the correct primary direction 3. The prediction is correct if it mentions moving toward the target in a direction that reasonably aligns with the groundtruth direction Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

Navigation T RANSITION M ODELING Evaluation Evaluate whether the prediction effectively anticipates how the agent would navigate toward the target object, even if the exact directional terms differ. Answer YES if the overall navigation plan is reasonable, or NO if it misunderstands or did not mention the spatial layout. # Context You are evaluating whether the prediction effectively anticipates how the agent would move to reach the target object. The exact directional terminology (left, right, ahead, etc.) may differ between the state information and the prediction, but the important factor is whether the prediction would lead to successful navigation. # Important: The Prediction Comes First Remember: The Next State Prediction is made BEFORE the Groundtruth Next State exists. Your task is to check if the prediction correctly anticipated what actually happened. # Groundtruth Next State Information: {state_information_dict} # Next State Prediction: {natural_language_description} Think step by step: 1. First, check if the prediction explicitly uses EXACT directional terms that appear in the groundtruth state: "ahead", "left", "right", "up", "down". - Terms like "move towards", "closer to", "near", "approaching", "in front of", "by", "at" DO NOT qualify - "Will be on the left/right/ahead" or "Will move left/right/forward" DO qualify - If no exact directional match to groundtruth is present, conclude with NO immediately 2. If explicit direction words exist, verify that they EXACTLY match the target object’ s direction in the groundtruth: - If target is "ahead", prediction must specify "ahead", "forward", "slightly left", OR "slightly right" (special case: we allow slightly left/right for ahead targets)

25

Driving Exploration in VLM Agents via Visual-Linguistic Curiosity - If target is "right", prediction must specify "right" - If target is "left", prediction must specify "left" 3. Even if the prediction mentions intermediate objects correctly, it MUST explicitly state the correct final direction to the target object 4. The prediction cannot substitute object references for directions (saying "move to X" instead of "move right") 5. Remember that the prediction was made BEFORE the groundtruth state was determined Your answer should be in the format of <think>...</think><answer>YES</answer> or < think>...</think><answer>NO</answer>.

B.7. Implementation Details Rejuvenation Settings We set the convergence threshold ϵ = 0.1, for consecutive duration steps, we use K = 30 for Sokoban and FrozenLake and K = 20 for the other tasks. Projector Architecture. We use a lightweight two-layer MLP projector to map the VLM hidden representation to the target visual feature space. Specifically, we take the final-layer hidden state associated with the </prediction> token as the prediction representation, denoted by x = ht+1 ∈ Rdvlm , and produce a visual space prediction ŷt+1 ∈ Rdvis . The projector is Proj(x) = W2 σ(LN(W1 x)) , implemented as Linear → LayerNorm → ReLU → Linear. In all experiments, we set dvlm = 2048 and dvis = 2048.

26