arXiv:2606.07244v1 [cs.RO] 5 Jun 2026
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation Haoxiang Shi
Xiang Deng
Haoyu Zhang
Harbin Institute of Technology (Shenzhen) Shenzhen, China Pengcheng Laboratory Shenzhen, China [email protected]
Harbin Institute of Technology (Shenzhen) Shenzhen, China [email protected]
Harbin Institute of Technology (Shenzhen) Shenzhen, China Pengcheng Laboratory Shenzhen, China [email protected]
Qiaohui Chu
Yaowei Wang
Liqiang Nie
Harbin Institute of Technology (Shenzhen) Shenzhen, China Pengcheng Laboratory Shenzhen, China [email protected]
Harbin Institute of Technology (Shenzhen) Shenzhen, China Pengcheng Laboratory Shenzhen, China [email protected]
Harbin Institute of Technology (Shenzhen) Shenzhen, China [email protected]
Node-Centric Waypoint Navigation
Trajectory Waypoint Navigation
Go past the dining room
Go past the dining room
and into the living room.
and into the living room. Trajectory Waypoint Predictor
Waypoint Predictor
Trajectory Candidates
Isolated Waypoints
Waypoint Navigator
Node-Level Planning Inconsistent Execution
Trajectory-Enhanced Navigator
Trajectory-Level Planning Consistent Execution
Figure 1: Comparison of navigation paradigms. Traditional node-centric methods (left) predict isolated waypoints, decoupling planning from control and causing inconsistent execution. Conversely, our Trajectory Waypoints paradigm (right) generates continuous trajectory candidates. This enables trajectory-level planning that explicitly evaluates path geometry, tightly coupling high-level planning with low-level execution.
Abstract Vision-Language Navigation in Continuous Environments (VLNCE) requires agents to follow natural-language instructions while navigating in real-world-like environments. Most VLN-CE approaches adopt a three-stage framework: a waypoint predictor proposes navigable waypoints, and a navigator selects the best waypoint,
with a low-level controller executing the movement to it. However, this decoupled paradigm often leads to unreachable waypoints or inconsistencies between planning and control. In this work, instead of predicting isolated waypoints, we introduce a novel paradigm called Trajectory Waypoint, which grounds each candidate waypoint in an executable trajectory. To realize this, we design a Trajectory
Arxiv Preprint, 2026,
Waypoint Predictor formulated as a TSDF-guided diffusion policy, which steers trajectory generation away from obstacles, inherently ensuring the reachability of the predicted waypoints. We further propose a trajectory-enhanced navigator that injects the associated trajectory as additional information for planning, enabling strict consistency between high-level semantic decisions and lowlevel execution. Extensive experiments on the VLN-CE benchmark show that our Trajectory Waypoint paradigm achieves superior performance over the baselines.
CCS Concepts • Computing methodologies → Robotic planning; Motion path planning; Scene understanding.
Keywords Vision-Language Navigation, Trajectory Generation, Waypoint
1
Introduction
Vision-Language Navigation (VLN) is an embodied navigation task in which an agent needs to follow language instructions grounded in observations to reach a specified goal location [3]. The agent continuously perceives its surroundings and makes sequential action decisions under partial observability, requiring it to align linguistic cues with scene understanding and to plan over long horizons in different environments. Within this setting, waypoints are a fundamental tool for structuring navigation. A waypoint is an intermediate navigational sub-goal that is spatially reachable in the environment and can be used to divide a long trajectory into shorter and more manageable segments. This decomposition provides an explicit interface between high-level planning and low-level control. In VLN, waypoints naturally serve as intermediate targets that guide the agent’s progress through the environment[7]. By converting continuous navigation into a sequence of waypoint-conditioned decisions, an agent can plan locally while maintaining global consistency with the instruction, thereby reducing the search space and mitigating the difficulty of long-horizon decision making in complex scenes. Vision-Language Navigation in Continuous Environments (VLNCE) [23] typically relies on waypoint-based methods operating under a hierarchical framework. First, a waypoint predictor generates a set of node-centric waypoints in the local environment [16]. Second, a high-level navigator selects the optimal waypoint that aligns semantically with the natural language instruction. Finally, a low-level controller drives the agent to execute actions toward the selected target [2]. Recent waypoint predictors have become increasingly stronger by injecting richer spatial priors into the proposal stage. Typical improvements include leveraging explicit geometry or occupancy cues to better identify traversable regions, and adopting stronger visual representations from large-scale pretraining to improve local navigability estimation [2, 31]. Despite these advances, two major issues still remain. First, the candidate distribution remains suboptimal; predicted node-centric waypoints often overlap with obstacles or non-traversable regions, inevitably introducing unreachable targets. Second, the decoupling of waypoint selection and action execution leads to planning–control
Haoxiang Shi et al.
inconsistencies, where the agent’s executed endpoint could deviate from the planned waypoint. An intuitive alternative to address these issues is to generate executable trajectories directly [35], bypassing the decoupled process of predicting node-centric waypoints and executing them via a separate controller. Recent progress in generative models provides a promising foundation for this approach. Such models have demonstrated a remarkable ability to generate temporally coherent action chunks[4, 5, 39]. In the navigation domain, works like NoMaD [32] have shown that a diffusion-based policy trained on demonstration trajectories can generate collision-free and multi-modal trajectories conditioned on observations, enabling controllable trajectory prediction. Inspired by these advances, we move beyond waypointbased methods toward a trajectory-centric waypoint paradigm, which naturally alleviates the limitations of hierarchical designs in VLN-CE. In this work, we propose a novel paradigm called Trajectory Waypoint, which moves beyond conventional node-centric waypoints to generate trajectory candidates. Instead of predicting spatially isolated coordinates, our method generates a diverse set of controllable, collision-free trajectories originating from the agent’s current position. Each candidate thus encapsulates not merely a spatial destination, but an entire executable path, inherently ensuring that every proposed candidate is physically reachable by construction. To generate these executable trajectory candidates, we propose the Trajectory Waypoint Predictor, formulating it as a diffusion policy [13] that supports multi-sample generation. To promote safety, we introduce TSDF-based cost guidance [43] that biases the sampling process away from obstacles, using a local signeddistance field to penalize trajectory samples approaching collision. The resulting guided sampling yields a diverse set of navigable, low-risk trajectories that explore different directions around the agent. Furthermore, we design a trajectory-enhanced navigator that capitalizes on the rich geometric information embedded in these trajectory candidates. Unlike traditional node-centric waypoints that provide only coarse, point-wise guidance and lack path geometry, our complete trajectories offer dense spatiotemporal context. To fully exploit this advantage, we condition the planning module directly on these trajectories, explicitly encoding both their spatial geometry and temporal structure. This formulation allows the navigator to evaluate not merely the final destination, but specifically how it gets there, enabling tight alignment between the path geometry and fine-grained instruction semantics. We comprehensively validate our Trajectory Waypoint paradigm on the VLN-CE benchmark. Crucially, within the VLN-CE scenes, Our Trajectory Waypoint Predictor significantly outperforms existing waypoint predictors in target reachability, achieving a substantially higher %Open. Building on these highly reliable trajectory candidates, extensive experiments demonstrate that our overall framework significantly outperforms the baselines on the downstream VLN-CE tasks. In this work, our main contributions are as follows:
(1) Unlike existing waypoint-based paradigms for VLN, we propose a trajectory-centric navigation paradigm named Trajectory
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
Waypoint, replacing traditional node-centric waypoint prediction with the generation of executable trajectory candidates. Compared to prior methods, this framework improves sub-goal feasibility and resolves planning-execution inconsistency. (2) We further develop a trajectory-enhanced navigator that plans among trajectory candidates, rather than isolated node-centric waypoints, by conditioning the navigator on enriched trajectory information. This design enables instruction-conditioned path selection while tightly coupling high-level planning with lowlevel execution. (3) We demonstrate that the proposed Trajectory Waypoint paradigm consistently improves performance on the VLN-CE benchmark, yielding robust gains over the baselines.
2 Related Work 2.1 Waypoint-Based Methods in VLN Early VLN research predominantly operated within discrete environments, modeling navigation as a node selection problem on predefined connectivity graphs [6, 8, 10, 11, 15, 50, 52]. As the field transitioned to Vision-Language Navigation in Continuous Environments (VLN-CE), directly transferring these discrete strategies proved unstable due to the extended action horizons required for continuous control [9, 14, 21, 23]. To bridge this gap, waypoint predictors were introduced [16] to dynamically generate local connectivity graphs, effectively re-discretizing the continuous space. This established the foundation for the waypoint-based methods, allowing high-level planners to select discrete targets that are subsequently executed by low-level controllers. Recently, Vision-Language Models (VLMs) have been integrated to introduce powerful zero-shot reasoning capabilities to the field [20, 25, 28, 29, 42, 45, 49, 51]. Despite these semantic advancements, these models remain structurally constrained by the legacy of nodecentric waypoints. Lacking fine-grained geometric comprehension, they frequently propose visually salient but physically unreachable sub-goals. Because planning is segregated from execution, this reliance on isolated waypoints inevitably induces a severe planningcontrol inconsistency, particularly in obstacle-dense environments.
2.2
Generative Models in VLN
To bypass the limitations of node-centric waypoints, recent efforts have explored end-to-end generative control paradigms. Driven by advancements in generative models, these frameworks demonstrate that temporal visual context can be directly mapped to low-level control signals, effectively internalizing spatial reasoning within the policy [40, 46, 47]. Concurrently, world models and VisionLanguage-Action (VLA) models have emerged in the VLN domain [19, 24, 27, 39, 41, 44]. These approaches utilize VLMs for high-level semantic foresight while employing lightweight generative models to synthesize smooth, collision-free trajectories, thereby resolving the horizon problem and ensuring physical consistency. Building upon these advancements, our work introduces a novel trajectory-centric waypoint paradigm. Unlike legacy waypointbased methods that suffer from geometric unreachability, our framework directly predicts fully executable trajectory candidates. This continuous representation fundamentally transforms how the agent perceives spatial transitions. By evaluating complete geometric
Arxiv Preprint, 2026,
paths rather than isolated coordinates, we tightly align high-level navigation planning with low-level physical control, successfully grounding the interpretability of hierarchical planning in the executable path.
3 Preliminary 3.1 Task Formulation & Waypoint-Centric Paradigm The VLN-CE task is formulated as a Partially Observable Markov Decision Process where an agent navigates a continuous 3D space Cfree guided by a natural language instruction I. At each step 𝑡, the agent perceives the environment through a panoramic observation 𝑂𝑡 , consisting of 12 RGB-D image pairs captured at 30◦ intervals following standard protocols: 𝐷𝑒𝑝𝑡ℎ
𝑅𝐺𝐵 𝑂𝑡 = {(𝐼𝑡,𝜃 , 𝐼𝑡,𝜃
) | 𝜃 ∈ {0◦, 30◦, . . . , 330◦ }}.
(1)
This observation is then mapped to an atomic action 𝑎𝑡 from the discrete action space A = {forward, turn_left, turn_right, stop}. To bridge the gap between high-level semantic instructions and low-level continuous execution, existing works predominantly adopt a waypoint-centric paradigm. Inheriting the connectivitygraph structure from legacy discrete environments, this paradigm abstracts navigation into a tractable node selection task. Specifically, a trained waypoint predictor discretizes the egocentric panorama into a polar grid and learns a navigability distribution 𝑝𝜃 (𝑤𝑡 |𝑠𝑡 ): exp(𝑓𝜃 (𝑤𝑡(𝑘 ) , 𝑠𝑡 )) 𝑝𝜃 (𝑤𝑡(𝑘 ) |𝑠𝑡 ) = Í , (𝑗) 𝑗 exp(𝑓𝜃 (𝑤 𝑡 , 𝑠𝑡 ))
(2)
to generate a local set of node-centric waypoints W𝑡 . Then a highlevel navigator selects the optimal sub-goal 𝑤ˆ 𝑡 ∈ W𝑡 , which is executed by a deterministic low-level controller 𝜋low (e.g., a “Rotatethen-Forward” heuristic) to translate the selected sub-goal.
3.2
Limitations of the Waypoint-Centric Paradigm
Fundamentally bound by the connectivity-graph abstraction, this waypoint-centric paradigm suffers from two critical deficiencies in continuous environments: (1) Geometric Unreachability: By reducing continuous space to isolated nodes, predictors often ignore complex 3D geometric constraints. A predicted node-centric waypoint is frequently physically invalid (e.g., located inside obstacles or across nontraversable gaps). Formally, defining the physically reachable set as 𝑊𝑅 ⊂ Cfree , this point-centric abstraction often yields highly-scored but unreachable targets: ∃𝑤 ∈ W𝑡 s.t. 𝑝𝜃 (𝑤 |𝑠𝑡 ) > 𝛿 ∧ 𝑤 ∉ 𝑊𝑅 .
(3)
Selecting such a sub-goal forces the underlying controller into terminal collisions. (2) Planning-Control Inconsistency: Waypoint-based methods strictly decouple the semantic planning of 𝑤ˆ 𝑡 from the physical control 𝜋 low . The planner assumes a direct, unobstructed path to 𝑤ˆ 𝑡 . However, the reactive low-level controller generates an actual execution trajectory Texec that often diverges significantly
Arxiv Preprint, 2026,
Haoxiang Shi et al.
to avoid obstacles, resulting in Texec (𝜋low, 𝑤ˆ 𝑡 ) ≠ 𝑤ˆ 𝑡 .
(4)
This disconnect violates instruction semantics (e.g., swinging wide instead of “passing near the table”), inevitably leading to catastrophic error accumulation over long horizons.
4
Trajectory Waypoint
To address the aforementioned challenges, we propose Trajectory Waypoint, a novel trajectory-centric waypoint paradigm that shifts the atomic unit of VLN from node-centric waypoints to executable trajectory candidates. As illustrated in Figure 2, our framework consists of two stages: a Trajectory Waypoint Predictor (TWP) that generates geometrically feasible trajectory candidates, and a Trajectory-Enhanced Navigator (TEN) that grounds these trajectories in the navigation instruction.
4.1
Trajectory Waypoint Predictor
The Trajectory Waypoint Predictor (TWP) is designed to approximate the distribution of feasible trajectory candidates conditioned on the current observation. Formally, we model a trajectory candidate 𝜏 as a sequence of 𝑇 relative displacements: 𝜏 = {Δ𝑥 1, Δ𝑥 2, . . . , Δ𝑥𝑇 },
Δ𝑥𝑖 ∈ R2,
(5)
where 𝑇 denotes the prediction horizon. Accordingly, the Trajectory Waypoint is defined as the cumulative sum of these displacements from the current state 𝑥𝑡 : traj
𝑤𝑡
= 𝑥𝑡 +
𝑇 ∑︁
Δ𝑥𝑖 ,
(6)
𝑖=1
where 𝑥𝑡 represents the agent’s current position. Observation Encoding: We process the panoramic observation 𝑂𝑡 using a dual-stream architecture to capture complementary semantic and geometric information. For visual inputs, we employ DINOv3 to extract robust object-level semantic features. In parallel, depth images are encoded by a ResNet-50 pre-trained with DDPPO to extract occupancy and geometric cues. To fuse the RGB and depth features, we introduce a Feature Fusion Module inspired by [31]. The fused representations are subsequently processed by a Transformer-based panorama encoder to capture global context, resulting in the final unified observation 𝑡 embedding 𝐹 env ∈ R12×𝐻 . Trajectory Waypoint Location Estimation: Instead of generating trajectory candidates blindly in the continuous space, we first predict coarse navigational intentions to anchor the subsequent generation process. We employ a lightweight MLP-based predictor 𝑡 that maps the observation embedding 𝐹 env to a joint probability distribution over discretized movement actions: 𝑡 𝑃loc = Softmax(MLPloc (𝐹 env )) ∈ R |Θ| × |𝑆 | ,
(7)
where |Θ| and |𝑆 | denote the number of discretized direction bins and step count categories, respectively. From this distribution, we select the top-𝐾 candidates to form a seed set: Wseed = {(𝜃ˆ𝑘 , 𝑠ˆ𝑘 ) | 𝑘 = 1, . . . , 𝐾 }. (8) 𝑘 ˆ Each sampled intention 𝑝 loc = (𝜃 𝑘 , 𝑠ˆ𝑘 ) serves as a location prior for generating a corresponding trajectory candidate.
Trajectory Waypoint Generation via Environment-Guided Diffusion Policy: We formulate the generation of trajectory candidates as a conditional modeling problem, employing a diffusion policy to model the distribution 𝑝𝜙 (𝜏 |𝑆𝑡𝑘 ). To bridge the discrete location prior with continuous trajectory generation, we construct a condition state 𝑆𝑡𝑘 for each selected candidate (𝜃ˆ𝑘 , 𝑠ˆ𝑘 ). Specifically, 𝑘 we retrieve the sector-wise visual feature 𝐹 target —aligned with the ˆ predicted direction 𝜃 𝑘 —from the global environment embedding 𝑡 , and fuse it with the positional embedding of the prior: 𝐹 env 𝑘 𝑆𝑡𝑘 = 𝐹 target ⊕ 𝐸 loc (𝜃ˆ𝑘 , 𝑠ˆ𝑘 ).
(9)
This state 𝑆𝑡𝑘 guides the diffusion process to generate physically feasible paths toward the intended region. The diffusion process consists of a forward pass that incrementally adds Gaussian noise to a ground-truth trajectory 𝜏0 , and a learnable reverse process that recovers a feasible trajectory from pure noise 𝜏𝑀 ∼ N (0, 𝐼 ). We employ a denoising network 𝜖𝜙 to predict the noise component at each diffusion step 𝑖. The trajectory is iteratively refined via the following update rule: 1 1 − 𝛼𝑖 𝜏𝑖 −1 = √ 𝜏𝑖 − √ 𝜖𝜙 (𝜏𝑖 , 𝑖, 𝑆𝑡𝑘 ) + 𝜎𝑖 z, (10) 𝛼𝑖 1 − 𝛼¯𝑖 where 𝑖 denotes the diffusion step from 𝑀 down to 1, z ∼ N (0, 𝐼 ), ¯ 𝜎 are noise schedule parameters. By performing this reand 𝛼, 𝛼, verse sampling process once for each of the top-𝐾 priors, we gen𝐾 , each with erate a set of 𝐾 diverse trajectory candidates {𝜏 (𝑘 ) }𝑘=1 a fixed prediction horizon 𝑇 , covering the most promising navigational paths. Affordance Constraints for Trajectory Waypoint: To rigorously ensure the geometric reachability of the predicted trajectory 𝐾 , we incorporate a physical validity constraint candidates {𝜏 (𝑘 ) }𝑘=1 during the diffusion sampling process. Inspired by [43], we construct a local Truncated Signed Distance Field (TSDF) map 𝑀TSDF from the current depth observation. We define a differentiable collision cost Jsafe that penalizes trajectory points 𝑝𝑖 falling within the truncation distance of an obstacle: Jsafe (𝜏) =
𝑇 ∑︁
max(0, 𝜖 − 𝑀TSDF (𝑝𝑖 )).
(11)
𝑖=1
During each denoising step 𝑖, we apply inference-time guidance by modifying the predicted mean using the gradient of this safety cost: guided 𝜏𝑖 −1 = 𝜏𝑖 −1 − 𝜆∇𝜏 Jsafe (𝜏). (12) This effectively “pushes” the generated trajectory away from obstacles via the TSDF gradient, ensuring that the final trajectory 𝜏0 lies strictly within the reachable set 𝑊𝑅 . Adaptive Trajectory Truncation: The standard diffusion policy generates trajectories with a fixed horizon 𝑇 , which implicitly assumes that endpoints are distributed at rigid intervals. However, diverse navigation scenarios often require flexible action horizons to match local geometric constraints. To mitigate this rigidity, we implement a dynamic pruning mechanism guided by the predicted step count 𝑠ˆ𝑘 from the location prior 𝑝 𝑘loc . We truncate the generated trajectory to this estimated optimal length: 𝑘 𝜏final = {Δ𝑥 1𝑘 , . . . , Δ𝑥𝑠𝑘ˆ𝑘 }.
(13)
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
Arxiv Preprint, 2026,
Stage 1: Trajectory Waypoint Predictor
Task Input
Step 2: Waypoint Prior
Step 1: Perception
Output
Step 3: Trajectory Generation
Dino_v3
DDPPO Res-50
Feature Fusion Module
Observation Features (Fenv )
Diffusion Policy
Location MLP Waypoints መ 𝑠Ƹ Prior 𝜃,
Panoramic RGB
Predicted Trajectories(𝜏𝑓𝑖𝑛𝑎𝑙 )
Predicted Step( 𝑠Ƹ )
Stage 2: Trajectory-Enhanced Navigator Step 1: Trajectory Waypoint Encoding
Step 2: Trajectory-Enhanced Map
Map Feature
Visited Nodes
Panoramic Depth Instruction: Exit from the living room.
Trajectory Waypoints
Global Topo Map
Semantic Edge
Current Node
Visual Feature
Position Embeds
Step Embeds
…
Fused Feature
… Trajectory Embedding ( E𝜏 ) Metric Grid
Continuous movement Execution
Map Features
Text Encoder
Local Metric Map
(Topo+Metric)
Step 3: Cross-Modal Policy Dual-Branch Transformer
Trajectory Score
0.65
…
0.93
Execute Best Trajectory Candidate
Figure 2: Overview of the proposed Trajectory Waypoint framework. The Trajectory Waypoint Predictor (Stage 1) generates diverse, physically feasible trajectory candidates via environment-guided diffusion. The Trajectory-Enhanced Navigator (Stage 2) then evaluates these continuous paths within a hybrid map, directly coupling high-level semantic reasoning with low-level execution.
This operation transforms the fixed-length output into variablelength trajectory candidates, ensuring the final physical endpoint traj (𝑤𝑡 ) aligns precisely with the environment’s affordances.
4.2
Trajectory-Enhanced Navigator
Unlike traditional navigators that merely reason about where to go, our Trajectory-Enhanced Navigator (TEN) reasons about how to get there. We achieve this by embedding the generated trajectory candidates into a topo-metric hybrid map, explicitly encoding both path geometry and semantics. Visually-Grounded Trajectory Encoding: The generated tra(𝑘 ) 𝐾 jectory candidates {𝜏final }𝑘=1 initially lack semantic context. To enable instruction-guided reasoning, we ground these paths in a local metric map M𝑡 ∈ R𝐻 ×𝑊 ×𝐷 constructed from projected panoramic visual features. Similar to [1], panoramic visual features 𝐹 rgb are projected into the grid cells of M𝑡 utilizing point clouds derived from the depth map. For a candidate trajectory 𝜏final = {𝑝 0, . . . , 𝑝𝑠ˆ }, we retrieve the visual feature 𝐼𝑖 at each step coordinate 𝑝𝑖 . We then fuse this visual context with positional and temporal information via learnable embeddings: ℎ𝑖 = 𝐼𝑖 + 𝑒 pos (𝑝𝑖 ) + 𝑒 step (𝑖),
(14)
where 𝑒 pos (·) and 𝑒 step (·) denote the position and time step embedding layers, respectively. The final trajectory embedding is represented as the sequence E𝜏 = {ℎ 0, ℎ 1, . . . , ℎ𝑠ˆ }, where the final hidden state ℎ𝑠ˆ serves as the trajectory endpoint feature ℎ 𝑤 .
Trajectory-Enhanced Map Representation: We maintain a topo-metric hybrid map M𝑡 = ⟨G𝑡 , Vmetric ⟩ for comprehensive scene representation. Unlike legacy waypoint-based methods that treat map nodes as isolated spatial coordinates, we actively enhance them using the path-dependent features of our trajectory candidates. For the topological graph G𝑡 , the candidate endpoints serve as ghost nodes, which represent the potential sub-goals and are connected to the current node via edges defined by 𝜏final . We use a learnable query vector 𝑞 traj to aggregate the sequential trajectory embedding E𝜏 into a fixed-dimensional edge representation 𝑒𝜏 : 𝑒𝜏 = Attention(𝑄 = 𝑞 traj, 𝐾 = E𝜏 , 𝑉 = E𝜏 ).
(15)
This trajectory edge feature explicitly captures the navigational cost and semantics of the path. We then inject it into the visual embedding 𝑣 node of the corresponding ghost node: 𝑣ˆnode = 𝑣 node + 𝑒𝜏 .
(16)
Simultaneously, we apply a parallel trajectory-to-cell enhancement mechanism to the local metric map Vmetric , enriching the dense spatial grid with path-aware semantics. Instruction-Grounded Trajectory Selection: To derive the final navigation planning, we feed the trajectory-enhanced hybrid map M𝑡 and the text-encoded instruction I into a dual-branch cross-modal Transformer. Specifically, the topological and metric representations are processed using Graph-Aware Self-Attention (GASA) and standard self-attention, respectively, producing the
Arxiv Preprint, 2026,
Haoxiang Shi et al.
features based on the instruction 𝑁𝑡 = {𝑛 1, . . . , 𝑛𝐾 } and 𝑀𝑡 = {𝑚 1, . . . , 𝑚𝐾 }. Subsequently, two Feed-Forward Networks compute a navigability score for each candidate trajectory based on these features. The agent then selects the optimal candidate via a Softmax distribution over these scores. traj
𝑃 (𝑤𝑡,𝑘 |I, M𝑡 ) = Softmax(Score(𝑛𝑘 , 𝑚𝑘 )).
(17)
Crucially, because our framework outputs a fully specified trajectory candidate, the agent directly executes the continuous movebest . This tight coupling between high-level semantic planment 𝜏final ning and low-level control inherently eliminates the planningexecution inconsistency prevalent in legacy waypoint-based methods.
• 𝒅𝒉 (Hausdorff Distance): Captures the worst-case geometric error by measuring the maximum distance from any point in the predicted candidate set to the nearest point in the ground truth. Navigation Metrics: For the downstream VLN-CE task, we report standard evaluation metrics established by [23]. • Success Rate (SR): The fraction of episodes where the agent successfully halts within 3.0 m of the target location. • Oracle Success Rate (OSR): The fraction of episodes where the agent comes within 3.0 m of the target at any point during its executed trajectory. • Success weighted by Path Length (SPL): Assesses navigation efficiency by penalizing unnecessarily long paths: SPL = SR ×
5 Experiment 5.1 Experiment Setup In this section, we present a comprehensive evaluation of the proposed Trajectory Waypoint paradigm. We primarily conduct experiments on the VLN-CE benchmark using Habitat simulator. 5.1.1 Training Details. TWP Training: A critical priority when training TWP is mitigating geometric overfitting. Previous works typically rely solely on the Matterport3D (MP3D) dataset, which provides only 61 building-scale environments for training. While visually diverse, this limited number of unique room configurations restricts model generalization. To address this, we significantly scale our training data by augmenting MP3D with the Habitat-Matterport 3D (HM3D) dataset, contributing an additional 794 scenes with superior visual fidelity and geometric complexity. Following the data generation protocol in [38], we sample pre-defined waypoints across HM3D and utilize the Habitat PathFinder to extract continuous demonstration trajectories connecting these points. In total, we constructed a robust dataset comprising over 120,000 waypointtrajectory pairs to train the TWP. TEN Training: For the Trajectory-Enhanced Navigator (TEN), we adopt the proven two-stage training paradigm introduced by [2]. The navigator is first pre-trained on the discrete Room-toRoom (R2R) dataset[3] to acquire foundational visual-linguistic alignment. Subsequently, we fine-tune the model online within the continuous Habitat simulator using the DAgger (Dataset Aggregation) algorithm, allowing the navigator to effectively adapt to our trajectory-based navigation paradigm. 5.1.2 Metrics. Predictor Quality Metrics: To comprehensively evaluate the generated trajectory candidates, we adopt standard proposal metrics from [31] and incorporate geometric evaluations specifically tailored to continuous paths. • |𝚫|: The absolute difference between the number of predicted candidates and the ground-truth waypoints. • %Open: The percentage of predicted trajectory points that fall strictly within the navigable mesh (free space). A higher %Open indicates superior geometric feasibility and safety. • 𝒅 𝒄 (Chamfer Distance): Measures the average geometric divergence between the predicted trajectory waypoint set and the ground-truth waypoints.
𝐿gt , max(𝐿gt, 𝐿pred )
(18)
where 𝐿gt and 𝐿pred denote the lengths of the shortest path and the agent’s actual path, respectively. • Collision Rate (CR): Defined as the average number of collisions encountered per action step.
5.2
Trajectory Waypoint Predictor
Table 1: Performance comparison of waypoint predictors on the scenes from the VLN-CE val-unseen split. Best results are in bold (↓: lower is better, ↑: higher is better). VLN-CE Val-Unseen #
Model |Δ|
%Open ↑
𝑑𝑐 ↓
𝑑ℎ ↓
1 2 3 4 5
Baseline U-Net[30] RecBERT[16] ETPNav[2] SmartWay[31]
1.37 1.21 1.40 1.39 1.41
80.18 52.54 79.86 84.05 87.26
1.08 1.01 1.07 1.04 1.03
2.16 2.00 2.00 2.01 1.96
6
TWP(Ours)
1.47
95.84
0.54
1.95
5.2.1 Main Results in VLN-CE Val-Unseen Scenes. Table 1 evaluates the performance of various predictors on the scenes from the VLNCE val-unseen split. The proposed TWP demonstrates a pronounced advantage over existing waypoint predictors in generating safe and highly actionable targets. Most notably, TWP achieves a state-ofthe-art %Open of 95.84, surpassing the strongest prior baseline by a remarkable absolute margin of 8.58. This substantial gain in reachability confirms that our trajectory candidates can reliably navigate continuous free space without colliding with unmapped obstacles. Beyond basic feasibility, TWP exhibits highly precise geometric alignment with optimal human-annotated waypoints. It dramatically reduces the Chamfer Distance (𝑑𝑐 ) to 0.54, nearly halving the error of prior methods, and attains the lowest Hausdorff Distance (𝑑ℎ ) of 1.95. Collectively, these metrics substantiate that our trajectory-centric waypoint predictor generates candidates that are not only inherently safe but also geometrically precise.
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
Table 2: Ablation study of the Trajectory Waypoint Predictor. RGB Encoder
TG
%Open ↑
𝑑𝑐 ↓
𝑑ℎ ↓
ResNet-50
✗ ✓
86.87 94.96
1.05 0.64
2.16 1.96
Dinov3
✗ ✓
88.34 95.84
0.87 0.54
2.06 1.95
Arxiv Preprint, 2026,
Table 3: Comparison with state-of-the-art methods on the R2R-CE Val-Seen and Val-Unseen splits. * indicates VLMbased methods. † denotes baselines utilizing the waypoint predictor from [16]. R2R-CE Val-Seen
5.2.2 Ablation Study for TWP. To isolate the driving factors behind our performance, we ablate the core components of the TWP in Table 2. Visual Encoder: Upgrading the visual backbone from ResNet50 to DINOv3 yields consistent geometric improvements. Even without TSDF guidance, swapping to DINOv3 increases %Open from 86.87 to 88.34 and noticeably reduces the spatial alignment error (𝑑𝑐 ) from 1.05 to 0.87. This confirms that DINOv3’s robust, fine-grained feature representations provide a fundamentally superior spatial understanding for baseline trajectory generation. TSDF Guidance (TG): Explicitly incorporating TG triggers the most dramatic performance leap across all configurations. For our best DINOv3 setup, applying TG elevates %Open from 88.34 to a peak of 95.84, while drastically cutting 𝑑𝑐 to 0.54. This empirically validates the critical role of our inference-time guidance: it actively repels generated trajectory candidates from obstacles and invalid zones, strictly expanding the safety margin and ensuring the physical feasibility of continuous traversal. 5.2.3 Qualitative Experiments for TWP. To rigorously assess the zero-shot generalization ability for our TWP, we densely sample and predict candidates across diverse scenes in the VLN-CE ValUnseen split. As illustrated in Figure 3, traditional predictors frequently propose invalid node-centric waypoints situated within non-traversable regions (indicated by blue nodes). Furthermore, their reliance on heuristic “turn-then-forward” controllers inevitably leads to execution inconsistencies, where the agent’s actual action execution diverges from the planned semantic goal. Conversely, guided by the TSDF-based gradient, our TWP successfully generates dense, collision-free trajectory candidates. These generated paths exhibit profound spatial consistency with ground-truth expert demonstrations, inherently guaranteeing kinematic execution fidelity for the downstream navigator. We provide more visualization results for different scenes in the Appendix. To further demonstrate the reliability of TWP under complex geometric constraints, Figure 4 isolates a challenging “visible but unreachable” scene, a compact church environment bisected by a low wall. Because the region behind the wall falls within the agent’s panoramic visual stream but is physically inaccessible, traditional predictors lacking intrinsic 3D comprehension consistently propose unfeasible sub-goals that penetrate the solid boundary, inevitably causing terminal collisions. In stark contrast, our TWP intrinsically integrates spatial TSDF priors during the stochastic diffusion phase. This continuous gradient guidance actively exerts a repulsive spatial force, steering the trajectory candidates strictly away from the impenetrable barrier. This confirms that our approach can reliably yield high-quality, physically executable paths even when subjected to highly deceptive spatial constraints.
R2R-CE Val-Unseen
Method NE ↓ OSR ↑ SR ↑ SPL ↑ NE ↓ OSR ↑ SR ↑ SPL ↑ Navid*[47] 5.62 51.5 43.5 38.4 Uni-Navid*[46] 4.40 64.5 59.1 55.4 MonoDream*[34] MapNav*[48] Dynam3D*[36] NaVILA*[12] 5.05 63.8 56.4 51.5
5.72 5.28 5.45 4.93 5.34 5.22
49.2 53.3 61.5 53.0 62.1 62.5
41.9 36.5 47.0 42.7 55.8 49.1 39.7 37.2 52.9 45.7 54.0 49.0
Sim2Sim† [22] 4.67 HPN+DN[21] 5.48 Ego2 -Map[18] CM2 [14] 4.81 VLN⟳BERT† [17] 5.74 GridMM† [37] 4.21 DWalker† [33] 5.53 ETPNav† [2] 3.95 BEVBert† [1] 3.77 Energy† [26] 3.90
6.07 6.31 4.94 6.23 5.74 5.11 5.54 4.71 4.57 4.69
52.0 40.0 59.0 41.3 53.0 61.0 59.0 65.0 67.0 65.0
43.0 36.0 36.0 34.0 52.0 46.0 37.0 30.6 44.0 39.0 49.0 41.0 49.0 44.0 57.0 49.0 59.0 50.0 58.0 50.0
Ours
5.3
61.0 53.0 58.3 59.0 69.0 66.0 72.0 73.0 73.0
52.0 44.0 46.0 43.0 52.8 41.8 50.0 44.0 59.0 51.0 59.0 48.0 66.0 59.0 68.0 60.0 68.0 59.0
3.75 74.6 68.8 60.2 4.57 68.1 60.3 51.4
VLN-CE Downstream Task
5.3.1 Main Results in R2R-CE. Table 3 presents a comprehensive performance comparison on the R2R-CE benchmark. We evaluate across both Val-Seen and Val-Unseen splits to rigorously assess domain-specific learning and zero-shot generalization. Notably, the baselines are categorized into traditional small-scale models and recent large-parameter models. As shown in Table 3, our method establishes a new state-ofthe-art on the challenging R2R-CE Val-Unseen split. By replacing traditional node-centric waypoints with geometrically grounded trajectory candidates, our agent attains the highest OSR of 68.1, SR of 60.3 and SPL of 51.4, which highlights that our approach not only navigates to the target reliably but does so through optimal, succinct paths without redundant exploration. Furthermore, results on the Val-Seen split reinforce the robust learning capacity of our model. Benefiting from the tight coupling of high-level planning and low-level execution, our method achieves the absolute highest SR of 68.8, alongside a peak SPL of 60.2 and an Oracle Success Rate (OSR) of 74.6. Notably, our approach strictly dominates both traditional baselines relying on legacy waypoint predictors and recent VLM-based architectures. This empirically validates the inherent superiority of our trajectory-centric waypoint paradigm in continuous domains. 5.3.2 Ablation Study for Waypoint Trajectory Framework. To systematically validate our structural design, Table 4 details an ablation study on the R2R-CE Val-Unseen split. Replacing our predictor with a standard baseline (w/o TWP) severely degrades the SR from 60.3 to 58.7 and SPL from 51.4 to 49.1, accompanied by a massive surge in the CR. This confirms that our geometrically feasible trajectory candidates are the primary driver for safe, collision-free navigation
Arxiv Preprint, 2026,
Haoxiang Shi et al.
Scene Mesh
Waypoint Predictor
TWP(Ours)
GT Trajectories
Figure 3: Qualitative results in the scenes from the VLN-CE val-unseen split. Blue points indicate targets generated in inaccessible regions, while yellow points represent physically accessible areas. Waypoint Predictor
Scene Mesh
TWP(Ours)
Figure 4: TWP performance under complex geometric constraints. The red point indicates the current location. Blue points indicate targets generated in inaccessible regions, while yellow points represent physically accessible areas. Table 4: Ablation study of our Trajectory Waypoint framework on the R2R-CE Val-Unseen split. R2R-CE Val-Unseen Method Ours w/o TEN w/o TSDF w/o TWP
NE ↓
OSR ↑
SR ↑
SPL ↑
CR ↓
4.54 4.68 4.67 4.75
68.1 66.8 67.1 65.2
60.3 59.3 59.2 58.7
51.4 50.3 50.7 49.1
0.004 0.004 0.05 0.12
and overall performance. Furthermore, substituting our TrajectoryEnhanced Navigator with a traditional map fusion module (w/o TEN) noticeably drops SR to 59.3 and SPL to 50.3, while leaving the CR largely unaffected since TEN operates purely at the highlevel semantic planning stage. This gap highlights a critical flaw in
legacy systems: fundamentally bound by the connectivity-graph’s point-centric waypoint navigation paradigm, they merely extract isolated coordinates and fail entirely to exploit the dense continuous context embedded within our trajectories. Finally, removing the inference-time TSDF guidance (w/o TSDF) causes a consistent decline across all metrics. Without this continuous gradient guidance acting as the physical constraint, a portion of the generated paths inevitably drifts into unreachable regions, directly increasing the CR and impairing overall reachability. 5.3.3 Effect of TSDF-Guide Steps in the Denoising Process. During the trajectory generation phase of our TWP, we incorporate a TSDF-guide into the diffusion denoising process to steer the candidate paths away from obstacles. To fully investigate the impact of the number of guidance steps, we conduct the ablation study by applying the TSDF-guide strictly for the first 𝑛 steps out of the
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
61.5
SR CR
0.045
Collision Rate (CR)
Success Rate (SR) %
61.0 60.5
0.030
60.0
0.015
59.5 59.0 58.5
0.060
0.000
0
1
2
3
TSDF Guidance Steps
4
5
Figure 5: Impact of TSDF-guide steps on navigation performance. total 𝑘 denoising steps. As shown in Figure 5, increasing the guidance steps consistently lowers the CR, proving its effectiveness in steering trajectories away from obstacles. However, the SR peaks at 3 steps and subsequently declines. This indicates a critical tradeoff: while moderate guidance improves trajectory safety, excessive guidance over-constrains the generative process, disrupting the diffusion model’s original denoising distribution and ultimately causing sub-optimal navigation performance.
6
Conclusion
In this paper, we introduce Trajectory Waypoint, a trajectorycentric paradigm for Vision-Language Navigation that resolves the unreachability and planning-control disconnect inherent in legacy waypoint-based methods. By replacing isolated waypoints with continuous, collision-free trajectories generated by our Trajectory Waypoint Predictor, and planning over these candidates via the Trajectory-Enhanced Navigator, we strictly align high-level planning with low-level execution. Extensive experiments on the R2R-CE benchmark confirm our approach significantly outperforms the baselines, providing a physically scalable foundation for robust embodied navigation.
References [1] Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, and Jing Shao. 2023. BEVBert: Multimodal Map Pre-training for Language-guided Navigation. Proceedings of the IEEE/CVF International Conference on Computer Vision (2023). [2] Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, and Liang Wang. 2024. Etpnav: Evolving topological planning for vision-language navigation in continuous environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024). [3] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. 2018. Vision-andlanguage navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3674–3683. [4] Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Robert Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, brian ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian LiBell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. 2025. $\pi_{0.5}$: a Vision-Language-Action Model with Open-World Generalization. In 9th Annual Conference on Robot Learning. https://openreview. net/forum?id=vlhoswksBO
Arxiv Preprint, 2026,
[5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. 2024. 𝜋 0 : A VisionLanguage-Action Flow Model for General Robot Control. CoRR abs/2410.24164 (2024). https://doi.org/10.48550/arXiv.2410.24164 [6] Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158 (2017). [7] Jiaqi Chen, Bingqian Lin, Xinmin Liu, Lin Ma, Xiaodan Liang, and Kwan-Yee K Wong. 2025. Affordances-oriented planning using foundation models for continuous vision-language navigation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 23568–23576. [8] Jiaqi Chen, Bingqian Lin, Ran Xu, Zhenhua Chai, Xiaodan Liang, and Kwan-Yee K Wong. 2024. Mapgpt: Map-guided prompting with adaptive path planning for vision-and-language navigation. arXiv preprint arXiv:2401.07314 (2024). [9] Peihao Chen, Dongyu Ji, Kunyang Lin, Runhao Zeng, Thomas Li, Mingkui Tan, and Chuang Gan. 2022. Weakly-supervised multi-granularity map learning for vision-and-language navigation. Advances in Neural Information Processing Systems 35 (2022), 38149–38161. [10] Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, and Ivan Laptev. 2021. History aware multimodal transformer for vision-and-language navigation. Advances in neural information processing systems 34 (2021), 5834–5847. [11] Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. 2022. Think global, act local: Dual-scale graph transformer for visionand-language navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 16537–16547. [12] An-Chieh Cheng, Yandong Ji, Zhaojing Yang, Zaitian Gongye, Xueyan Zou, Jan Kautz, Erdem Bıyık, Hongxu Yin, Sifei Liu, and Xiaolong Wang. 2024. Navila: Legged robot vision-language-action model for navigation. arXiv preprint arXiv:2412.04453 (2024). [13] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. 2023. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research (2023), 02783649241273668. [14] Georgios Georgakis, Karl Schmeckpeper, Karan Wanchoo, Soham Dan, Eleni Miltsakaki, Dan Roth, and Kostas Daniilidis. 2022. Cross-modal map learning for vision and language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15460–15470. [15] Weituo Hao, Chunyuan Li, Xiujun Li, Lawrence Carin, and Jianfeng Gao. 2020. Towards learning a generic agent for vision-and-language navigation via pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 13137–13146. [16] Yicong Hong, Zun Wang, Qi Wu, and Stephen Gould. 2022. Bridging the gap between learning in discrete and continuous environments for vision-and-language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 15439–15449. [17] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, and Stephen Gould. 2021. Vln bert: A recurrent vision-and-language bert for navigation. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition. 1643– 1653. [18] Yicong Hong, Yang Zhou, Ruiyi Zhang, Franck Dernoncourt, Trung Bui, Stephen Gould, and Hao Tan. 2023. Learning navigational visual representations with semantic map supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 3055–3067. [19] Junjun Hu, Jintao Chen, Haochen Bai, Minghua Luo, Shichao Xie, Ziyi Chen, Fei Liu, Zedong Chu, Xinda Xue, Botao Ren, et al. 2025. AstraNav-World: World Model for Foresight Control and Consistency. arXiv preprint arXiv:2512.21714 (2025). [20] Sungjune Kim, Gyeongrok Oh, Heeju Ko, Daehyun Ji, Dongwook Lee, Byung-Jun Lee, Sujin Jang, and Sangpil Kim. 2025. Test-time adaptation for online visionlanguage navigation with feedback-based reinforcement learning. In Forty-second International Conference on Machine Learning. [21] Jacob Krantz, Aaron Gokaslan, Dhruv Batra, Stefan Lee, and Oleksandr Maksymets. 2021. Waypoint models for instruction-guided navigation in continuous environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 15162–15171. [22] Jacob Krantz and Stefan Lee. 2022. Sim-2-sim transfer for vision-and-language navigation in continuous environments. In European conference on computer vision. Springer, 588–603. [23] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. 2020. Beyond the nav-graph: Vision-and-language navigation in continuous environments. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVIII 16. Springer, 104–120. [24] Guoxin Lian, Shuo Wang, Yucheng Wang, Yongcai Wang, Maiyue Chen, Kaihui Wang, Bo Zhang, Zhizhong Su, Deying Li, and Zhaoxin Fan. 2026. MapDream:
Arxiv Preprint, 2026,
Task-Driven Map Learning for Vision-Language Navigation. arXiv preprint arXiv:2602.00222 (2026). [25] Bingqian Lin, Yunshuang Nie, Ziming Wei, Yi Zhu, Hang Xu, Shikui Ma, Jianzhuang Liu, and Xiaodan Liang. 2024. Correctable landmark discovery via large models for vision-language navigation. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 12 (2024), 8534–8548. [26] Rui Liu, Wenguan Wang, and Yi Yang. 2024. Vision-language navigation with energy-based policy. Advances in Neural Information Processing Systems 37 (2024), 108208–108230. [27] Rui Liu, Wenguan Wang, and Yi Yang. 2024. Volumetric environment representation for vision-language navigation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 16317–16328. [28] Yuxing Long, Wenzhe Cai, Hongcheng Wang, Guanqi Zhan, and Hao Dong. 2024. Instructnav: Zero-shot system for generic instruction navigation in unexplored environment. arXiv preprint arXiv:2406.04882 (2024). [29] Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao. 2025. Vln-r1: Vision-language navigation via reinforcement fine-tuning. arXiv preprint arXiv:2506.17221 (2025). [30] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241. [31] Xiangyu Shi, Zerui Li, Wenqi Lyu, Jiatong Xia, Feras Dayoub, Yanyuan Qiao, and Qi Wu. 2025. SmartWay: Enhanced Waypoint Prediction and Backtracking for Zero-Shot Vision-and-Language Navigation. arXiv preprint arXiv:2503.10069 (2025). [32] Ajay Sridhar, Dhruv Shah, Catherine Glossop, and Sergey Levine. 2024. Nomad: Goal masked diffusion policies for navigation and exploration. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 63–70. [33] Hanqing Wang, Wei Liang, Luc Van Gool, and Wenguan Wang. 2023. Dreamwalker: Mental planning for continuous vision-language navigation. In Proceedings of the IEEE/CVF international conference on computer vision. 10873– 10883. [34] Shuo Wang, Yongcai Wang, Zhaoxin Fan, Yucheng Wang, Maiyue Chen, Kaihui Wang, Zhizhong Su, Wanting Li, Xudong Cai, Yeying Jin, et al. 2026. Monodream: Monocular vision-language navigation with panoramic dreaming. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40. 10074–10082. [35] Yunheng Wang, Yuetong Fang, Taowen Wang, Yixiao Feng, Yawen Tan, Shuning Zhang, Peiran Liu, Yiding Ji, and Renjing Xu. 2025. Dreamnav: A trajectory-based imaginative framework for zero-shot vision-and-language navigation. arXiv preprint arXiv:2509.11197 (2025). [36] Zihan Wang, Seungjun Lee, and Gim Hee Lee. 2025. Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation. In Advances in Neural Information Processing Systems. [37] Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, and Shuqiang Jiang. 2023. Gridmm: Grid memory map for vision-and-language navigation. In Proceedings of the IEEE/CVF International conference on computer vision. 15625–15636. [38] Zihan Wang, Yaohui Zhu, Gim Hee Lee, and Yachun Fan. 2025. Navrag: Generating user demand instructions for embodied navigation through retrievalaugmented llm. In Findings of the Association for Computational Linguistics: ACL 2025. 8430–8440. [39] Meng Wei, Chenyang Wan, Jiaqi Peng, Xiqian Yu, Yuqiang Yang, Delin Feng, Wenzhe Cai, Chenming Zhu, Tai Wang, Jiangmiao Pang, et al. 2025. Ground slow, move fast: A dual-system foundation model for generalizable vision-andlanguage navigation. arXiv preprint arXiv:2512.08186 (2025). [40] Meng Wei, Chenyang Wan, Xiqian Yu, Tai Wang, Yuqiang Yang, Xiaohan Mao, Chenming Zhu, Wenzhe Cai, Hanqing Wang, Yilun Chen, et al. 2025. Streamvln: Streaming vision-and-language navigation via slowfast context modeling. arXiv preprint arXiv:2507.05240 (2025). [41] Xuan Yao, Junyu Gao, and Changsheng Xu. 2025. Navmorph: A self-evolving world model for vision-and-language navigation in continuous environments. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 5536– 5546. [42] Shuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, and Xing Wei. 2025. JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation. arXiv preprint arXiv:2509.22548 (2025). [43] Yiming Zeng, Hao Ren, Shuhang Wang, Junlong Huang, and Hui Cheng. 2025. Navidiffusor: Cost-guided diffusion model for visual navigation. In 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 11994–12001. [44] Jiazhao Zhang, Anqi Li, Yunpeng Qi, Minghan Li, Jiahang Liu, Shaoan Wang, Haoran Liu, Gengze Zhou, Yuze Wu, Xingxing Li, et al. 2025. Embodied navigation foundation model. arXiv preprint arXiv:2509.12129 (2025). [45] Jiwen Zhang, Zejun Li, Siyuan Wang, Xiangyu Shi, Zhongyu Wei, and Qi Wu. 2026. SpatialNav: Leveraging Spatial Scene Graphs for Zero-Shot Vision-andLanguage Navigation. arXiv preprint arXiv:2601.06806 (2026). [46] Jiazhao Zhang, Kunyu Wang, Shaoan Wang, Minghan Li, Haoran Liu, Songlin Wei, Zhongyuan Wang, Zhizheng Zhang, and He Wang. 2024. Uni-navid: A video-based vision-language-action model for unifying embodied navigation
Haoxiang Shi et al.
tasks. arXiv preprint arXiv:2412.06224 (2024). [47] Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, and He Wang. 2024. Navid: Video-based vlm plans the next step for vision-and-language navigation. arXiv preprint arXiv:2402.15852 (2024). [48] Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, and Renjing Xu. 2025. Mapnav: A novel memory representation via annotated semantic maps for vlm-based vision-and-language navigation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13032–13056. [49] Duo Zheng, Shijia Huang, Yanyang Li, and Liwei Wang. 2025. EfficientVLN: A Training-Efficient Vision-Language Navigation Model. arXiv preprint arXiv:2512.10310 (2025). [50] Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, and Liwei Wang. 2024. Towards learning a generalist model for embodied navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13624–13634. [51] Gengze Zhou, Yicong Hong, Zun Wang, Chongyang Zhao, Mohit Bansal, and Qi Wu. 2025. Same: Learning generic language-guided visual navigation with state-adaptive mixture of experts. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 7794–7807. [52] Gengze Zhou, Yicong Hong, and Qi Wu. 2024. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 7641–7649.
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
Arxiv Preprint, 2026,
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation Supplementary Material In the supplementary material, we provide additional details on Trajectory Waypoint Predictor (TWP) training data collection, hyperparameter settings, and more qualitative results of the Trajectory Waypoint framework.
A
Trajectory Data Collection
To train the Trajectory Waypoint Predictor (TWP), we construct a safety-aware trajectory supervision dataset within the MatterPort3D (MP3D) and Habitat-Matterport 3D (HM3D) environment. Driven by the connectivity graph, navigation inherently follows a point-centric paradigm characterized by straight-line connections between graph nodes. However, these linear paths often pass dangerously close to obstacles, yielding suboptimal and potentially unsafe supervision. To mitigate this, we refine the raw rollouts using a Truncated Signed Distance Field (TSDF)-based adjustment, pushing trajectories toward higher-clearance regions.
Algorithm 1 TSDF-Adjusted Trajectory Collection Require: Connectivity graph G = (V, E), simulator M, TSDF parameters (𝐾, 𝜂 0, 𝜆) Ensure: Safety-aware trajectory set D 1: D ← ∅ 2: for all scan 𝑠 in dataset do 3: Initialize simulator M with scan 𝑠 4: for all source node 𝑣 𝑎 ∈ V do 5: (𝐶, meta) ← BuildTSDF(M, 𝑣 𝑎 ) ⊲ Construct local cost map 6: 7: 8: 9: 10: 11: 12: 13: 14:
Point-Centric Trajectory Generation. For each scan, we construct a connectivity graph, from which we derive directed source-target node pairs. For each source node, we synthesize panoramic depth observations to build a local 2D TSDF cost map centered at the source pose. Subsequently, for each valid source-target pair, we simulate a raw trajectory by repeatedly executing forward actions toward the target direction. If a collision is detected during this rollout, the sample is immediately discarded. Collision-free paths are retained for subsequent clearance refinement.
15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25:
TSDF-Based Clearance Refinement. Let T = {p1, . . . , p𝑁 } denote a raw, collision-free straight-line trajectory segment. To increase obstacle clearance, we fix the endpoints (p1 and p𝑁 ) and iteratively update the intermediate waypoints. Given the TSDF cost map 𝐶, each intermediate point is shifted along the normalized ascent direction of 𝐶:
p𝑖 ← p𝑖 + 𝜂 ·
∇𝐶 (p𝑖 ) , ∥∇𝐶 (p𝑖 )∥ 2
𝑖 = 2, . . . , 𝑁 − 1.
(19)
To prevent excessive path distortion, we apply a length constraint to the adjusted trajectory. If the length of the updated trajectory exceeds 1+𝜆 times the original raw trajectory length 𝐿0 , the current adjustment step is rejected, and the step size 𝜂 is halved. The detailed procedure is summarized in Algorithm 1.
for all neighbor 𝑣𝑏 of 𝑣 𝑎 do Traw ← ForwardRollout(M, 𝑣 𝑎 , 𝑣𝑏 ) if collision in Traw then continue end if T ← Traw if |T | ≥ 3 and 𝐶 ≠ ∅ then 𝐿0 ← Length(Traw ) 𝜂 ← 𝜂0 for all iteration 𝑘 ∈ {1, . . . , 𝐾 } do T′ ← T for all index 𝑖 ∈ {2, . . . , |T | − 1} do (𝑥𝑖 , 𝑧𝑖 ) ← WorldToMap(p𝑖 , meta) if out of map bounds then T ′ ← Traw break end if g ← ∇𝐶 (𝑥𝑖 , 𝑧𝑖 ) ⊲ Compute TSDF gradient if ∥g∥ 2 > 10−6 then g ⊲ Push toward p𝑖′ ← p𝑖′ + 𝜂 · ∥g∥ 2 higher clearance
end if end for 28: if Length(T ′ ) > (1 + 𝜆)𝐿0 then 29: 𝜂 ← 0.5𝜂 ⊲ Reject over-stretched update 30: else 31: T ← T′ ⊲ Accept update 32: 𝜂 ← 0.7𝜂 ⊲ Decay step size 33: end if 34: end for 35: end if 36: D ← D ∪ {(𝑣 𝑎 , 𝑣𝑏 , T )} 37: end for 38: end for 39: end for 40: return D 26: 27:
Arxiv Preprint, 2026,
Horizon-Aware Trajectory Stitching. While the TSDF-adjusted segments ensure local safety, individual point-to-point transitions are often shorter than the action prediction horizon required by our diffusion policy. To address this, we perform trajectory stitching to construct extended, horizon-aligned supervision. Starting from an initial adjusted segment (𝑣 𝑎 → 𝑣𝑏 ), we iteratively append valid subsequent segments. To maintain kinematic smoothness and prevent abrupt turns, the selection of the next segment is governed by an angle constraint. Let u denote the direction of the current segment and v represent a candidate subsequent direction. We uniformly sample from candidates whose turning angle satisfies 𝜃 (u, v) ≤ 𝜃 max (empirically set to 60◦ ). If no candidate meets this criterion, we select the one with the minimum turning angle. Ultimately, these seamlessly stitched, long-horizon trajectories serve as expert candidates, providing high-quality, safety-aware supervision for training the TWP.
B
Hyperparameter Settings
TWP Key Hyperparameters. We detail the core hyperparameters for our TWP and the TSDF-guided denoising process. Specifically, each trajectory candidate predicted by the TWP spans a horizon of 12 future actions. At each navigation step, the TWP samples up to 5 directional candidates based on the current panoramic observation. For the TWP prior, the angular and radial discretization strides are configured to 120 and 12, respectively. During the diffusion process, the model undergoes 10 denoising iterations. Regarding the TSDF guidance mechanism, following [43], we set the weight of the TSDF cost term for gradient computation to 0.003, with an update step size of 1.0 applied to the denoised outputs. This guidance is injected at every step for a maximum of 3 denoising iterations. Intuitively, the prediction horizon defines the temporal extent of the trajectory candidates, the prior parameters dictate the spatial discretization granularity, and the TSDF configurations govern the strength and frequency of the obstacle-aware geometric guidance.
C
Qualitative Analysis of TSDF Guidance
Figure 1 illustrates the qualitative impact of our TSDF guidance on trajectory generation. In narrow and cluttered environments, unguided predictions (raw_traj) frequently intersect with obstacles due to an inherent straight-line bias. The introduction of TSDF guidance (adjusted_traj) successfully steers the trajectories away from high-cost regions, ensuring they remain strictly within safe, navigable free space. Although this geometric compliance incurs a marginal increase in overall path length, it successfully eliminates collision risks. This confirms that the TSDF guidance effectively prioritizes kinematic safety over aggressive but risky planning.
D
Additional Visualizations of Trajectory Candidates Generation
To further demonstrate the robust trajectory generation capabilities of our TWP, Figure 2 presents extended qualitative comparisons across the remaining scenes in the VLN-CE val-unseen split. These supplementary visualizations consistently corroborate our primary findings: regardless of spatial complexity, the TWP successfully avoids generating invalid nodes in non-traversable regions. Instead, it reliably produces continuous, collision-free trajectory candidates
Haoxiang Shi et al.
that perfectly align with expert demonstrations, thereby ensuring strict execution fidelity across the entire val-unseen set.
Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigation
TSDF-Guided Trajectories in Different Environments
Figure 6: Visualization of TSDF-guided trajectory refinement.
Arxiv Preprint, 2026,
Arxiv Preprint, 2026,
Haoxiang Shi et al.
Scene Mesh
Waypoint Predictor
TWP(Ours)
GT Trajectories
Figure 7: Supplementary visualizations for the VLN-CE val-unseen split. Blue points indicate targets generated in inaccessible regions, while yellow points represent physically accessible areas.