Grounded World Model for Semantically Generalizable Planning
arXiv:2604.11751v1 [cs.RO] 13 Apr 2026
Quanyi Li∗ Independent
Lan Feng∗ EPFL
Letian Wang University of Toronto
Haonan Zhang Beihang University Alexandre Alahi EPFL
Wuyang Li EPFL Harold Soh NUS
Abstract In Model Predictive Control (MPC), world models predict the future outcomes of various action proposals, which are then scored to guide the selection of the optimal action. For visuomotor MPC, the score function is a distance metric between a predicted image and a goal image, measured in the latent space of a pretrained vision encoder like DINO and JEPA. However, it is challenging to obtain the goal image in advance of the task execution, particularly in new environments. Additionally, conveying the goal through an image offers limited interactivity compared with natural language. In this work, we propose to learn a Grounded World Model (GWM) in a vision-language-aligned latent space. As a result, each proposed action is scored based on how close its future outcome is to the task instruction, reflected by the similarity of embeddings. This approach transforms the visuomotor MPC to a VLA that surpasses VLM-based VLAs in semantic generalization. On the proposed WISER benchmark, GWM-MPC achieves a 87% success rate on the test set comprising 288 tasks that feature unseen visual signals and referring expressions, yet remain solvable with motions demonstrated during training. In contrast, traditional VLAs achieve an average success rate of 22%, even though they overfit the training set with a 90% success rate.
1
Introduction
Figure 1: Compared to existing World Models like DINO-WM and JEPA-WM, Grounded World Model enables goal specification via natural language, enabling a new approach to build VLA. A world model is inherently a state transition function that can predict future outcomes given the current state and a sequence of actions or a trajectory [20], enabling the agent to understand, predict, and plan within the physical world [1, 51]. Planning with world models is achieved through Model Predictive Control (MPC), where a batch of candidate trajectories is proposed and fed into the world ∗ Equal contribution. Code is available at https://github.com/QuanyiLi/gwm-wiser
Preprint.
model to predict their outcomes. Subsequently, the trajectory yielding the minimum cost is executed in the environment. To capture sufficient dynamic and semantic details, modern world models are usually trained with videos featuring realistic physics. During training, the current and future states are represented in either pixel space [7, 24, 61] or latent space [1, 63, 20]. Latent world models, such as DINO-WM [63] and JEPA-WM [51], have shown great potential for visuomotor planning, as they circumvent computationally expensive pixel reconstruction. For latent world models where state transition is defined in the latent space, the score function used for MPC is usually Mean Squared Error (MSE) between the embedding of each predicted future and that of the goal image. However, obtaining the goal image before task execution is challenging, especially for novel tasks where no demonstration is available. Furthermore, a goal image is not a human-friendly interface, compared to natural language, yet its use in the context of the latent world model has remained unexplored. In this work, we propose Grounded World Model (GWM) that operates within a vision-languagealigned latent space, allowing it to ground predicted future outcomes to specific semantics. Specifically, GWM learns the transition function in the latent space of a pretrained multi-modal retrieval model, Qwen3-VL-Embedding [33]. This foundation model can encode not only images and text, but also videos into a shared embedding space, where cosine similarity can be computed. It can be used off-the-shelf as the score function to select the best-matching robot behavior video, given the instruction. Compared to image-text contrastive models (e.g., CLIP [43]), it is more capable of understanding temporal action sequences, benefiting robot behavior recognition. As shown in Fig. 1, we use GWM to predict future outcomes for multiple candidate trajectory proposals in the foundation model’s latent space, and execute the trajectory that yields the highest cosine similarity against the instruction. We refer to this Vision Language Action (VLA) system as GWM-MPC. Unlike VLAs built by fine-tuning pretrained Vision-Language Models (VLMs), where knowledge forgetting can occur due to weight updates [57, 47, 59, 34, 65, 56, 25, 55], GWM leverages the pretrained latent space to learn the transition function without altering the foundation model. Consequently, GWM largely preserves the multi-modal world knowledge of Qwen3-VL-Embedding. Integrating it into MPC disentangles action generation and semantic understanding, effectively translating video understanding capabilities into semantically generalizable planning. As a result, GWM-MPC generalizes to novel visual signals and referring expressions, even those requiring active reasoning, as long as the motions required to complete the task have been demonstrated previously.
Figure 2: Experimental results on WISER for VLAs. The success rate gap on training and test tasks indicates the semantic generalizability. The larger the gap, the worse the generalizability. To benchmark semantic generalizability, we introduce the World-knowledge Integrated Semantic Embodied Reasoning (WISER) benchmark. It consists of 24 subsets corresponding to distinct categories of world knowledge, such as numbers, food, animals, and landmarks. Each subset has 12 training or test tasks, yielding 288 tasks in total for either the training or the test sets. The test tasks are constructed with world knowledge and referring expressions that are unseen during training. Despite this, the motions required to complete the test tasks are already demonstrated during training, making the test tasks inherently solvable. The objective is to learn from the training tasks and generalize to the test tasks in a zero-shot manner. If VLAs indeed inherit knowledge from pretrained VLMs, they must be able to recall the correct motions in zero-shot tests, even when the visual signals and referring expressions are previously unseen. However, Fig. 2 shows that traditional VLAs fail to generalize with an average success rate of 22% during test, even though they overfit the training to an average success rate of 90% across all 288 tasks. Some VLAs struggle to generalize even though they manage to complete all training tasks without a single failure. In contrast, GWM-MPC solves 87% of test tasks, demonstrating strong semantic generalization and suggesting our approach is a promising alternative to build VLAs. Additionally, the rendering-based action encoder used by GWM is training-free and embodiment-agnostic, enabling zero-shot generalization to the xArm6 robot despite its different action space, kinematics, and appearance. Ablation studies confirm that the system is robust to hyperparameters and its performance is bottlenecked by the foundation model, pointing to a clear direction for future improvement. 2
2
Method
The goal of VLA is to inherit the semantic generalizability of pretrained foundation models [6, 5]. We thus begin to formulate the semantic generalization problem and introduce our GWM-MPC solution. 2.1
Semantic Generalization in Planning
We assume there is a training dataset that consists of I trajectories, denoted as D = {T 1 , . . . , T I }. Each trajectory is a sequence of transitions T i = {(oi0 , j0i , ai0 , ℓi ), . . . , (oiT , jTi , aiT , ℓi )}, where oit , jti , and ait respectively denote the camera images, joint positions, and actions at timestep t for trajectory i. The variable ℓi represents the natural language task instruction, which remains constant throughout the entire episode. Using the dataset D, we aim to learn a policy that maps the current observation to an action chunk: at:t+c = f (ot , jt , ℓ), where c is the chunk size. During inference, these actions are sequentially executed in the environment until a new observation (ot+c+1 , jt+c+1 ) is received, at which point the policy generates a new action chunk. This closed-loop rollout terminates once the task ℓ is completed. A naive way to build such a policy to solve the demonstrated task, on which D was collected, is through trajectory or action chunk retrieval. This method involves simply iterating through the dataset D to find the transition that best matches the current observation (ot , jt , ℓ): ∗ at:t+c = aik∗ :k∗ +c , where (i∗ , k ∗ ) = arg min dist (ot , jt , ℓ), (oik , jki , ℓi ) (1) (i,k)∈V
Here, V is the set of all valid index pairs in the dataset D, where T i ∈ D and k denotes the timestep within that trajectory. It is reminiscent of the early non-parametric machine learning method, KNN, and N = 1 here. The dist(·, ·) works as the kernel function, which can be a learnable one, especially when the feature is in a high-dimensional space like images. In this case, the distance can be computed in a latent space for action retrieval [23], enabling generalization to new visual inputs. Conceptually, a parametric end-to-end policy pθ (at:t+c |ot , jt , ℓ) can be viewed as retrieving trajectories from a continuous proxy dataset D′ , which augments D by interpolating between the discrete demonstrations in D to generalize to novel, yet in-distribution, datapoints. Despite this, we still do not expect neural networks to produce trajectories that deviate too much from those demonstrated in D, especially when training is performed from scratch, and D is not sufficiently large. For the same reason, the language instruction ℓ tends to serve as a one-hot label [34], inducing poor novel instruction following ability; Moreover, the model may exploit visual shortcuts, selecting actions based on spurious correlations [54], such as associating the actions with the scene layout. Both issues indicate a lack of genuine vision-language understanding by the model, preventing extrapolation. VLAs are proposed to address this by initializing from pretrained foundation models. They are thus expected to possess the capability: semantic generalization. This aims at making a policy trained with D go beyond language instructions and visual signals in D. Ideally, regardless of how the current task instruction ℓ and observation ot appear—and no matter how significantly they differ from those in the training dataset D—the policy should still complete the task, as long as the motions required by this task have been demonstrated during training. This generalization is supposed to be achieved by inheriting open-world knowledge and leveraging the aligned vision-language feature space from the pretrained VLM. However, our experiments show that VLAs do not exhibit this capability. 2.2
Model Predictive Control (MPC)
The MPC framework typically comprises three steps: proposing candidate trajectories, predicting their future states or outcomes, and selecting the optimal trajectory using a score function. We use KNN to propose trajectories for three reasons. First, as discussed in Section 2.1, a parametric policy fundamentally retrieves from a continuous proxy of D and cannot generalize to motions beyond the demonstrations; training a separate action generation model pθ (at:t+c |jt ) would thus introduce additional learnable parameters without expanding the reachable trajectory space. Second, samplingbased methods like CEM [45] and gradient-based methods like Langevin MCMC [53] must search a high-dimensional action space without informative priors, making them inefficient when the set of valid trajectories is small and sparse. KNN sidesteps both issues by directly retrieving demonstrated trajectories from D, requiring no learned parameters and no open-ended search. Proposals are generated through Eq. 1 with a simplified kernel function MSE jt , jki , by iterating the demonstration 3
Figure 3: The training and inference workflow of GWM-MPC. All proposed trajectories are tokenized into images by rendering the robot URDF with the same camera extrinsics and intrinsics as the thirdperson RGB camera. Thus, observation and actions can be uniformly encoded as et by the vision encoder of Qwen3-VL-Embedding. The GWM then produces the future outcome embeddings pt for each candidate action. The foundation model’s backbone finally projects those embeddings to a shared vision-language space and gets {zt0 , . . . , ztN }. A sequence of actions is selected if it leads to the future with maximum cosine similarity against the goal embedding zg that is derived from the instruction ℓ with the same foundation model. During training, ground truth future is used to calculate the MSE loss in the vision encoder’s latent space. Notably, no language supervision is required. dataset D and looking for the available future actions at joint position jt . In this work, we keep the number of action proposals N = 12 for subsequent future outcome prediction and scoring: At:t+c = aik:k+c | (i, k) ∈ I ∗ , where I ∗ = top-N arg min MSE jt , jki (2) (i,k)∈V
If the robot behaviors are not restricted to those in D, trajectory proposals can be generated using other methods, such as grasping pose synthesis algorithms or visuomotor policies. Unlike VLMs that produce discrete text tokens [32, 37], large pretrained retrieval models can naturally produce a continuous scalar between 0-1, making them a better choice for a score function. In this work, we use Qwen3-VL-Embedding [33]. It comprises a vision encoder to map images and videos into the language feature space, followed by a transformer backbone that integrates tokenized text and visual features into a unified embedding z. Retrieval models serve as a score function by encoding the target task or user instruction into embeddings zg and the future outcome of N trajectories at timestep t into {zt1 , . . . , ztN }. Finally, the policy selects the sequence of actions whose predicted future outcome embedding exhibits the highest cosine similarity with the instruction embedding: ∗ ztn · zg a∗t:t+c = ant:t+c , where n∗ = arg max , a∗t:t+c ∈ At:t+c (3) n n∈{1,...,N } ∥zt ∥2 ∥zg ∥2 For a pixel-space world model, obtaining future outcome embeddings {zt1 , . . . , ztN } requires predicting future observed images ont+1:t+c ∼p(·|ot , ant:t+c ) for each proposed trajectory in At:t+c , so they can be encoded by the retrieval models to get ztn . Though training pixel-space prediction models is feasible given D [61], by using pretrained video diffusion models [16], reconstruction in pixel space captures redundant details and is expensive and less efficient on both training and inference. 2.3
Grounded World Model
To address these problems, we propose training the world model within the latent space of the multi-modal retrieval model from scratch. We call our model Grounded World Model (GWM), as its output can be grounded to specific semantics. Its training utilizes the representation space of the foundation model, whose weights remain frozen. Consequently, its vision-language understanding ability and world knowledge are largely preserved. GWM can optimize the scoring step, as shown in Eq. 3, by directly predicting the latent embedding of the future state as ztn ∼p(·|ot , ant:t+c ). The full inference and training process is depicted in Fig. 3. We introduce the details as follows. 4
Figure 4: Overview of the WISER Benchmark. Observations include the instruction ℓ, current joint positions and gripper states jt , and camera input ot . The benchmark comprises 24 world-knowledge categories, each partitioned into training and held-out test splits. Notably, all images, descriptions, and cube colors in the test set are entirely novel and non-overlapping with the training data. For example, even though cubes occupy identical positions (e.g., second from left), the spatial referring expressions and colors differ between the training and test. In each split, cube ordering is randomized across categories. Only 12 unique trajectories are shared by the training and test tasks. Rendering-based Action Tokenization (RAT). To predict the embedding zt , the model must encode both the current observation and the sequence of actions. Since the WISER benchmark utilizes target joint positions as the action space, we can sequentially render these actions into images using the third-person main camera parameters and the robot’s URDF. This approach allows us to leverage the feature extraction capabilities of the Qwen vision encoder without introducing additional learnable parameters. This method is highly generalizable: even when employing the delta gripper pose as the action space, inverse kinematics can be used to compute joint positions for future timesteps, making the rendering feasible. Therefore, this approach is embodiment-agnostic and can serve as a unified tokenizer for robot actions and states. In our ablation study, we demonstrate that RAT outperforms the traditional learnable action encoder and enables zero-shot generalization to the xArm6 robot. Training and Inference. The encoding produces a feature vector with the vision encoder of the foundation model et = E(ot , at:t+c ). Then the GWM predicts the outcome of the action trajectory by pt = Pθ (et ). Pθ is parameterized with a standard transformer model. The detailed model architecture and configuration are available in the appendix 6.6. During training, the supervision signal is derived from ground truth future image sequences, ēt = E(ot:t+c ). Since et , pt , and ēt share the same shape, we can directly feed pt into the foundation model’s backbone to obtain zt without projection layers. This is useful in experiments where we perform a sanity check of GWM and compute the performance upper bound. If we pass the ground truth future embedding ēt into the backbone, the MPC system degrades to a purely retrieval-based one. This case replaces the dist(·, ·) of Eq. 1 with the embedding similarity between the ground truth future observation oik:k+c and ℓ. As a result, the sequence of actions inducing a future that best aligns with ℓ is retrieved from At:t+c for execution: ∗
at:t+c = aik∗ :k∗ +c , where
(i∗ , k ∗ ) = arg max Embedding Similarity(oik:k+c , ℓ)
(4)
(i,k)∈I ∗
However, demonstrations are unavailable in novel scenarios where visual signals in ot differ significantly from those in D, but only the trajectories required to complete task ℓ exist in the training set. The generalizability of GWM thus enables using pt to approximate the unavailable ēt during the test.
3
WISER Benchmark
Training and Test Split. To evaluate the semantic generalizability of planners built upon pretrained foundation models, we build the World-knowledge Integrated Semantic Embodied Reasoning 5
(WISER) Benchmark, where each task requires the robot to pick one cube and place it onto a mark or image. Unlike dexterous motion, the trajectory required for each task is simple and rigid. We intentionally adopt this design, so the test-time failure can be directly attributed to poor semantic generalization rather than failing to learn complex motions. The benchmark comprises 24 categories. For each category, there is a training scene and a test scene. Both scenes have the same layout with four cubes in front of the robot and three images in front of the cubes. Thus, for either training scene or test scene, there are 4×3=12 pick-and-place tasks. For the training and test sets in the WISER benchmark, there are 12×24=288 tasks, respectively. The difference between training tasks and test tasks can be found in Fig. 4. In addition to the knowledge reflected in the three images, the cube colors differ between the training and test scenes. Furthermore, for test tasks, the methods for referring to the cube to pick and the place to drop have never been shown during training. If the policy can inherit the world knowledge and the open-vocabulary visual signal understanding ability from the foundation models after training, it is expected to complete the test tasks by retrieving or recalling the correct trajectory from the 12 unique trajectories shared between the training and test tasks. In the appendix 6.7, we provide visualizations for all tasks and the task instructions. Simulation. We developed the benchmark using ManiSkill [48], leveraging its GPU-parallelization to simultaneously simulate all 12 tasks across either training or test scenes. Demonstrations were collected solely on training tasks using a Franka Panda robot via MPlib [22]. The controller utilizes privileged information, such as goal positions and cube poses, to perform motion planning with a 100% success rate. The PD controller then tracks the planning results, a sequence of target joint positions, at a control frequency of 20Hz and simulation frequency of 100Hz. During data collection, we record the main camera stream (224×448), wrist camera stream (128×128), joint positions, gripper states, task instruction ℓ, and actions at . We collect 6 trajectories per task, with the robot’s initial states randomized to increase diversity. This results in a training dataset of 6×288=1728 trajectories, aimed at expanding state-space coverage and mitigating compounding errors when training VLAs. During closed-loop evaluation, the robot is consistently reset to a fixed retract pose. We impose a maximum limit of 120 steps (6 seconds) for both collection and evaluation. The dataset has LeRobot V2.1 and V3.0 versions [9]. We also provide the RLDS version [44]. Metrics. We employ three binary metrics to evaluate picking, placement, and overall task success. Grasp indicates whether the correct cube is successfully picked. Reach denotes whether the gripper’s Tool Center Point (TCP) reaches the designated goal with nearly zero speed, even though the grasp fails. Success signifies that the cube is correctly placed at the goal point, defined as Success = Grasp × Reach. We evaluate all policies on both the training tasks and the test tasks. Each task is evaluated once, and the metrics are averaged across the 288 training or test tasks.
4
Experiments
Appendix 6.1 provides implementation details for baselines. For GWM, we use the same training dataset, but exclude language labels and wrist-camera observations. We set the world model prediction horizon to c =60 steps. Rather than feeding the model the full 60-step future action sequence, we down-sample the rendered sequence of images (actions) into 6 keyframes for inference efficiency. Consequently, the model only needs to predict the embeddings of 6 future frames to represent the outcome of executing 60 steps. Despite this, the MPC replans every 20 steps. For each inference, N = 12 sequences are proposed using Eq. 2 and are subsequently scored according to Eq. 3. The sequence of actions yielding the maximum cosine similarity to zg is selected for execution. In practice, the zg is obtained by encoding not only the task prompt but the system prompt, and the current observation for best scoring accuracy. The final score is also a weighted combination of picking and placing tasks. Details on the score function design are available in the appendix 6.2. 4.1
Main Results
VLAs. The main results are presented in Table 1, where the best performance is highlighted in bold and the second best is underlined. None of the VLAs generalize well to the test tasks, achieving an average test success rate of only 22%, despite these tasks requiring the same skills demonstrated in training. For some VLAs like SmolVLA and π0 , they achieve nearly 100% success rate on training tasks, while during test, their performance is even worse than random trajectory retrieval (8% vs. 1/12 =8.3%). The failure mode in the test scenes is consistent across all baselines: they typically 6
Table 1: Evaluation Results for SOTA VLAs and GWM-MPC on the WISER Benchmark. Training Set
Test Set
H100 GPU Hours
Grasp
Reach
Success
Grasp
Reach
Success
InstructVLA[56] SmolVLA[46] Wall-OSS[57] GR00T-N1.6[42] InternVLA-A1[10] π0.5 [25] π0 [3] XVLA[62] UniVLA[8] Motus[2] Baseline Average
70 75 80 100 100 100 100 100 120 300 -
0.98 0.99 1.00 1.00 1.00 1.00 1.00 1.00 0.79 0.78 0.95
0.92 1.00 1.00 1.00 0.91 0.99 1.00 0.88 0.62 0.72 0.90
0.89 0.99 1.00 1.00 0.88 0.99 1.00 0.88 0.63 0.72 0.90
0.79 0.29 0.68 0.72 0.63 0.70 0.47 0.44 0.38 0.34 0.54
0.51 0.31 0.50 0.18 0.40 0.38 0.14 0.17 0.18 0.14 0.29
0.47 0.08 0.40 0.18 0.26 0.26 0.08 0.17 0.13 0.14 0.22
GWM-MPC
20
0.97
0.95
0.92
0.99
0.88
0.87
0.15 0.74 0.87 0.78 0.90 0.08
0.28 0.47 0.97 0.98 1.00 0.26
0.44 0.42 0.86 0.74 0.93 0.44
0.17 0.24 0.83 0.72 0.93 0.09
Method
GWM Ablation Study DreamDojo-MPC [16] GWM-MPC-AC GWM-MPC-xArm6 GWM-MPC w/ 12 D GT-MPC MPC w/o GWM
24 20 20 -
0.22 0.91 0.96 0.97 0.97 0.27
0.41 0.77 0.91 0.81 0.92 0.41
grasp the wrong cube or place it onto a random image. The top-performing VLAs are WALL-OSS and InstructVLA. Both models are pretrained with an auxiliary embodied VQA task, which improves the success rate by retaining knowledge from the foundational VLMs. Despite this, in appendix 6.4, we show that the base VLM of InstructVLA can localize the correct destination for 81% test scenarios, whereas finetuning still brings some knowledge forgetting, resulting in a 51% TCP reaching success rate. In appendix 6.2, we also show that InstructVLA overfits to sentence structures and loses the ability to understand decomposed instructions. Motus demands more computation to do the auxiliary task: pixel-space future prediction. For Motus and UniVLA, the gap between the training and test performance still reflects their poor semantic generalizability. Among all baselines, GR00T-N1.6 and InternVLA-A1 utilize a relative (delta) action space, which does not improve generalizability according to the results. All VLAs demonstrate some generalizability during the cube-picking stage, which doesn’t require world knowledge yet but just the ability to recognize unseen cube colors and referring expressions, achieving a 54% test average grasping success rate. InstructVLA and GR00T-N1.6 even manage to grasp the cube in over 70% of test tasks. We attribute this to the large number of cube-picking demonstrations present in the pretraining datasets. Our experiments cover most of the VLA training recipes, such as Latent Action Pretraining, Knowledge-Insulation, Mixture-of-Transformers, VQA auxiliary task, and video-action joint training. We thus confirm that poor semantic generalizability is a common issue for VLM-based VLAs. GWM. The GWM-MPC achieves the best test-scene performance, yielding an 87% success rate across 288 test tasks that feature unseen referring expressions, spatial relationship descriptions, and visual signals. This demonstrates that GWM can effectively capture scene semantics by recognizing predicted future robot behaviors and their interactions with scene objects, specifically cubes and images. As GWM-MPC retrieves trajectories from the training dataset, the failure can only result from the incorrect scoring and action selection. In other words, the scoring accuracy of Qwen3VL-Embedding bounds the performance of GWM-MPC. We also use the same dataset D to train an explicit world model, DreamDojo [16]. During inference, it produces a video representing the outcome of a sequence of actions, which is used in the same way as GWM in the MPC procedures. DreamDojo learns to reconstruct pixels quickly and accurately, while we found it struggles to follow the actions. For example, it may generate videos grasping the cube on the leftmost side, while the actions sent to DreamDojo are to grasp the cube next to the leftmost one. One possible reason is that only the ego-centric human videos are used to pretrain the latent action encoder of the DreamDojo. RAT. We also train an alternative model that encodes raw robot actions (represented as a list of numerical values) using a learnable module, which then feeds the resulting embedding into 7
Figure 5: Ablation results on the GT-MPC for planning-related hyperparameter choosing. the transformer alongside the embedding of the current observation E(ot ). The detailed model architecture is provided in Appendix 6.6. The evaluation results for this model are denoted as GWM-MPC-AC. This specific action tokenization scheme exhibits the same training-test performance gap as VLAs. We attribute this to the fact that image-represented actions align more easily with the current observation by utilizing the same vision encoder to extract features. Furthermore, RAT enables zero-shot cross-embodiment generalization. To demonstrate this capability, we collected the 12 unique trajectories using an xArm6 robot, recording only joint positions and gripper states to propose future actions following Eq. 2. We then reused the GWM, trained exclusively on Panda data, to convey the outcome of the xArm’s movements to the score module. The experiment, denoted as GWM-MPC-xArm6, shows that RAT and GWM enable zero-shot generalization to a new embodiment with different action spaces, forward kinematics, and appearance, achieving 87% and 83% success rates on training and test tasks, respectively. Training & Inference Efficiency. Training the GWM is computationally efficient: it requires only 20 GPU hours on our proposed WISER benchmark. Moreover, our approach avoids action learning and thus mitigates data reliance by employing KNN-based or retrieval-based action proposals. This aligns with recent findings [14], which suggest that retrieval-based planners can outperform purely learning-based alternatives while requiring fewer demonstrations. To evaluate data efficiency, we trained an additional GWM on a reduced dataset, denoted as GWM-MPC w/ 12 D. This subset covers only 288/2=144 training tasks from half of the 24 categories, providing just a single demonstration per task. The resulting GWM-MPC- 12 D model maintains a competitive 72% test success rate. The inference efficiency comparison is in appendix 6.3. Since N = 12 GWM inferences are required for MPC, our method underperforms all VLAs that only require generating one trajectory. Performance Upper Bound & Sanity Check. The performance of GWM on the training tasks is lower than that of other purely learning-based methods. This is because the Qwen3-VL-Embedding bottlenecks our system’s performance. In the GT-MPC experiment, we feed the backbone with the ground-truth future representation, ēt , of each sequence of actions. This setup either excludes the GWM entirely or assumes its prediction pt has zero error relative to ēt , thereby establishing the theoretical upper bound of the entire system. As shown in Table 1, GT-MPC fails to achieve a 100% success rate on both training and test tasks. Surprisingly, the standard GWM-MPC exhibits a slightly higher success rate than GT-MPC on the training tasks (92% vs. 90%). This suggests that GWM’s prediction introduces little noise and may even regularize the scoring process. Additionally, we feed the backbone with et , which is the embedding of the current observation ot alongside the sequence of actions at:t+c . This forms the “MPC w/o GWM” experiment, which serves as a sanity check by isolating the GWM module. The results show that without a world model to predict future outcomes, the system is reduced to a random trajectory selector, failing on almost all tasks. 4.2
Score Function Ablation Study
Because the GWM’s performance is bounded by the Qwen3-VL-Embedding, hyperparameter selection can be determined by running the GT-MPC directly. As illustrated in Fig. 5, we investigate the influence of the replanning interval (default is 20), the world model prediction horizon (default is 60), and the future subsampling rate or number of future keyframes (default is 6). The results indicate that the Qwen3-VL-Embedding is relatively robust to the replanning interval, although it achieves optimal performance at an interval of 20 on the WISER benchmark. We also test the trained model, GWM-MPC, with different replanning intervals and obtain consistent results. However, for both the prediction horizon and the future subsampling rate, specific thresholds must be met before achieving 8
satisfactory performance. If the prediction horizon is too short, the foundation model cannot infer the policy’s intention. Additionally, an extreme subsampling rate, such as keeping only 2 or 4 frames from a 60-frame future, confuses the Qwen3-VL-Embedding, preventing it from accurately scoring the robot behaviors. Furthermore, the ablation study on model size demonstrates that the larger model indeed excels over the smaller one in comprehending videos and predicting futures. We also evaluate the Perception Encoder [4] and find that it scores videos with zero accuracy. In appendix 6.5, we built GT-MPC for LIBERO-goal [39] and find it can accurately select actions for 80% tasks in zero-shot.
5
Related Work
World Models. Given a sequence of actions and the current observation, a world model predicts what will happen next [20]. Most existing research on world models aims to model the transitions in pixel space, taking images as input and predicting another set of images. One application of these pixel-space world models is policy evaluation and data synthesis [18, 50, 21, 38, 49, 24, 16]. Another application is model-based planning [66, 64, 61, 16]. Rather than predicting the future in explicit representations like images, some works propose predicting the future on the latent space of pretrained models [17, 63, 1, 51, 60, 13], which improves learning efficiency by avoiding pixel-level reconstruction. Our work extends this thread of research by allowing the specification of goals with natural, open-vocabulary instructions rather than goal images, which are hard to obtain and interact with. Similar to previous works [61, 28, 31, 32, 11, 37], GWM-based planning follows the three common MPC steps: scoring, ranking, and selection. However, GWM operates in the latent space, where grounding is easier and facilitates out-of-distribution (OOD) generalization [19]. In addition, prior works usually use sampling-based methods like CEM [45] or gradient-based methods [15] to propose or search trajectories, while we use K-Nearest Neighbors to retrieve skills from the training dataset for the reasons discussed in Section 2.1 and Section 2.2. VLAs and Benchmarks. Our method enables the construction of a VLA system that acts according to visual inputs and natural language instructions. Unlike our MPC system, most VLAs are built end-to-end based on pretrained VLMs by pretraining on large-scale robot data and fine-tuning on target tasks [34, 25, 2, 29, 56, 8]. This paradigm was initially proposed to inherit knowledge from pretrained foundation models to achieve semantic generalization [6], allowing them to become generalist policies capable of finishing tasks in zero or a few shots. However, some works have found that VLAs may merely overfit to specific tasks by learning shortcuts, lacking actual semantic generalizability [52, 1, 57, 47, 59, 34, 65, 56, 25, 55]. Furthermore, there are currently no benchmarks available to evaluate how much knowledge from pretrained foundation models has been retained in VLAs, or to measure their semantic generalizability. Most benchmarks collect data on evaluation or test scenarios [41, 12, 65, 40, 35], where the distribution gap between training and testing consists only of visual interference and trivial object pose perturbations. Some recent works have recognized the lack of such benchmarks and thus conducted proprietary semantic generalization experiments in simulation [59, 58, 56] and the real world [27]. The setting of GrinningFace [59] is close to ours, while the task is simpler with only 3 different motions, and the images to place the cube are from the emoji dataset. Compared to existing options, the proposed WISER benchmark provides a more standard, comprehensive, and scalable way to test the semantic generalizability.
6
Conclusion
In this work, we formulate the semantic generalization problem in the context of planning. We argue that policies taking advantage of pretrained vision-language models are supposed to possess the ability to address this problem. We thus design a benchmark to evaluate state-of-the-art VLAs on this, and find that all of them deviate from the goal of inheriting the knowledge from pretrained models or being a generalist. On the other hand, we find that training a latent world model in a grounded latent space can provide an alternative to build a VLA system for acquiring the semantic generalizability from the pretrained retrieval model. When planning with MPC, the proposed GWM can address 87% of unseen tasks with novel visual signals and object referring expressions, whereas the best VLA achieves only a 47% success rate on the same test tasks. The rendering-based action tokenizer additionally allows cross-embodiment generalization without introducing new parameters to encode actions. The fact that the performance is bottlenecked by the pretrained model points out a future direction to fine-tune the Qwen3-VL-Embedding with robot data for further improvements. 9
References [1] Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Mojtaba, Komeili, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, Sergio Arnaud, Abha Gejji, Ada Martin, Francois Robert Hogan, Daniel Dugas, Piotr Bojanowski, Vasil Khalidov, Patrick Labatut, Francisco Massa, Marc Szafraniec, Kapil Krishnakumar, Yong Li, Xiaodong Ma, Sarath Chandar, Franziska Meier, Yann LeCun, Michael Rabbat, and Nicolas Ballas. V-jepa 2: Self-supervised video models enable understanding, prediction and planning, 2025. URL https://arxiv.org/abs/2506.09985. [2] Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, Hongyan Zhao, Hanyu Liu, Zhizhong Su, Lei Ma, Hang Su, and Jun Zhu. Motus: A unified latent action world model, 2025. URL https://arxiv.org/abs/2512.13030. [3] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0 : A vision-language-action flow model for general robot control, 2026. URL https://arxiv. org/abs/2410.24164. [4] Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, Junke Wang, Marco Monteiro, Hu Xu, Shiyu Dong, Nikhila Ravi, Daniel Li, Piotr Dollár, and Christoph Feichtenhofer. Perception encoder: The best visual embeddings are not at the output of the network, 2025. URL https: //arxiv.org/abs/2504.13181. [5] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-2: Vision-language-action models transfer web knowledge to robotic control, 2023. URL https://arxiv.org/abs/2307.15818. [6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deeksha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan, Huong Tran, Vincent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. Rt-1: Robotics transformer for real-world control at scale, 2023. URL https://arxiv.org/abs/2212.06817. [7] Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, Yusuf Aytar, Sarah Bechtle, Feryal Behbahani, Stephanie Chan, Nicolas Heess, Lucy Gonzalez, Simon Osindero, Sherjil Ozair, Scott Reed, Jingwei Zhang, Konrad Zolna, Jeff Clune, Nando de Freitas, Satinder Singh, and Tim Rocktäschel. Genie: Generative interactive environments, 2024. URL https: //arxiv.org/abs/2402.15391. [8] Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions, 2025. URL https://arxiv.org/abs/2505.06111. 10
[9] Remi Cadene, Simon Alibert, Francesco Capuano, Michel Aractingi, Adil Zouitine, Pepijn Kooijmans, Jade Choghari, Martino Russi, Caroline Pascal, Steven Palma, Mustafa Shukor, Jess Moss, Alexander Soare, Dana Aubakirova, Quentin Lhoest, Quentin Gallouédec, and Thomas Wolf. Lerobot: An open-source library for end-to-end robot learning. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=CiZMMAFQR3. [10] Junhao Cai, Zetao Cai, Jiafei Cao, Yilun Chen, Zeyu He, Lei Jiang, Hang Li, Hengjie Li, Yang Li, Yufei Liu, Yanan Lu, Qi Lv, Haoxiang Ma, Jiangmiao Pang, Yu Qiao, Zherui Qiu, Yanqing Shen, Xu Shi, Yang Tian, Bolun Wang, Hanqing Wang, Jiaheng Wang, Tai Wang, Xueyuan Wei, Chao Wu, Yiman Xie, Boyang Xing, Yuqiang Yang, Yuyin Yang, Qiaojun Yu, Feng Yuan, Jia Zeng, Jingjing Zhang, Shenghan Zhang, Shi Zhang, Zhuoma Zhaxi, Bowen Zhou, Yuanzhen Zhou, Yunsong Zhou, Hongrui Zhu, Yangkun Zhu, and Yuchen Zhu. Internvlaa1: Unifying understanding, generation and action for robotic manipulation, 2026. URL https://arxiv.org/abs/2601.02456. [11] Shirui Chen, Cole Harrison, Ying-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J. Ratliff, Jiafei Duan, Dieter Fox, and Ranjay Krishna. Topreward: Token probabilities as hidden zero-shot rewards for robotics, 2026. URL https://arxiv.org/abs/2602.19313. [12] Tianxing Chen, Zanxin Chen, Baijun Chen, Zijian Cai, Yibin Liu, Zixuan Li, Qiwei Liang, Xianliang Lin, Yiheng Ge, Zhenyu Gu, Weiliang Deng, Yubin Guo, Tian Nian, Xuanbing Xie, Qiangyu Chen, Kailun Su, Tianling Xu, Guodong Liu, Mengkang Hu, Huan ang Gao, Kaixuan Wang, Zhixuan Liang, Yusen Qin, Xiaokang Yang, Ping Luo, and Yao Mu. Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation, 2025. URL https://arxiv.org/abs/2506.18088. [13] Matthieu Destrade, Oumayma Bounou, Quentin Le Lidec, Jean Ponce, and Yann LeCun. Valueguided action planning with jepa world models, 2025. URL https://arxiv.org/abs/2601. 00844. [14] Kamil Dreczkowski, Pietro Vitiello, Vitalis Vosylius, and Edward Johns. Learning a thousand tasks in a day. Science Robotics, 10(108), November 2025. ISSN 2470-9476. doi: 10.1126/ scirobotics.adv7594. URL http://dx.doi.org/10.1126/scirobotics.adv7594. [15] Pete Florence, Corey Lynch, Andy Zeng, Oscar Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning, 2021. URL https://arxiv.org/abs/2109.00137. [16] Shenyuan Gao, William Liang, Kaiyuan Zheng, Ayaan Malik, Seonghyeon Ye, Sihyun Yu, Wei-Cheng Tseng, Yuzhu Dong, Kaichun Mo, Chen-Hsuan Lin, Qianli Ma, Seungjun Nah, Loic Magne, Jiannan Xiang, Yuqi Xie, Ruijie Zheng, Dantong Niu, You Liang Tan, K. R. Zentner, George Kurian, Suneel Indupuru, Pooya Jannaty, Jinwei Gu, Jun Zhang, Jitendra Malik, Pieter Abbeel, Ming-Yu Liu, Yuke Zhu, Joel Jang, and Linxi "Jim" Fan. Dreamdojo: A generalist robot world model from large-scale human videos, 2026. URL https://arxiv.org/abs/ 2602.06949. [17] Raktim Gautam Goswami, Amir Bar, David Fan, Tsung-Yen Yang, Gaoyue Zhou, Prashanth Krishnamurthy, Michael Rabbat, Farshad Khorrami, and Yann LeCun. World models can leverage human videos for dexterous manipulation, 2025. URL https://arxiv.org/abs/ 2512.13644. [18] Yanjiang Guo, Lucy Xiaoyang Shi, Jianyu Chen, and Chelsea Finn. Ctrl-world: A controllable generative world model for robot manipulation, 2025. URL https://arxiv.org/abs/2510. 10125. [19] Pranay Gupta, Henny Admoni, and Andrea Bajcsy. Adapting by analogy: Ood generalization of visuomotor policies via functional correspondence, 2025. URL https://arxiv.org/abs/ 2506.12678. [20] David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution, 2018. URL https://arxiv.org/abs/1809.01999. 11
[21] Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models, 2025. URL https://arxiv.org/abs/2509.24527. [22] Hao Su’s Lab. MPlib: A lightweight motion planning library, 2024. URL https://github. com/haosulab/MPlib. GitHub repository. [23] Chengyang He, Xu Liu, Gadiel Sznaier Camps, Guillaume Sartoretti, and Mac Schwager. Demystifying diffusion policies: Action memorization and simple lookup table alternatives, 2025. URL https://arxiv.org/abs/2505.05787. [24] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving, 2023. URL https://arxiv.org/abs/2309.17080. [25] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Manuel Y. Galliker, Dibya Ghosh, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Devin LeBlanc, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Allen Z. Ren, Lucy Xiaoyang Shi, Laura Smith, Jost Tobias Springenberg, Kyle Stachowicz, James Tanner, Quan Vuong, Homer Walke, Anna Walling, Haohuan Wang, Lili Yu, and Ury Zhilinsky. π0.5 : a vision-language-action model with open-world generalization, 2025. URL https://arxiv.org/abs/2504.16054. [26] Keller Jordan. Muon: An optimizer for hidden layers in neural networks. kellerjordan.github.io/posts/muon/, 2024. Accessed: 2026-03-03.
https://
[27] Nikita Kachaev, Mikhail Kolosov, Daniil Zelezetsky, Alexey K. Kovalev, and Aleksandr I. Panov. Don’t blind your vla: Aligning visual representations for ood generalization, 2025. URL https://arxiv.org/abs/2510.25616. [28] Gi-Cheon Kang, Junghyun Kim, Kyuhwan Shim, Jun Ki Lee, and Byoung-Tak Zhang. Clip-rt: Learning language-conditioned robotic policies from natural language supervision, 2025. URL https://arxiv.org/abs/2411.00508. [29] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success, 2025. URL https://arxiv.org/abs/2502.19645. [30] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/1412.6980. [31] Jacky Kwok, Xilun Zhang, Mengdi Xu, Yuejiang Liu, Azalia Mirhoseini, Chelsea Finn, and Marco Pavone. Scaling verification can be more effective than scaling policy learning for vision-language-action alignment, 2026. URL https://arxiv.org/abs/2602.12281. [32] Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn. Roboreward: General-purpose vision-language reward models for robotics, 2026. URL https://arxiv.org/abs/2601.00675. [33] Mingxin Li, Yanzhao Zhang, Dingkun Long, Keqin Chen, Sibo Song, Shuai Bai, Zhibo Yang, Pengjun Xie, An Yang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. Qwen3-vl-embedding and qwen3-vl-reranker: A unified framework for state-of-the-art multimodal retrieval and ranking, 2026. URL https://arxiv.org/abs/2601.04720. [34] Quanyi Li. Task reconstruction and extrapolation for π0 using text latent, 2025. URL https: //arxiv.org/abs/2505.03500. [35] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation, 2024. URL https://arxiv.org/abs/2405.05941. [36] Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. EAGLE-2: Faster inference of language models with dynamic draft trees. In Empirical Methods in Natural Language Processing, 2024. 12
[37] Anthony Liang, Yigit Korkmaz, Jiahui Zhang, Minyoung Hwang, Abrar Anwar, Sidhant Kaushik, Aditya Shah, Alex S. Huang, Luke Zettlemoyer, Dieter Fox, Yu Xiang, Anqi Li, Andreea Bobu, Abhishek Gupta, Stephen Tu, Erdem Biyik, and Jesse Zhang. Robometer: Scaling general-purpose robotic reward models via trajectory comparisons, 2026. URL https: //arxiv.org/abs/2603.02115. [38] Yue Liao, Pengfei Zhou, Siyuan Huang, Donglin Yang, Shengcong Chen, Yuxin Jiang, Yue Hu, Jingbin Cai, Si Liu, Jianlan Luo, Liliang Chen, Shuicheng Yan, Maoqing Yao, and Guanghui Ren. Genie envisioner: A unified world foundation platform for robotic manipulation, 2025. URL https://arxiv.org/abs/2508.05635. [39] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning, 2023. URL https: //arxiv.org/abs/2306.03310. [40] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks, 2022. URL https://arxiv.org/abs/2112.03227. [41] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adeet Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlekar, and Yuke Zhu. Robocasa: Large-scale simulation of everyday tasks for generalist robots, 2024. URL https://arxiv.org/abs/2406.02523. [42] NVIDIA, :, Johan Bjorck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi "Jim" Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, Joel Jang, Zhenyu Jiang, Jan Kautz, Kaushil Kundalia, Lawrence Lao, Zhiqi Li, Zongyu Lin, Kevin Lin, Guilin Liu, Edith Llontop, Loic Magne, Ajay Mandlekar, Avnish Narayan, Soroush Nasiriany, Scott Reed, You Liang Tan, Guanzhi Wang, Zu Wang, Jing Wang, Qi Wang, Jiannan Xiang, Yuqi Xie, Yinzhen Xu, Zhenjia Xu, Seonghyeon Ye, Zhiding Yu, Ao Zhang, Hao Zhang, Yizhou Zhao, Ruijie Zheng, and Yuke Zhu. Gr00t n1: An open foundation model for generalist humanoid robots, 2025. URL https://arxiv.org/abs/2503.14734. [43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020. [44] Sabela Ramos, Sertan Girgin, Léonard Hussenot, Damien Vincent, Hanna Yakubovich, Daniel Toyama, Anita Gergely, Piotr Stanczyk, Raphael Marinier, Jeremiah Harmsen, Olivier Pietquin, and Nikola Momchev. Rlds: an ecosystem to generate, share and use datasets in reinforcement learning, 2021. [45] Reuven Y Rubinstein and Dirk P Kroese. The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation, and machine learning, volume 133. Springer, 2004. [46] Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, Simon Alibert, Matthieu Cord, Thomas Wolf, and Remi Cadene. Smolvla: A vision-language-action model for affordable and efficient robotics, 2025. URL https://arxiv.org/abs/2506.01844. [47] Wenxuan Song, Ziyang Zhou, Han Zhao, Jiayi Chen, Pengxiang Ding, Haodong Yan, Yuxin Huang, Feilong Tang, Donglin Wang, and Haoang Li. Reconvla: Reconstructive visionlanguage-action model as effective robot perceiver. arXiv preprint arXiv:2508.10333, 2025. [48] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li, Tongzhou Mu, Nan Xiao, Arnav Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang, Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai, 2025. URL https://arxiv.org/abs/2410. 00425. 13
[49] Gemini Robotics Team, Krzysztof Choromanski, Coline Devin, Yilun Du, Debidatta Dwibedi, Ruiqi Gao, Abhishek Jindal, Thomas Kipf, Sean Kirmani, Isabel Leal, Fangchen Liu, Anirudha Majumdar, Andrew Marmon, Carolina Parada, Yulia Rubanova, Dhruv Shah, Vikas Sindhwani, Jie Tan, Fei Xia, Ted Xiao, Sherry Yang, Wenhao Yu, and Allan Zhou. Evaluating gemini robotics policies in a veo world simulator, 2026. URL https://arxiv.org/abs/2512. 10675. [50] GigaWorld Team, Angen Ye, Boyuan Wang, Chaojun Ni, Guan Huang, Guosheng Zhao, Haoyun Li, Jiagang Zhu, Kerui Li, Mengyuan Xu, Qiuping Deng, Siting Wang, Wenkang Qin, Xinze Chen, Xiaofeng Wang, Yankai Wang, Yu Cao, Yifan Chang, Yuan Xu, Yun Ye, Yang Wang, Yukun Zhou, Zhengyuan Zhang, Zhehao Dong, and Zheng Zhu. Gigaworld-0: World models as data engine to empower embodied ai, 2025. URL https://arxiv.org/abs/2511.19861. [51] Basile Terver, Tsung-Yen Yang, Jean Ponce, Adrien Bardes, and Yann LeCun. What drives success in physical planning with joint-embedding predictive world models?, 2026. URL https://arxiv.org/abs/2512.24497. [52] Guodong Wang, Chenkai Zhang, Qingjie Liu, Jinjin Zhang, Jiancheng Cai, Junjie Liu, and Xinmin Liu. Libero-x: Robustness litmus for vision-language-action models, 2026. URL https://arxiv.org/abs/2602.06556. [53] Ying Wang, Oumayma Bounou, Gaoyue Zhou, Randall Balestriero, Tim G. J. Rudner, Yann LeCun, and Mengye Ren. Temporal straightening for latent planning, 2026. URL https: //arxiv.org/abs/2603.12231. [54] Youguang Xing, Xu Luo, Junlin Xie, Lianli Gao, Hengtao Shen, and Jingkuan Song. Shortcut learning in generalist robot policies: The role of dataset diversity and fragmentation, 2025. URL https://arxiv.org/abs/2508.06426. [55] Kechun Xu, Zhenjie Zhu, Anzhe Chen, Shuqi Zhao, Qing Huang, Yifei Yang, Haojian Lu, Rong Xiong, Masayoshi Tomizuka, and Yue Wang. Seeing to act, prompting to specify: A bayesian factorization of vision language action policy, 2025. URL https://arxiv.org/abs/2512. 11218. [56] Shuai Yang, Hao Li, Yilun Chen, Bin Wang, Yang Tian, Tai Wang, Hanqing Wang, Feng Zhao, Yiyi Liao, and Jiangmiao Pang. Instructvla: Vision-language-action instruction tuning from understanding to manipulation, 2025. URL https://arxiv.org/abs/2507.17520. [57] Andy Zhai, Brae Liu, Bruno Fang, Chalse Cai, Ellie Ma, Ethan Yin, Hao Wang, Hugo Zhou, James Wang, Lights Shi, Lucy Liang, Make Wang, Qian Wang, Roy Gan, Ryan Yu, Shalfun Li, Starrick Liu, Sylas Chen, Vincent Chen, and Zach Xu. Igniting vlms toward the embodied space, 2025. URL https://arxiv.org/abs/2509.11766. [58] Borong Zhang, Jiahao Li, Jiachen Shen, Yishuai Cai, Yuhao Zhang, Yuanpei Chen, Juntao Dai, Jiaming Ji, and Yaodong Yang. Vla-arena: An open-source framework for benchmarking vision-language-action models, 2025. URL https://arxiv.org/abs/2512.22539. [59] Chuheng Zhang, Rushuai Yang, Xiaoyu Chen, Kaixin Wang, Li Zhao, Yi Chen, and Jiang Bian. How do vlas effectively inherit from vlms?, 2025. URL https://arxiv.org/abs/2511. 06619. [60] Zeyu Zhang, Danning Li, Ian Reid, and Richard Hartley. Geoworld: Geometric world models, 2026. URL https://arxiv.org/abs/2602.23058. [61] Wentao Zhao, Jiaming Chen, Ziyu Meng, Donghui Mao, Ran Song, and Wei Zhang. Vlmpc: Vision-language model predictive control for robotic manipulation, 2024. URL https:// arxiv.org/abs/2407.09829. [62] Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, Ya-Qin Zhang, Jiangmiao Pang, Jingjing Liu, Tai Wang, and Xianyuan Zhan. X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model, 2025. URL https://arxiv.org/abs/2510.10274. 14
[63] Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. Dino-wm: World models on pre-trained visual features enable zero-shot planning, 2025. URL https://arxiv.org/abs/ 2411.04983. [64] Siyuan Zhou, Yilun Du, Jiaben Chen, Yandong Li, Dit-Yan Yeung, and Chuang Gan. Robodreamer: Learning compositional world models for robot imagination, 2024. URL https://arxiv.org/abs/2404.12377. [65] Xueyang Zhou, Yangming Xu, Guiyao Tie, Yongchao Chen, Guowen Zhang, Duanfeng Chu, Pan Zhou, and Lichao Sun. Libero-pro: Towards robust and fair evaluation of vision-languageaction models beyond memorization, 2025. URL https://arxiv.org/abs/2510.03827. [66] Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, and Abhishek Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets, 2025. URL https://arxiv.org/abs/2504.02792.
15
Appendix 6.1
Implementation Details for Baselines
We summarize the configurations of the evaluated VLAs in Table 2. We observed that the more frequent the replanning, the more difficult closed-loop control for VLAs becomes, due to compounding errors. Replanning every 20 steps (1-second simulation time) is a sweetspot for VLAs. Increasing the replanning frequency to replan every 10 steps brings more or less performance drops. We thus set the replanning interval to 20 steps for most baselines. For some VLAs that suffer from compounding error, we replan every 40 steps to improve their performance and thus increase their action thunk size c. Based on this, for GWM-MPC, we tune its parameters based on the replanning interval of 20 steps. Other hyperparameters are selected in terms of the open-loop future video classification accuracy with Qwen3-VL-embedding, using the training data only. For models like InternVLA-A1, SmolVLA, Wall-OSS, π0.5 , and π0 , we found that increasing the action chunk size c did not yield performance improvements. Consequently, we set c to their 20-step replanning interval to minimize the number of trainable parameters. However, specific models required distinct settings: Motus, GR00T-N1.6, and UniVLA suffer from error accumulation with 20-step replanning, and XVLA’s TCP reaching success rate degrades with smaller chunk sizes. Therefore, they require a larger chunk size. For GR00T, we omit results for the LeRobot-implemented GR00T-N1.5 as it underperformed the official GR00T-N1.6. We exclude the use of the wrist camera for GR00T-N1.6, because it harms the performance a lot. Across all models, we strictly adhere to official fine-tuning setups (e.g., full-parameter vs. action-head only vs. lora-finetune), training them until closed-loop performance plateaus on training scenes prior to zero-shot evaluation on the test scenes. We also tested openvla-oft [29], but its training task success rate would gradually drop when the training set covers more tasks. When training openvla-oft on one out of 24 subsets, it can overfit the 12 training tasks to 100% succress rate, while increasing the dataset size to the full WISER training set, it only reaches 24% success rate. So we exclude it.
Model
Table 2: Configuration and Implementation Details of Evaluated VLAs c Replan Interval Dataloader Implementation Action
Motus XVLA GR00T-N1.6 InstructVLA OpenVLA-OFT SmolVLA Wall-OSS π0.5 π0 InternVLA-A1 UniVLA
6.2
48 40 40 16 20 20 20 20 20 20 40
40 20 40 16 20 20 20 20 20 20 40
LeRobot LeRobot LeRobot RLDS RLDS LeRobot LeRobot LeRobot LeRobot LeRobot RLDS
Official LeRobot Official Official Official LeRobot LeRobot LeRobot LeRobot Official Official
Absolute Absolute Relative Absolute Absolute Absolute Absolute Absolute Absolute Relative Absolute
Score Function Design
To obtain zg , we feed a multimodal prompt into Qwen3-VL-Embedding. A retrieval-oriented system prompt s is prepended with content: “Retrieve the video which can best finish the manipulation task specified by the user, given the layout of the workspace and the current frame observation.” This system prompt steers the model to produce embeddings that align task descriptions with future visual outcomes, but doesn’t disclose any task-specific information. The task instruction ℓ, which follows the template “Pick up the {X} and place it onto the {Y}”, is decomposed into two sub-task prompts: ℓpick =“Pick up the {X} from the table” and ℓplace =“Place the grasped object to the {Y} on the table”, where {X} and {Y} are extracted from ℓ via pattern matching. Each sub-task prompt is independently encoded with visual context—the initial observation o0 and the current observation ot —to produce sub-task embeddings: zgpick = Qwen3-VL-Embed(s, ℓpick , o0 , ot ),
zgplace = Qwen3-VL-Embed(s, ℓplace , o0 , ot ). 16
(5)
The initial observation o0 provides a static visual anchor of the workspace layout, while the current observation ot supplies dynamic context at the time of replanning. Both images and the text prompt are jointly processed by Qwen3-VL-Embedding to produce a normalized embedding vector. Given N candidate action sequences with predicted future embeddings {zt1 , . . . , ztN }, the cosine similarities against each sub-task embedding are computed and normalized across candidates via softmax: exp(cos(ztn , zgplace )) n σplace = PN . m place )) m=1 exp(cos(zt , zg
exp(cos(ztn , zgpick )) n , σpick = PN m pick m=1 exp(cos(zt , zg ))
(6)
The final selection score depends on the current grasp state, which is determined by the contact sensor on the robot gripper: ( n σpick if the object has not been grasped, n S = (7) n σplace if the object has been grasped, and the action sequence with the highest score is selected: n∗ = arg maxn S n . We also experimented with canceling the grasp-based weighted combination, and scoring trajectories directly with the embedding that resulted from: zg = Qwen3-VL-Embed(s, ℓ, o0 , ot ) As shown in Tab. 3, it brings a 15% performance drop in terms of test success rate, which, as we suggested, is caused by the Qwen-3-VL-Embedding instead of the GWM. We also tried to apply the task prompt decomposition to the best VLA baseline, InstructVLA. However, we find that InstructVLA experiences a significant performance drop on both the training tasks (89% → 52%) and the test tasks (47% → 30%), when decomposing each task into two subtasks. This result suggests that VLM-based VLAs can overfit to the specific sentence structures seen during training, rather than genuinely understanding the compositional semantics of each clause. Even a simple rephrasing or decomposition of the task prompt—without altering its underlying meaning—is sufficient to induce a notable performance degradation. In contrast, GWM-MPC can leverage the intact language understanding ability of the foundation model, so that a task decomposition further boosts its performance, which aligns with the intuition that atomic tasks should be easier to address than compositional long-horizon tasks, which are basically a chain of atomic tasks. Table 3: Ablation on Prompt Decomposition for GWM-MPC and InstructVLA. Training Set
Prompt
6.3
Test Set
Grasp
Reach
Success
Grasp
Reach
Success
GWM ℓpick + ℓplace GWM ℓ
0.97 0.93
0.95 0.88
0.92 0.82
0.99 0.92
0.88 0.76
0.87 0.73
InstructVLA ℓpick + ℓplace InstructVLA ℓ
0.98 0.98
0.53 0.92
0.52 0.89
0.80 0.79
0.30 0.51
0.30 0.47
Inference Efficiency
Figure 6: For all methods, we measure the inference efficiency with the rollout FPS, which is how many times the env.step is called in one second. VLA baselines have better inference efficiency than the GWM-MPC when evaluated on the test tasks. It is because we need to forward the GWM N = 12 times to get future embeddings for all proposals. Also, we generate future embeddings sequentially rather than in parallel because the Qwen encoder produces bugs when batching input. This deteriorates the inference efficiency.
17
6.4
Visual Grounding Evaluation for InstructVLA.
It is possible that the base VLM inherently lacks the ability to recognize the captured workspace images from WISER, and consequently, the VLA fine-tuned from it cannot successfully complete the manipulation tasks. To rule out this possibility, we assess the visual understanding capabilities of the base Eagle-2B model [36] before the fine-tuning of the best VLA baseline, InstructVLA. Specifically, we design a visual grounding evaluation on the WISER benchmark. For each of the 24 test task configurations, we reset the simulation environment and capture the initial observation from the main camera. We then ask the foundation VLM to identify which of the three destination images (left, middle, or right) best matches the referring expression extracted from the task instruction. The prompt sent to Eagle-2B is: There are three images with white backgrounds at the bottom of the table. Answer which image best describes: place referring expression? Answer with: left, middle, or right. The results show that the base VLM achieves an 81% accuracy on spatially localizing the destination image across 288 test scenarios, demonstrating that it already possesses a strong visual understanding of the scene layout before any robotic fine-tuning is applied. However, after finetuning with OXE data and the WISER training data, its TCP reaching success rate is only 51%, indicating that this spatial localization capability is somehow compromised. 6.5
GT-MPC for LIBERO-goal
LIBERO [39] is a widely adopted benchmark for VLAs. Among its 100 tasks, only 10 from the LIBERO-Goal split strictly require semantic understanding. This is because these tasks share identical scene layouts, compelling VLAs to differentiate between them solely based on task instructions. For the remaining tasks, a purely visuomotor policy often suffices to map the scene layout directly to the target trajectory or action without needing to process the instruction. Consequently, we evaluate our Model Predictive Control (MPC) framework equipped with Qwen3-VL-Embedding specifically on this split. Since the LIBERO demonstrations are collected in the exact same test environments with identical visual appearances and instructions, future trajectories can be proposed using KNN, and the respective future frames can be directly retrieved from the training dataset. Also, we do not need to train a GWM, because we have the GT future videos already. Following the standard LIBERO evaluation protocol, we run 50 episodes for each task. The results are presented in Table 4. These results demonstrate that Qwen3-VL-Embedding serves as an effective zero-shot video classifier, capable of selecting the optimal action by evaluating the future observations given all candidates. Our GT-MPC system yields a zero success rate on only two tasks. This failure stems from an inability to recognize the correct behavior required to fulfill the event described by the prompt. For the task “open the middle drawer of the cabinet”, the system fails because the scoring function initially assigns a higher value to the action trajectory associated with “open the top drawer and put the bowl inside”. Nevertheless, this indicates that the foundation model successfully captures the correct macro movement direction for the gripper. Among the completed tasks, several do not achieve a 100% success rate. This is primarily because the KNN action generator lacks robustness against small perturbations in object positions. In addition, the error is accumulated in closed-loop running because of using the delta action space. Employing a learning-based visuomotor policy for action proposal could effectively alleviate this issue. Table 4: Task Success Rates on libero-goal Split Task
Description
0 1 2 3 4 5 6 7 8 9
open the middle drawer of the cabinet put the bowl on the stove put the wine bottle on top of the cabinet open the top drawer and put the bowl inside put the bowl on top of the cabinet push the plate to the front of the stove put the cream cheese in the bowl turn on the stove put the bowl on the plate put the wine bottle on the rack
SR 0.0% 72.0% 96.0% 80.0% 100.0% 98.0% 0.0% 100.0% 68.0% 100.0%
Average
71.4%
18
Figure 7: Libero-goal environment.
6.6
Model Architectures & Hyperparameters
The transformer backbone for the GWM and the action-conditioned version has the same structure and uses the same training hyperparameters. The details can be found in table 5. The difference between the two action tokenization schemes is shown in Fig. 8.
Figure 8: Difference between GWM and its raw action conditioned version. Captured images are just exemplary; the main camera is placed in front of the robot as shown in Fig. 4.
Table 5: Transformer configuration and hyperparameters of the (GWM). Hyperparameter Value Architecture Hidden dimension (dmodel ) FFN intermediate dimension (dffn ) Attention head dimension (dhead ) Number of layers Number of attention heads Number of KV heads (GQA) Input / Output dimension Input sequence length Positional encoding Normalization FFN activation Precision
4096 8192 128 5 32 8 4096 1620 2D RoPE RMSNorm (ϵ = 10−5 ) SwiGLU bfloat16
Training Optimizer Learning rate (Muon, hidden weights) Learning rate (Adam, embed/head) Adam β Weight decay LR scheduler Min learning rate Epochs Gradient clipping Loss function
Muon [26] + Adam [30] 0.01 5 × 10−5 (0.9, 0.95) 0.01 Cosine annealing 10−6 10 1.0 MSE
19
6.7
WISER Benchmark
All training and test tasks are shown as follows. Images are AI-generated to avoid copy right issue.
20
21
22
23
24
25
26
27