1
Learning Action Priors for Cross-embodiment Robot Manipulation
arXiv:2606.26095v1 [cs.RO] 24 Jun 2026
Dong Jing1,2 , Tianqi Zhang2 , Jiaqi Liu2 , Jinman Zhao3 , Zelong Sun1 , Li Erran Li4 , Zhiwu Lu1† , Mingyu Ding2† 1 Renmin University of China, 2 University of North Carolina at Chapel Hill, 3 University of Toronto, 4 Amazon While the VLM already encodes rich semantic structure, the action module is often initialized from random weights or from parameters transferred from unrelated modalities. It therefore leaves the action module to learn physical motion and temporal dynamics from scratch, creating a bottleneck in modeling robot action distributions. As a result, early VLA training must learn both the action distribution and the cross-modal alignment between perception, language, and control at the same time. The gradients produced by such an under-trained action head can be unstable and poorly structured, interfering with the foundation backbone and slowing convergence. This issue becomes more pronounced in cross-embodiment settings [20], [21], where different robots exhibit substantially different tasks, action spaces, and motion distributions. To address this imbalance, we introduce action prior learning before cross-modal VLA alignment, targeting only the action module and using only action data. As shown in Figure 1(a), the action module first learns to move from unconditioned action trajectories, without any visual observation or language instruction. Much like a blindfolded apprentice practicing basic movements, it focuses on the intrinsic temporal structure of physical motion instead of learning action dynamics and crossmodal alignment simultaneously. The learned prior is then transferred to Stage 2, where the VLA learns to perceive the scene and follow instructions on top of an action module that already encodes plausible motion patterns, as illustrated in Figure 1(b). This decoupled training process yields smoother Index Terms—action prior, vision-language-action model, crosstrajectories and faster convergence, as summarized in Figembodiment, robot manipulation. ure 1(b,c), because VLA training begins from a structured action module rather than an untrained action head. I. I NTRODUCTION We instantiate this action prior with an encoder-decoder In recent years, VLAs have rapidly emerged as a domi- action module. The encoder is designed to extract the motion nant paradigm for robotic manipulation. These models build semantics that emerge across an action sequence, rather than upon foundation VLMs [1], [2], [3], [4] or video generation from isolated atomic actions. A single action usually records models [5], [6], [7], [8], [9], which encode rich multi-modal only an infinitesimal state change over a short time interval, semantic knowledge. Such semantic foundations allow VLAs to while meaningful behaviors such as translating, rotating, or connect high-level language instruction and visual observations grasping arise from the continuous composition of actions over with low-level physical execution. This paradigm has achieved a temporal horizon. The encoder therefore compresses an entire remarkable success across diverse robotic platforms, such as action chunk into a compact latent embedding that captures tabletop manipulators [10], [11], [12], quadruped robots [13], its temporal dynamics and macroscopic motion structure. [14], and humanoid systems [15], [16], [17]. These advances Conditioned on this embedding, the decoder reconstructs the open a credible path toward generalist robotic policy. However, original action chunk via a flow-matching objective, which the success of current VLAs is still largely driven by visual and provides a flexible way to model the continuous action linguistic priors inherited from large-scale foundation-model distribution and its uncertainty. training, while the action generation module itself usually We transfer this learned prior to VLA training through three receives no comparable prior over physical motion. tightly connected uses of the learned action module: decoder Missing action priors create a key imbalance in the standard reuse, early-stage latent distillation, and history compression. VLA training pipeline. Existing methods typically append At the action-output side, the decoder that already models an action module to a foundation VLM and optimize the plausible robot motion from Stage 1 is reused as the VLA action whole model jointly via imitation learning [11], [18], [19]. head. At the representation side, the encoder provides a latent Abstract—Most Vision-Language-Action (VLA) models build on a Vision-Language Model (VLM) backbone by attaching an action module and optimizing the full policy jointly. This design inherits strong visual and linguistic priors from the VLM, but leaves the action module to learn physical motion almost from scratch. As a result, the policy lacks an explicit motion prior, forcing early optimization to simultaneously discover temporal action dynamics and cross-modal alignment, a challenge further amplified in cross-embodiment settings with heterogeneous action distributions. In this work, we propose to pretrain the action module with motion priors before cross-modal VLA alignment. Specifically, we introduce a two-stage training framework that equips the action module with cross-embodiment temporal motion structure before VLA training begins. In Stage 1, a lightweight flow-matching-based encoder-decoder action module efficiently learns temporal motion structure solely from unconditioned action trajectories, without processing visual or language tokens. In Stage 2, this learned prior is transferred to VLA training through decoder reuse and early-stage latent distillation, aligning visuallanguage features with the action embedding space while still allowing end-to-end policy refinement. In addition, the trained encoder serves as a compact history compressor, summarizing state-action histories into a single temporal context token for history-aware modeling at negligible cost. Extensive experiments across 13 diverse cross-embodiment tasks on both simulated and real-world platforms validate the effectiveness of our approach. Compared with VLA training without action priors, our model achieves faster convergence, higher success rates, and substantially stronger performance on data-scarce real-world tasks. Moreover, scaling up the action data in Stage 1 yields a more generalizable action prior that directly improves downstream VLA performance.
2
(a) Stage 1: Learn to move
(b) Stage 2: Learn to see and act
(c) Effect of Learning Action Prior
Action Prior Transfer Pick up the red cup
w/ action prior smooth & efficient w/o action prior wobbly & inefficient
No Vision and Instruction. Just Learn Movements.
See, Understand, Act. Faster and Better Convergence.
Fig. 1: Illustration of motivation: a policy should first learn to move, and then learn to see and act. (a) In Stage 1, the action module is trained purely on action trajectories, without any visual observation or language instruction, to efficiently acquire a general action prior. (b) In Stage 2, the prior is transferred to VLA training, enabling model to perceive scenes and follow instructions with an action module that already knows how to move. With the action prior, the policy produces smooth trajectories. Without it, the trajectories are unstable. (c) The action prior accelerates VLA training and improves performance. target for early-stage distillation: given the ground-truth action bottleneck in VLA training, and introduce a two-stage chunk, it produces a structured embedding that supervises the framework that efficiently learns action-centric robot VLM’s predicted action feature and quickly anchors it to the motion before cross-modal policy alignment. learned motion latent space. This alignment is gradually relaxed, 2) We introduce a flow-matching encoder-decoder action allowing later training to refine the policy end-to-end without module that learns compact motion-aware embeddings being permanently constrained by the encoder. Finally, the and transfers them as action priors for VLA training. This encoder also serves as a compact history compressor, mapping structure further yields a history context token for efficient past state-action trajectories into a single latent token that injects history-aware decision making. temporal context into the VLM. Together, these mechanisms 3) We validate the method across 13 cross-embodiment turn the action prior from a standalone reconstruction model tasks in simulation and the real world, showing faster into a practical initialization, alignment signal, and memory convergence, higher success rates, improved long-tail interface for VLA training, leading to smoother rollouts and stability, and favorable scaling with more action data. faster convergence as shown in Figure 1(b,c). We evaluate our method on a demanding cross-embodiment II. R ELATED W ORK setup that spans 13 tasks across two simulated benchmarks, LIBERO and RoboCasa, and a real-world Franka platform. All A. Vision-Language-Action Models models are trained jointly on heterogeneous embodiments and Vision-Language-Action models (VLAs) [22], [23], [24], evaluated without environment-specific fine-tuning. Our model [17], [16], [25], [26], [27], [28] have rapidly emerged as a has several appealing benefits. 1) Action prior learning provides dominant paradigm for robotic manipulation by leveraging the most direct benefits expected from the proposed design: the semantic priors of foundation Vision-Language Models faster convergence and higher success rates over from-scratch (VLMs) [29], [30]. Early efforts such as RT-2 [11] demonstrated VLA training. 2) The gains are especially pronounced on data- that directly fine-tuning a VLM on robotic data can produce scarce long-tail tasks. In real-world tasks with a small number effective visuomotor policies. Subsequent works have explored of demonstrations, the action prior improves long-tail stability diverse architectural designs. OpenVLA [12] and its efficient by making the policy behave more stably and decisively, variant OpenVLA-OFT [31] adopt an open-source VLM mitigating the uneven convergence in which data-rich simulated backbone with action tokenization for scalable training. On the tasks dominate while rare real-world behaviors remain underfit. action generation side, Diffusion Policy [32] and ACT [33] 3) We further observe favorable scaling with action-only demonstrate that expressive generative models can effectively data: increasing the amount of unconditioned action data in capture multi-modal action distributions, and RDT-1B [34] Stage 1 yields a broader action prior, and it transfers directly to scales diffusion-based action generation to a foundation model downstream VLA training. 4) History integration brings huge for bimanual manipulation. Building on these ideas, Cogains for long-horizon tasks via enlarged action receptive field. gACT [19] appends a diffusion-based action module to a VLM, The learned encoder effectively extracts and compresses past generating actions conditioned on the VLM’s latent features. state-action trajectories into a compact latent token, expanding π0 [18] employs a mixture-of-transformers architecture with a the temporal context visible to the policy. flow-matching action head, and π0 -FAST [35] further improves Our main contributions are summarized as follows: efficiency through discrete action tokenization. Other lines of 1) We identify the absence of action priors as a structural work focus on enriching spatial reasoning [36], [37], [38],
3
incorporating chain-of-thought reasoning [39], [40], [41], visual trace prompting [42], or introducing world model capabilities from Video Generatation Models [43], [24], [44], [45], [46]. Despite these advances, most VLA models still rely on action modules that lack motion priors comparable to the semantic priors inherited by the VLM backbone. This creates a huge gap between the foundation model’s visual-language knowledge and the action head’s understanding of physical motion. B. Cross-Embodied Robot Manipulation
Another line of work learns action-like representations from visual trajectories or videos. LAPA [59] discretizes latent actions between consecutive video frames with a VQ-VAE [60]. IGOR [61] uses image-goal representations as atomic control units, while Moto [62] learns latent motion tokens through a video-generation-style objective. These methods use frameto-frame visual change as supervision, making the learned representation closely tied to visual prediction. This design is attractive for web-scale videos [63], [64], [65], but the resulting tokens often describe how images evolve rather than directly modeling the robot’s motor distribution. World-model-based approaches further use visual dynamics to learn latent action or motion structure. VLA-JEPA [66] predicts future state representations with a latent world model, and LAWM [67] learns latent actions through self-supervised world modeling on unlabeled videos. AdaWorld [68] learns adaptable world models with latent actions, while DreamDojo [69] scales generalist robot world modeling to large-scale human videos. DynaMo [70] learns dynamics-aware visual features through in-domain dynamics prediction. These methods place greater emphasis on predicting forward dynamics in the visual space. Action-like variables are then recovered by inverting these dynamics, which differs from directly modeling the robot action distribution. In contrast, our goal is to fit the action space itself as a reusable prior for downstream VLA training. Overall, these approaches show that latent action or dynamics representations can improve robot policy learning. Most of them, however, start from visual data, future-state prediction, or vision-action policy learning. Our setting instead uses robot action trajectories directly, without visual observations or language instructions. This isolates the intrinsic statistics of physical motion and keeps the learning stage lightweight.
Training a single policy across multiple robot embodiments promises broader generalization and improved data efficiency. This vision has been fueled by the emergence of largescale, multi-embodiment datasets. Open X-Embodiment [20] consolidated over one million trajectories from 22 robot platforms across 60+ datasets, establishing the first large-scale cross-embodiment benchmark. BridgeData V2 [47] provides over 60k trajectories of diverse manipulation behaviors for scalable, open-vocabulary policy learning. DROID [48] collects 76k demonstrations across 564 scenes by 50 operators spanning three continents. AgiBot World [49] further scales to over one million trajectories across 217 tasks collected from 100 real robots. RoboMIND [50] contributes 107k trajectories across four embodiments with a unified collection protocol. RoboTwin [51] provides a scalable simulation benchmark for bimanual manipulation with domain randomization. LIBERO [52] and RoboCasa [53] offer diverse simulation benchmarks for lifelong learning and everyday household tasks, respectively. On the modeling side, Octo [54] adopts a transformer architecture with modality-specific tokenizers to handle heterogeneous observation and action spaces. GR00T N1 [16] leverages a diffusion transformer for humanoid robot control across diverse embodiments. X-VLA [55] introduces softIII. M ETHODOLOGY prompt tokens and a unified action space to disambiguate between embodiments, enabling scalable cross-embodiment A. Preliminary and Task Formulation training. UniVLA [25] learns task-centric latent actions to Standard VLA Formulation. Vision-Language-Action models achieve embodiment-agnostic policy transfer. Universal Ac- (VLAs) formulate robotic manipulation as a conditional action tions [56] proposes a standardized action representation that generation task. Given visual observations ot and a language maps heterogeneous action spaces into a common format. A instruction lt at timestep t, the objective is to predict a future key challenge in cross-embodiment settings is that the diverse action chunk a = [at , at+1 , . . . , at+H−1 ], where H denotes action distributions across platforms significantly increase the the action chunk horizon [33], [71]. Most VLAs are built upon difficulty of action prediction. This motivates mechanisms foundation Vision-Language Models (VLMs). Let θ denote the that expose the policy to heterogeneous action distributions parameters of the backbone. To introduce the action modality, before cross-modal policy learning, rather than leaving the an action module parameterized by ϕ is appended, and the action module to absorb embodiment differences only through full parameters {θ, ϕ} are optimized jointly via a behavioral downstream VLA supervision. cloning (BC) objective over the embodiment dataset D: C. Action Prior Learning Action prior learning is related to methods that introduce action-side structure before or during policy learning. ACT [33] uses a CVAE [57] encoder-decoder over action sequences, but its encoder is observation-conditioned and trained end-to-end inside the full policy. APT [58] follows a staged VA-to-VLA recipe, first learning a vision-action expert and then transferring it to instruction-conditioned VLA training. It is related to our sequential training setup, but it is mainly designed to improve instruction following.
min E(o,l,a)∼D [LBC (fθ,ϕ (o, l), a)], θ,ϕ
(1)
where fθ,ϕ represents the entire VLA network, and LBC denotes a generic behavioral cloning objective for action prediction, instantiated as regression, diffusion, or flow matching depending on the action head. Bottleneck. However, a critical bottleneck arises during the initial phase of training. Because the action module parameters ϕ are typically initialized randomly or transferred from a disjoint modality such as vision or language, the action module
4
Stage 1: Learn Action Prior
Stage 2: Action-Prior-Guided VLA Training
(action-only training, no vision, no language)
(distill prior into VLM, reuse decoder as action head)
#"#$
... #"#%
ℒ()+,-
#"#$
... #"#%
Early-stop Distillation Schedule
ℒ'()*
' ( 1
reuse
Flow-matching Decoder
timestep "~$(0,1)
"
timestep "~$(0,1)
' ( = max(0, 1 − (/1)*+", )
Flow-matching Decoder 0
1)*+",
step (
ℒ-.-"# = ℒ/0*) + ' ( 4 (ℒ"#$%& + ℒ0*+.& )
Latent Embedding *(+,-, .! , !)
51 (62 , 7- , 8- , #3$4- , 9)
"′
'56789 = (' − )* (
:
(stop-grad on 1)
Encoder !
"
:(;8%, 62 , <)
Latent Alignment Distillation
!!
%&"
""#$ #"#$ ... ""#% #"#%
soft dataset tokens .!
Encoder !
ground-truth states & actions <
Pre-trained VLM $%
interleaved proprioceptive states and actions !
reuse (optional)
""#$ #"#$ ... ""#% #"#%
""&% #"&% ...
""
#"
history states & actions
!!
Data Sources LIBERO
RoboCasa
Real World
dataset images tokens
text
history query token 1$%&# token
History Compression
Fig. 2: Architecture of our proposed framework. In Stage 1, interleaved state-action sequences and dataset-specific soft-prompt tokens are fed into the encoder to produce a latent action embedding z. The flow-matching decoder then reconstructs the original actions conditioned on z. In Stage 2, the VLM processes soft data tokens, visual observations, language instructions, and a learnable query token to predict z ′ . The predicted embedding z ′ is aligned with the encoder’s z through a decaying distillation loss and passed to the Stage 1-initialized decoder for action generation. The same encoder compresses historical trajectories into a single token zhist , injecting temporal context into the VLM at negligible cost. enters training with little knowledge of the temporal structure and distribution of robot motion. Early optimization must solve two coupled problems at once: the action module has to learn a continuous action distribution, while the VLM backbone has to align visual-language representations with this stillevolving action space. When the action head is under-trained, the gradients ∇θ LBC backpropagated to the VLM can be poorly structured and unstable, forcing the foundation backbone to adapt to a moving action target and slowing convergence.
extended action chunks into meaningful latent embeddings. Crucially, isolated atomic actions carry little semantic meaning on their own. It is the continuous composition of actions across a temporal horizon that gives rise to meaningful physical motions, such as translating, rotating, and grasping. The action module should therefore capture these temporal dynamics and project them into a regularized latent space. Formulation. To learn this prior, we introduce a standalone action module comprising an encoder Eϕenc and a decoder Dϕdec , with joint parameters ϕ = {ϕenc , ϕdec }. Given trajectory data τ with action chunk a from the embodiment dataset D, we optimize ϕ by minimizing an action reconstruction objective without any visual-language conditioning:
B. Learning Action Prior Motivation. While large-scale visual-language training successfully establishes robust visual and linguistic representations in VLMs, the action modality itself contains rich intrinsic prior knowledge that remains largely underexploited. As shown in Figure 2, we therefore introduce a dedicated action prior ϕ∗ = arg min E(τ,a)∼D [Lrecon (Dϕdec (Eϕenc (τ )), a)], (2) ϕ learning stage that trains the action module independently on low-level trajectory data through a reconstruction objective, enabling it to acquire an Action Prior before it is coupled with where ϕ∗ = {ϕ∗enc , ϕ∗dec } denotes the parameters that successvisual-language inputs. This stage is action-centric because no fully capture the action prior. visual observations or language instructions are used. The learned action module is then reused in Stage 2 in three Definition of Action Prior. We define the Action Prior along complementary ways: the decoder initializes the VLA action two fundamental dimensions: 1) Distribution Modeling: the head, the encoder provides a structured latent target for earlycapability to fit the underlying continuous action distribution. stage distillation, and also compresses historical state-action 2) Structural Abstraction: the ability to compress temporally trajectories into a compact context token.
5
C. Encoder-Decoder Action Modeling
D. VLA Training with Action Prior Distillation
Having acquired a robust action prior through the indepenWe instantiate the Stage 1 action module with a Transformerdent learning stage, we transfer it to VLA training by reusing based encoder-decoder architecture. As illustrated in Figure 2, the learned action module and distilling its latent structure into this module is trained independently on low-level trajectory the VLM. data via the reconstruction objective, decoupling the acquisition of the action prior from visual-language alignment. VLA Architecture. As shown in Figure 2, we adopt a Action Encoder. The action encoder Eϕenc compresses a low- foundation VLM as the backbone fθ . At timestep t, the VLM level trajectory into a single dense latent embedding that cap- takes the soft-prompt dataset embeddings pk , current visual tures the global temporal structure of motion. To construct the observations ot , the language instruction lt , and a learnable input sequence, we interleave the robot’s proprioceptive states query token q as inputs. The output feature corresponding to s and actions a into a sequence τ = [st , at , st+1 , at+1 , . . .]. q is extracted and ℓ2 -normalized as the predicted action latent ′ Actions describe local motion increments, while proprioceptive embedding z : z ′ = fθ (pk , ot , lt , q). (6) states anchor these increments to the robot’s body configuration. In general cross-embodiment settings, state and action spaces Instead of learning the action head from scratch, we initialize differ in dimensionality, semantics, and control conventions the VLA action decoder with the Stage 1 decoder and jointly across platforms. To mitigate this heterogeneity, following X- optimize it during VLA training. Conditioned on z ′ , the decoder VLA [55], we prepend learnable dataset embeddings pk to predicts the velocity field for action generation, yielding the indicate the source embodiment and disambiguate embodiment- primary prediction loss: specific action semantics. Lpred = ∥Dϕdec (xr , r, z ′ ) − vr ∥2 . (7) A learnable summary token cls is also prepended, and the full encoding process is formulated as: Latent Alignment Distillation. The Stage 1 encoder provides z = Eϕenc (cls, pk , τ ), (3) a direct way to supervise the VLM in the learned action latent space. Given the ground-truth trajectory, the action encoder where z is the ℓ2 -normalized output feature of cls, serving as E ϕenc extracts a structured latent embedding z, which serves the latent action embedding. as a teacher target for the VLM-predicted embedding z ′ . The By aggregating the entire sequence into a single token, alignment loss is defined as: z is explicitly trained to capture the global dependencies Lalign = ∥z ′ − sg(z)∥2 , (8) and structural dynamics of the action chunk, representing the macroscopic motion semantics that emerge from the continuous where sg(·) denotes the stop-gradient operator. This prevents combination of actions over a temporal horizon. the alignment objective from pulling the encoder toward the Flow-Matching-based Action Decoder. Conditioned on the latent embedding z, the decoder Dϕdec reconstructs the original action chunk. We adopt a flow-matching paradigm for this purpose, which provides a flexible generative objective for continuous and potentially multi-modal action distributions. Sharing a similar Transformer architecture as the encoder, the decoder takes the latent embedding z alongside noisecorrupted actions xr as input. We define a probability path between standard Gaussian noise ϵ ∼ N (0, I) and the groundtruth actions a, where the noise-corrupted actions at training flow-matching timestep r ∈ [0, 1] is constructed via linear interpolation: xr = ra + (1 − r)ϵ. (4)
current VLM prediction, while still encouraging the VLM to enter the motion-aware latent space learned in Stage 1. To maintain the coherence of this latent space during joint VLA optimization, we also keep the reconstruction objective Lrecon on the encoder-decoder branch. This auxiliary loss uses the same flow-matching formulation as in Stage 1 with the groundtruth embedding z, preserving the action latent geometry rather than directly supervising the VLM prediction. Early-step Distillation Strategy. While Lalign provides critical early-stage guidance, maintaining this strict constraint throughout training could overly restrict the VLM’s representational flexibility and hinder end-to-end performance refinement. We therefore propose an early-step distillation strategy, introducing a monotonically decaying weight λ(k) at training step k: k λ(k) = max 0, 1 − , (9) Ndecay
The target velocity field that drives noises toward clean actions is vr = a − ϵ. Following π0.5 [72], the timestep r is integrated into the normalization layer by projecting it into modulation weights. where N decay is a predefined decay threshold. The decoder is optimized as a velocity prediction network The total VLA training objective is formulated as: using the flow-matching objective: L = Lpred + λ(k)(Lalign + Lrecon ) (10) Lrecon = ||Dϕdec (xr , r, z) − vr ||2 (5) During early training, the distillation losses anchor the VLM By minimizing this objective over low-level trajectory data to a physically meaningful action representation subspace. As without visual-language conditioning, the decoder learns a k exceeds Ndecay , these constraints are phased out, and the robust and generalizable action distribution prior, laying a model transitions to standard end-to-end VLA optimization stable foundation for the subsequent VLA training. driven by Lpred .
6
generalization across different spatial layouts, object categories, task goals, and long-horizon sequences, respectively. The four task suites require progressively longer trajectories. During evaluation, we test each task for 50 episodes with a fixed random seed, resulting in 500 episodes per suite. At each inference step, the model predicts an action chunk and the first 5 actions are executed by default, after which a new prediction is made. This cycle repeats until the task is completed or the maximum step limit is reached. RoboCasa [53]. RoboCasa is a large-scale simulation framework featuring diverse manipulation environments built around Encoder-based History Compression. We address this issue the GR1 humanoid robot platform. We select 5 tabletop tasks by reusing the action encoder as a compact history compressor. from the RoboCasa GR1 benchmark, each involving structured Because the encoder is trained to summarize trajectory structure rearrangement operations across varied objects and receptacle in Stage 1, the same mechanism can also extract motionconfigurations. For each task, we use 200 demonstration aware context from past state-action sequences. Let τhist = trajectories, yielding 1,000 trajectories in total for training. [st−Hh , at−Hh , . . . , st−1 , at−1 ] denote the historical trajectory Each task is evaluated for 50 episodes, totaling 250 episodes over a horizon Hh . As illustrated in Figure 2, we pass τhist across all five tasks. The first 10 actions of each predicted through the learned encoder to obtain a single historical latent action chunk are executed. token zhist : Real-World Tasks. Beyond simulation, we design 4 pick-andzhist = Eϕenc (cls, pk , τhist ). (11) place tasks on a real Franka robot arm for evaluation. The training demonstrations are collected via teleoperation, with This token is then inserted into the VLM input, updating the 50 demonstration trajectories per task. Each task involves a forward pass to condition on temporal context: distinct set of objects and requires qualitatively different motion z ′ = fθ (pk , ot , lt , zhist , q). (12) trajectories, ensuring diversity in both scene configuration and manipulation style. Each task is evaluated for 10 trials, and This design injects historical information in a form that is the first 8 actions of each predicted action chunk are executed. both compact and semantically structured. Instead of exposing The full training mixture contains approximately 565k statethe VLM to raw numerical trajectories, zhist carries motion action transitions. The four real-world tasks contribute only dynamics already organized by the action prior. Meanwhile, 42,967 frames, accounting for about 7.6% of the mixture, which compressing the entire history into one token provides a wider makes them a natural long-tail testbed. temporal view with negligible computational overhead. Unified Action-State Space. To enable joint training across History-conditioned Training. During training, we compute heterogeneous embodiments, we define a unified action space the full objective in Equation 10 under both the historyof 37 dimensions and a unified state space of 74 dimensions, conditioned and history-free settings, and average two losses: as detailed in Table I. Each embodiment’s native action and Ltotal = (L(zhist ) + L(∅))/2, (13) state vectors are mapped into corresponding slots of the unified vector, with all unused dimensions zero-padded. Specifically, where L(zhist ) and L(∅) denote the training objective condiLIBERO uses 7-DoF end-effector delta actions (consisting of tioned with and without the historical embedding, respectively. 3-DoF position, 3-DoF rotation, 1-DoF gripper) mapped to the This dual-mode training encourages the policy to benefit from right-arm slot, RoboCasa GR1 uses 29-DoF actions covering temporal context when available, while avoiding over-reliance both arms, hands, and waist, and real-world Franka uses 8on the history token. DoF joint-space actions. For states, LIBERO provides 8-DoF end-effector readings, real-world Franka provides 8-DoF joint IV. E XPERIMENT angles, and GR1 provides full-body joint states expanded to A. Experimental Setup 58 dimensions via sine-cosine transformation. This slot-based design prevents interference between embodiments, while the Cross-Embodiment Data Setting. As shown in Figure 3, our learnable dataset tokens pk further disambiguate embodimentexperiments cover 13 tasks across two simulated benchmarks specific patterns. (LIBERO and RoboCasa) and a real-world Franka platform. The three environments differ substantially in embodiment type, Baselines and Variants. We evaluate our method against action and state representation, scene composition, and task representative VLA architectures and internal variants under distribution, making joint policy learning considerably more the same cross-embodiment training protocol. For a controlled demanding than single-embodiment training. All models are comparison, all models are trained within the same StarVLA trained on the full cross-embodiment mixture and evaluated codebase [73] on the identical cross-embodiment dataset, on each benchmark without environment-specific fine-tuning. without using externally trained VLA checkpoints. To ensure LIBERO [52]. LIBERO comprises four task suites: Spatial, fairness and enhance cross-embodiment capability, all methods Object, Goal, and Long. Each suite contains 10 tasks and use learnable soft-prompt dataset tokens pk . 500 demonstrations for training, and is designed to probe No Action Prior. This baseline adopts the same encoderE. Temporal Context Compression with the Action Encoder
Motivation. Sequential manipulation often requires temporal context beyond the current observation. Past states and actions reveal whether the robot has approached, grasped, hesitated, or already completed part of a motion. However, directly appending raw state-action history to the VLM input is inefficient and poorly matched to the VLM’s image-language token space. Long histories introduce substantial token overhead, and their low-level numerical values force the VLM to learn temporal abstraction from scratch.
7
LIBERO
Spatial
RoboCasa GR1
Object
Bottle to Cabinet Close
Cuttingboard Box to Basket
Real-world Tasks
Placemat to Basket
Grasp Middle Cup
Grasp Coke
Move Tide
Stack Cups
Cross-embodied 13 tasks in total across 3 benchmarks
Goal
Long
Plate to Bowl
Tary to Cardboard Box
Fig. 3: Overview of the 13 cross-embodiment tasks across LIBERO, RoboCasa GR1, and real-world Franka manipulation. The suite covers heterogeneous embodiments, action-state spaces, scenes, and task distributions. TABLE I: Unified 37-D action and 74-D state representation across embodiments. Unused slots are zero-padded.
State 74D
Action 37D
Component
Dim
Left arm Right arm Left hand Right hand Waist Joint angles + gripper
[0:7] [7:14] [14:20] [20:26] [26:29] [29:37]
EE pose + gripper Joint angles + gripper Full body
[0:8] [8:16] [16:74]
LIBERO ✓
GR1
TABLE II: Training configuration for Stage 1 action prior learning and Stage 2 VLA training.
Real
✓ ✓ ✓ ✓ ✓ ✓
✓ ✓ ✓
decoder action architecture and flow-matching decoder as our full method, but entirely removes Stage 1 action prior learning. The action module is initialized with generic LLM parameters as shown in Table II and trained jointly with the VLM using only the prediction loss Lpred . This configuration is structurally similar to CogACT [19], and therefore also serves as a direct comparison with that approach. Action-State Prior. This variant introduces Stage 1 action prior learning with both action and proprioceptive state trajectories, and transfers the learned module to Stage 2 through decoder initialization and latent distillation. Action-State Prior + History. This is our full model, which additionally injects the encoder-compressed history token into the VLM during Stage 2. GR00T [16]. GR00T employs a diffusion transformer [74], [75], [76] as the action prediction head. The VLM backbone extracts visual-language features that condition the diffusion transformer through cross-attention layers to generate actions via iterative denoising. We re-train GR00T under the same data and optimization protocol for a controlled comparison. π0.5 [72]. π0.5 features a mixture-of-transformers architecture comprising a VLM for visual-language understanding and a lightweight action expert for action generation. The two modules interact through cross-attention, and the action expert
Backbone Chunk size H Batch size Optimizer Learning rate LR scheduler Min LR Training steps Warmup steps Ndecay GPUs Training time
Stage 1
Stage 2
Qwen3-0.6B 15 2048 AdamW 1×10−4 Warmup-Cosine 5×10−7 5,000 1,000 – 8×H200 2 hours
Qwen3-VL-2B 15 256 AdamW 2.5×10−5 Warmup-Cosine 2.5×10−8 50,000 5,000 5,000 8×H200 20 hours
generates actions via flow-matching. We adopt PaliGemma-2B as the VLM backbone and Gemma-300M as the action expert. Training hyperparameters. Table II summarizes the training configurations for both stages. In Stage 1, the action encoder and decoder are initialized with Qwen3-0.6B [77] and trained solely on unconditioned cross-embodiment state-action trajectories to acquire the action prior. In Stage 2, we adopt Qwen3VL-2B [78] as the VLM backbone and jointly optimize the VLM and action module for 50,000 steps across 13 crossembodiement tasks. The alignment and reconstruction losses are linearly decayed to zero over the first Ndecay = 5,000 steps. All experiments are conducted on 8×NVIDIA H200 GPUs with BF16 mixed precision. B. Main Results on Cross-Embodiment Manipulation Overall Performance. Table III summarizes the comparison across all 13 cross-embodiment tasks. We focus on whether action prior learning improves VLA training under two central challenges: heterogeneous embodiments and uneven data distributions, especially the long-tail setting represented by realworld tasks. The results show three consistent benefits: stronger overall performance, substantially better long-tail real-world behavior, and complementary gains from history compression.
8
TABLE III: Comparison with state-of-the-art methods on 13 cross-embodiment tasks. Success rates (%) are reported. All methods use the same StarVLA codebase and training data. Sim Avg. denotes the average success rate over the 9 simulation tasks from LIBERO and RoboCasa GR1, and Overall Avg. averages all 13 tasks. Real-world task abbreviations: GMC: Grasp Middle Cup; GC: Grasp Coke; MT: Move Tide; SC: Stack Cups. LIBERO
Method
RoboCasa GR1
Spatial
Object
Goal
Long
B2C
C2B
Pm2B
Pl2B
T2C
GR00T [16] π0.5 [72]
91.2 95.6
97.6 98.4
92.2 95.8
88.6 91.0
56 62
38 46
20 46
8 28
50 32
No Action Prior Action-State Prior Action-State Prior + History
96.4 97.0 97.6
97.6 97.2 99.0
96.6 97.6 96.6
90.2 91.0 96
62 68 56
46 40 50
28 42 50
18 22 28
44 44 46
Sim Avg.
Real-world Franka
Overall Avg.
GMC
GC
MT
SC
60.2 66.1
45 60
15 30
20 15
10 0
48.6 53.8
64.3 66.5 ↑2.2 68.8 ↑4.5
65 75 75
5 35 50
45 60 60
25 75 80
55.3 64.9 ↑9.6 68.0 ↑12.7
Action prior learning improves both simulation and real- Without the action prior, the policy moves as if unfamiliar world performance. Action-State Prior improves the overall with its own arm. On the data-scarce real-world tasks, the average from 55.3% to 64.9%, and adding history further raises No Action Prior policy frequently produces jerky and unstable it to 68.0%. It also improves the simulation average from motions. In Fig. 4(a), for example, it grasps the first cup within 64.3% to 66.5%, with the history variant reaching 68.8%. The 10 seconds but then freezes before aligning it with the second gain is especially pronounced on the real-world Franka tasks, cup, remaining stuck until timeout. With the action prior, the where the average success rate increases from 35.0% to 61.3% motions become fast and precise, and the tasks are completed with the action prior and to 66.3% with history enabled. Our in a stable manner. We attribute this difference to the division full model also clearly outperforms GR00T and π0.5 , which of labor in our two-stage design. The action prior stage has reach 48.6% and 53.8% overall, respectively. Notably, π0.5 already fitted the action distribution of the dataset efficiently. remains competitive in simulation but collapses on the real The VLA stage therefore mainly learns to match the given robot, whereas our approach maintains strong performance task with motion patterns that the action module has mastered, across both domains. leaving little hesitation during execution. The action prior helps most on long-tail real-world tasks. Each real-world task contains only 50 teleoperated demonstrations; together, the real-world subset contributes only 42,967 frames, or 7.6% of all training frames, forming the data-scarce tail of the mixture. The gains from the action prior are largest on this tail: Grasp Coke improves from 5% to 35%, and Stack Cups improves from 25% to 75%. In contrast, the competing methods struggle on these tasks, suggesting that the action prior is particularly valuable when downstream VLA supervision is sparse and unevenly represented. π0.5 provides a clear example: despite its strong simulation performance, it fails on every Stack Cups trial. We analyze the behavioral causes in Section IV-C. History compression provides a complementary boost. Enabling history raises Grasp Coke from 35% to 50% and Stack Cups from 75% to 80%, lifting the overall average from 64.9% to 68.0%. We attribute this gain to the enlarged temporal receptive field provided by the compressed history token, which helps reduce hesitation around critical decision points, as analyzed below. This suggests that the encodercompressed history is most useful when the policy must decide whether to keep aligning or commit to the terminal action. C. Qualitative Analysis of Long-tail Real-world Behavior To understand why the action prior helps most on the datascarce real-world tasks, we manually inspect the rollouts of all methods. Fig. 4 visualizes representative key frames on the two most challenging long-tail tasks, Stack Cups and Grasp Coke. The comparison reveals three characteristic behavior patterns that are not visible from success rates alone.
π0.5 fits the simulated tasks well but underfits the long-tail real-world tasks. As shown in Fig. 4(a), π0.5 never lifts the first cup high enough, so it always collides with the second cup and fails on every Stack Cups trial. A similar pattern appears on Grasp Coke, where it arrives at the target pose but never issues the grasp action (Fig. 4(b)). Considering its excellent performance on the simulation suites, we believe π0.5 allocates its capacity to the data-rich simulated tasks while learning the long-tail real-world tasks insufficiently. History compression enlarges the action receptive field and removes hesitation at decision points. Grasp Coke illustrates this effect most clearly. In many failed rollouts of the historyfree variants, the gripper aligns well with the bottle but never commits to the grasp. This failure is closely related to the teleoperated demonstrations: before executing a grasp, the human operator often briefly pauses to align the gripper with the object. A policy with a short temporal window can misinterpret this alignment phase as requiring further correction, causing it to stall near the grasp pose. As shown in Fig. 4(b), the No Action Prior policy reaches the target region at 10 seconds, yet it keeps making small adjustments around the aligned pose until timeout. By compressing past states and actions into a single latent token, the learned encoder lets the VLM infer whether the robot is still approaching, aligning, or ready to grasp. Equipped with this context, the Action-State Prior + History policy reaches the target region at the same time but commits to the grasp within 5 seconds, completing the task decisively. This decisive behavior raises Grasp Coke from 35% to 50%, showing that the encoder effectively uses history.
9
(a) Task 1: Stack Cups
execution timeline
Fail "$.%
lifts too low, collides with cup 2 approach t = 0s
grasp cup 1 t = 10s
lift too low t = 15s
move slowly t = 20s
stuck t = 30s
Fail No Action Prior
gets stuck before alignment approach t = 0s
grasp cup 1 t = 10s
lift & move t = 12s
frozen t = 30s
timeout t = 40s
Success Action-State Prior
precise and fast
approach t = 0s
grasp cup 1 t = 8s
lift high t = 10s
(b) Task 2: Grasp Coke
move & adjust t = 15s
release t = 18s
execution timeline
Fail
"$.%
arrives at the target pose, but never grasps approach t = 0s
approach t = 5s
adjust t = 10s
still... t = 25s
timeout t = 40s
Fail No Action Prior
keeps adjusting and hesitating, never grasps approach t = 0s
reach target region t = 10s
adjust... t = 20s
hesitate... t = 30s
timeout t = 40s
Success
Action-State Prior + History
smooth and precise, without hesitation approach t = 0s
reach target region t = 10s
adjust t = 12s
grasp t = 15s
lift t = 18s
Fig. 4: Qualitative comparison on two long-tail real-world tasks: (a) Stack Cups and (b) Grasp Coke. Rows show representative key frames for each method. Without an action prior, policies move unstably. π0.5 underfits the real-world tail despite strong simulation results. History compression reduces hesitation near the grasp pose.
10
No Action Prior
Action-State Prior
(a) Prediction Loss
8 6 4 2 1 0.5
Action-State Prior + History
(b) Gradient Norm
~7.1
1200
8× lower initial loss
(c) Validation MAE 1.6 1.3 1.0 0.7 0.4 0.15
~1405
800 400
~0.9
120
~89
0.4
16× smaller initial gradients
90
0.3
~700 steps
unstable fluctuations
60
~2.1k steps
MAE = 0.1
0.10
0.2
0.05 30
0.1 0.0
0.00
0 0
0.5k
1k
1.5k
2k
Training Steps
2.5k
3k
0
0.5k
1k
1.5k
2k
Training Steps
2.5k
3k
0
0.5k
1k
1.5k
2k
Training Steps
2.5k
3k
Fig. 5: Training dynamics during the first 3,000 VLA training steps for three variants: No Action Prior, Action-State Prior, and Action-State Prior + History. From left to right: prediction loss, gradient norm, and MAE validation score. The y-axis of each plot is broken to simultaneously show the large initial gap and the fine-grained later behavior. With action prior learning, the initial prediction loss is reduced by nearly 8× (from ∼7.1 to ∼0.9), and the initial gradient norm is reduced by nearly 16× (from ∼1,400 to ∼90), leading to significantly faster and more stable convergence. Training efficiency. This acceleration is especially meaningful because Stage 1 is much cheaper than Stage 2 VLA training. Beyond final success rates and rollout behavior, we further As summarized in Table II, action prior learning uses a smaller examine how action prior learning changes the early optimiza- backbone, processes no image or language tokens, supports an tion process of VLA training. The key question is whether 8× larger batch size, and runs for only 5k steps. On the same Stage 1 action prior learning can reduce the burden of Stage 2 8×H200 GPUs, this stage takes about 2 hours, compared with VLA training, where the model must align visual-language 20 hours for the 50k-step VLA training run. This corresponds features with continuous action generation. Fig. 5 compares to only about 10% additional wall-clock cost relative to the the first 3,000 Stage 2 steps of three variants under the same downstream VLA training stage. Thus, Stage 1 is not another data, hyperparameters, and VLM backbone, isolating the effect expensive VLA training phase; it is a lightweight actionof action prior initialization and history conditioning. only calibration stage that removes a large part of the early Stable initialization. The no-prior model starts from a poorly optimization burden before cross-modal learning begins. calibrated action module and suffers a much larger initial prediction loss. In contrast, prior-based variants begin from a E. Ablation Study decoder that already models feasible motion, reducing the initial We isolate three design choices behind the proposed framework: loss by nearly 8× and reaching low loss much earlier. This Q1. Does incorporating proprioceptive state into the action suggests that Stage 1 handles the low-level action calibration encoder improve the learned action prior? before cross-modal alignment begins. Stage 2 can therefore Q2. Does history integration help, and does the learned action focus less on discovering valid motion patterns from scratch and encoder compress history more effectively than naive MLP more on adapting these patterns to visual-language conditions. projection? Q3. How long should the latent embedding distillation last? Smoother gradients. Random action initialization also produces large and recurrent gradient spikes, while the learned Implementation Details. All ablations are conducted on the 9 decoder keeps gradients small and stable from the beginning. simulation tasks using the same Stage 2 data, VLM backbone, This matters because stable action gradients reduce interference and training configuration as the main comparison (Table II). with the already useful representations in the VLM backbone. Rows 2–5 in Table IV add Stage 1 action prior learning with Faster validation convergence. The MAE curve confirms that different encoder inputs and history settings; history-enabled the gain is not limited to training loss. Prior-based variants variants use a 15-step history horizon and Ndecay = 5,000. D. Analysis of Training Dynamics and Optimization Stability
reach the same validation error much earlier, indicating that the learned action prior also serves as an optimization prior for cross-modal policy learning. The history variant further reaches the lowest MAE after the initial transient, consistent with its stronger downstream performance in Table III.
Effect of Action Encoder Input. Adding proprioceptive state to the action encoder improves the average success rate from 65.4% to 68.3% (Rows 2–3 in Table IV). The gain is most visible on RoboCasa, where the GR1 humanoid has a highdimensional state space: Pl2B improves by 20 points, C2B by
11
TABLE IV: Ablation of action/state prior learning and history integration on 9 simulation tasks. Rows compare whether Stage 1 uses action-only or action-state trajectories, and whether Stage 2 receives encoder-compressed history context. Success rates (%) are reported for LIBERO and RoboCasa, with Avg. computed across all tasks. Red numbers in the Avg. column denote absolute gains over the No Action Prior baseline. Prior Type
#
Action 1 2 3 4 5
✓ ✓ ✓ ✓
VLA Training
State
LIBERO
History
Spatial
Object
Goal
Long
B2C
C2B
Pm2B
Pl2B
T2C
Avg.
✓ ✓
96.4 97.2 97.2 96.6 98.0
98.2 99.0 97.4 97.6 98.0
97.2 96.6 96.4 94.0 96.2
87.2 87.4 89.2 93.2 93.4
60 72 62 70 76
38 36 48 56 56
30 34 40 50 50
30 24 44 18 32
44 42 40 52 44
64.6 65.4 ↑0.8 68.3 ↑3.7 69.7 ↑5.1 71.5 ↑6.9
✓ ✓
TABLE V: Comparison of history integration strategies. Ours uses the learned action encoder for history compression. LIBERO
Method
RoboCasa
97.4 96.4 89.2 62 97.8 95.6 93.2 74 98.0 96.2 93.4 76
48 46 56
40 42 50
44 32 32
LIBERO Avg.
Overall Avg.
Spatial Object Goal Long B2C C2B Pm2B Pl2B T2C No History 97.2 Naive MLP 97.0 Ours 98.0
RoboCasa
Avg.
40 68.3 38 68.4 ↑0.1 44 71.5 ↑3.2
RoboCasa Avg.
12 points, and Pm2B by 6 points. The same trend holds when history is enabled, with Row 5 outperforming Row 4 by 1.8%. This suggests that state information anchors relative action Fig. 6: Effect of the distillation horizon under a fixed 30kdeltas to the robot’s absolute body configuration. Actions alone step VLA budget. Overall and RoboCasa performance peak at describe how the end effector or joints should change, but N decay = 5,000, while LIBERO peaks at 10k but remains the same delta can correspond to different physical motions nearly saturated. Too-short distillation underuses the prior, depending on the current pose. By interleaving state and whereas too-long distillation constrains VLM refinement. action tokens, the encoder observes both the local motion and the kinematic context from which it is executed. This is especially useful for complex embodiments such as GR1, where used for action prior learning and reaches 71.5%. The gap body configuration provides important cues for disambiguating is most visible on RoboCasa, where the encoder improves similar action chunks. over the naive baseline by 10 points on C2B, 8 points on History context improves temporally extended manipulation. Pm2B, and 6 points on T2C. This indicates that the action History integration further improves the action-state prior from encoder does more than reduce dimensionality: it converts raw 68.3% to 71.5% (Rows 3 and 5). The benefit is strongest temporal traces into structured motion embeddings that are on sequential or multi-stage tasks, including LIBERO Long easier to fuse with visual-language features. Overall, the full (+4.2 points) and several RoboCasa tasks such as B2C model (Row 5) improves by 6.9% over the no-prior baseline (+14), Pm2B (+10), and C2B (+8). The action-only prior (Row 1), showing that state conditioning and encoder-based shows a similar trend, with Row 4 outperforming Row 2 history compression are complementary. From an architecture by 4.3%. These improvements support the intuition that a perspective, this insertion strategy also keeps the VLA interface single observation-action pair can be insufficient for temporally compact. Instead of appending long raw history tokens and extended manipulation. Historical state-action context exposes asking the VLM to infer temporal structure from scratch, the whether the policy has just approached, adjusted, or already encoder provides a single motion-aware summary token. This committed to a subgoal, allowing the VLA model to make makes history conditioning easy to reuse across embodiments, because the VLM receives a fixed-size temporal summary rather more stable decisions. Encoder-based history compression is more effective than than embodiment-specific raw trajectories. This is consistent naive projection. Table V verifies that the improvement with the role of the action prior: Stage 1 action-only learning comes from structured history compression rather than merely first organizes motion history into a latent representation, and exposing the VLM to more numerical tokens. Naive MLP Stage 2 only needs to condition that representation on images projection barely improves over no history (68.4% vs. 68.3%), and language. suggesting that raw historical state-action vectors are not easy for the VLM to interpret directly. In contrast, the learned action encoder compresses history into the same latent space
Effect of Distillation Steps. With a fixed 30k-step VLA budget, Ndecay controls how long the alignment and reconstruction losses guide the VLM. Fig. 6 shows a clear trade-off. Without
12
68.8
70 60 50 40 30 20
Stage 1: Action Prior Learning (~2 h)
Average Success Rate (%)
68.3
20k 10k
15k
25k
30k
64.6
+4.2 @ 30k steps
+2.8 vs Action-State Prior @ 15k steps
More Action-State Prior above No Action Prior after ~8 h despite the 2 h offset, and above Action-State Prior at every checkpoint
+14.2 @ 5k steps
No Action Prior Action-State Prior More Action-State Prior
remain unchanged, so any downstream improvement must come from a stronger action prior rather than more VLA supervision. Importantly, the Stage 1 configuration is also kept fixed: we use the same backbone, batch size, learning rate, and 5k training steps as in Table II. We only increase the amount of action data, rather than training the action module for more steps, because the action-only learning stage converges rapidly under large-batch optimization. Specifically, we augment Stage 1 with LIBERO-90 [52] and the full 1,000 demonstrations for each selected RoboCasa task, increasing the Stage 1 action-data mixture from 565k to roughly 2.3M transitions (4×).
The Stage 1 cost is quickly amortized. Fig. 7 includes the full 2-hour Stage 1 cost on the wall-clock axis, so the prior variants 0 2 4 6 8 10 12 14 Total Wall-clock Time (hours, 8×H200) start with an explicit time offset. Even under this accounting, Action-State Prior catches up with the no-prior baseline by Fig. 7: Average success rate versus total wall-clock training around 8 hours and finishes higher. This confirms that the time on 8×H200 GPUs. The prior variants include the 2-hour extra Stage 1 cost is quickly amortized during Stage 2 training. Stage 1 cost as an offset for fair accounting. More Action-State The reason is that Stage 1 processes only compact state-action Prior scales the Stage 1 action data while keeping the action- tokens, without expensive image or language encoders, making prior training configuration and cost unchanged. It consistently the learning stage much cheaper than VLA training. The curve outperforms the standard Action-State Prior throughout VLA therefore reinforces the efficiency contrast discussed above: the training and achieves the best final success rate. This shows that action prior adds a small upfront cost, but shortens the much scaling action prior learning is meaningful: a broader action- more expensive VLA optimization process. only distribution transfers directly into faster and stronger Scaling action-prior data further strengthens downstream downstream VLA training. VLA training. At equal VLA step counts, the action prior consistently improves performance. At 5k steps, Action-State distillation (Ndecay = 0), the model corresponds to the no-prior Prior reaches 32.2%, compared with 21.2% for No Action baseline and reaches 64.6%. Activating distillation for 3k steps Prior. Scaling the Stage 1 data further raises this early result already improves the average to 65.1%, and the best overall to 35.4%, showing that a richer action distribution provides a result appears at 5k steps with 68.3%. The gain is mainly stronger initialization. The advantage persists later in training: driven by RoboCasa, where performance increases from 40.4% the scaled prior reaches 64.4% by 15k steps, nearly matching to 46.8%. LIBERO is less sensitive because its average already the no-prior model’s final 30k-step result, and finishes at 68.8%. stays around 95%, although its best point appears at 10k steps. These results indicate that action prior learning is not only This trend reflects the role of latent distillation in our training efficient, but also scalable: adding action-only data strengthens design. During early Stage 2 training, Lalign anchors the the motion prior and transfers directly to downstream VLA VLM-predicted action embedding to the structured latent space performance. Although our Stage 1 corpus is still modest in learned by the action encoder, providing a stable bridge from scale, this trend suggests a clear path toward training action visual-language features to action generation. However, the priors on broader action-only robot datasets, where more diverse VLM eventually needs to refine representations using visual motion distributions may yield stronger priors that could be observations and language instructions that are absent from transferred to the following VLA training. 5k
the action-only encoder. When distillation is kept for too long, the VLM is forced to stay close to a narrower state-action representation, and full-training distillation drops to 63.1%, below the no-prior baseline. The best strategy is therefore to use the action prior as early guidance and then release the constraint for end-to-end refinement. In this sense, Ndecay = 5,000 provides a practical balance: it gives the VLM enough latent guidance to enter the action space smoothly, while leaving most of Stage 2 for task-conditioned visual-language refinement. F. Scaling Action Prior Data and Training Efficiency Setup. Finally, we examine whether action prior learning benefits from more Stage 1 data and whether the extra Stage 1 cost is justified in wall-clock time. This experiment changes only the data used to train the action prior. The Stage 2 VLA data, VLM backbone, training steps, and all hyperparameters
V. C ONCLUSION We presented a two-stage training framework that introduces a dedicated action prior learning stage before VLA training. By training a flow-matching-based encoder-decoder action module solely on unconditioned action trajectories, we equip the action head with structured knowledge of the action distribution before it encounters visual-language inputs. An early-stage latent alignment distillation strategy further accelerates cross-modal convergence by anchoring the VLM to the learned action embedding space. Experiments across 13 crossembodiment tasks validate that the proposed approach yields faster convergence, higher success rates, and effective history compression at negligible cost. We further demonstrate that enriching the Stage 1 data produces a stronger action prior that directly transfers to improved downstream VLA performance, confirming the scalability of the proposed framework.
13
R EFERENCES [1] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” Advances in Neural Information Processing Systems, 2023. [2] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” CVPR, 2024. [3] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang et al., “Qwen2. 5-vl technical report,” arXiv preprint arXiv:2502.13923, 2025. [4] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [5] Y. Du, M. Yang, B. Dai, H. Dai, O. Nachum, J. B. Tenenbaum, D. Schuurmans, and P. Abbeel, “Learning universal policies via textguided video generation,” Advances in Neural Information Processing Systems, 2023. [6] K. Black, M. Nakamoto, P. Atanasov, H. Walke, C. Finn, A. Kumar, and S. Levine, “Zero-shot robotic manipulation with pretrained image-editing diffusion models,” ICLR, 2024. [7] C.-L. Cheang, G. Chen, Y. Jing, T. Kong, H. Li, Y. Li, Y. Liu, H. Wu, J. Xia, Y. Yan et al., “Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation,” arXiv preprint arXiv:2410.06158, 2024. [8] N. Agarwal, A. Ali, J. Allen, M. Antolini, A. Aubame, A. Azzolini, J. Bai, M. Bala, Y. Balaji, J. Bapst et al., “Cosmos 3: Omnimodal world models for physical ai,” arXiv preprint arXiv:2606.02800, 2026. [9] S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang et al., “World action models are zero-shot policies,” arXiv preprint arXiv:2602.15922, 2026. [10] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu et al., “Rt-1: Robotics transformer for real-world control at scale,” arXiv preprint arXiv:2212.06817, 2022. [11] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “Rt-2: Vision-languageaction models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023. [12] M. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn, “Openvla: An open-source vision-language-action model,” arXiv preprint arXiv:2406.09246, 2024. [13] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, 2022. [14] D. Hoeller, N. Rudin, D. Sako, and M. Hutter, “Anymal parkour: Learning agile navigation for quadrupedal robots,” Science Robotics, vol. 9, 2024. [15] Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn, “Humanplus: Humanoid shadowing and imitation from humans,” CoRL, 2024. [16] J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang et al., “Gr00t n1: An open foundation model for generalist humanoid robots,” arXiv preprint arXiv:2503.14734, 2025. [17] C. Cheang, S. Chen, Z. Cui, Y. Hu, L. Huang, T. Kong, H. Li, Y. Li, Y. Liu, X. Ma et al., “Gr-3 technical report,” arXiv preprint arXiv:2507.15493, 2025. [18] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter et al., “pi 0: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024. [19] Q. Li, Y. Liang, Z. Wang, L. Luo, X. Chen, M. Liao, F. Wei, Y. Deng, S. Xu, Y. Zhang et al., “Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation,” arXiv preprint arXiv:2411.19650, 2024. [20] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain et al., “Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 6892–6903. [21] L. Wang, X. Chen, J. Zhao, and K. He, “Scaling proprioceptive-visual learning with heterogeneous pre-trained transformers,” Advances in Neural Information Processing Systems, 2024. [22] H. Yuan, Z. Liang, A. Chen, Y. Wang, H. Li, P. Lin, Y. Huang, Z. Lei, T. Zhang, J. Zhang et al., “Qwen-robotmanip technical report: Alignment unlocks scale for robotic manipulation foundation models,” arXiv preprint arXiv:2606.17846, 2026.
[23] S. Deng, M. Yan, S. Wei, H. Ma, Y. Yang, J. Chen, Z. Zhang, T. Yang, X. Zhang, W. Zhang, H. Cui, Z. Zhang, and H. Wang, “Graspvla: a grasping foundation model pre-trained on billion-scale synthetic action data,” 2025. [24] W. Zhang, H. Liu, Z. Qi, Y. Wang, X. Yu, J. Zhang, R. Dong, J. He, H. Wang, Z. Zhang, L. Yi, W. Zeng, and X. Jin, “Dreamvla: A visionlanguage-action model dreamed with comprehensive world knowledge,” CoRR, vol. abs/2507.04447, 2025. [25] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li, “Univla: Learning to act anywhere with task-centric latent actions,” arXiv preprint arXiv:2505.06111, 2025. [26] Y. Mu, Q. Zhang, M. Hu, W. Wang, M. Ding, J. Jin, B. Wang, J. Dai, Y. Qiao, and P. Luo, “Embodiedgpt: Vision-language pre-training via embodied chain of thought,” Advances in Neural Information Processing Systems, vol. 36, 2024. [27] Contributors, “Starvla: A lego-like codebase for vision-language-action model developing,” GitHub repository, 1 2025. [28] D. Jing, J. Nie, T. Zhang, J. Liu, H. Yao, Z. Lu, and M. Ding, “Tempovla: Learning speed-controllable vision-language-action policies,” arXiv preprint arXiv:2606.06491, 2026. [29] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge et al., “Qwen3-vl technical report,” arXiv preprint arXiv:2511.21631, 2025. [30] A. Steiner, A. S. Pinto, M. Tschannen, D. Keysers, X. Wang, Y. Bitton, A. Gritsenko, M. Minderer, A. Sherbondy, S. Long et al., “Paligemma 2: A family of versatile vlms for transfer,” arXiv preprint arXiv:2412.03555, 2024. [31] M. J. Kim, C. Finn, and P. Liang, “Fine-tuning vision-language-action models: Optimizing speed and success,” arXiv preprint arXiv:2502.19645, 2025. [32] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” arXiv preprint arXiv:2303.04137, 2023. [33] T. Z. Zhao, V. Kumar, S. Levine, and C. Finn, “Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware,” in Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, July 2023. [34] S. Liu, L. Wu, B. Li, H. Tan, H. Chen, Z. Wang, K. Xu, H. Su, and J. Zhu, “Rdt-1b: a diffusion foundation model for bimanual manipulation,” arXiv preprint arXiv:2410.07864, 2024. [35] K. Pertsch, K. Stachowicz, B. Ichter, D. Driess, S. Nair, Q. Vuong, O. Mees, C. Finn, and S. Levine, “Fast: Efficient action tokenization for vision-language-action models,” arXiv preprint arXiv:2501.09747, 2025. [36] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang et al., “Spatialvla: Exploring spatial representations for visual-language-action model,” arXiv preprint arXiv:2501.15830, 2025. [37] F. Li, W. Song, H. Zhao, J. Wang, P. Ding, D. Wang, L. Zeng, and H. Li, “Spatial forcing: Implicit spatial representation alignment for vision-language-action model,” arXiv preprint arXiv:2510.12276, 2025. [38] W. Tang, J.-H. Pan, Y.-H. Liu, M. Tomizuka, L. E. Li, C.-W. Fu, and M. Ding, “Geomanip: Geometric constraints as general interfaces for robot manipulation,” arXiv preprint arXiv:2501.09783, 2025. [39] Q. Zhao, Y. Lu, M. J. Kim, Z. Fu, Z. Zhang, Y. Wu, Z. Li, Q. Ma, S. Han, C. Finn et al., “Cot-vla: Visual chain-of-thought reasoning for vision-language-action models,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 1702–1713. [40] M. Zawalski, W. Chen, K. Pertsch, O. Mees, C. Finn, and S. Levine, “Robotic control via embodied chain-of-thought reasoning,” arXiv preprint arXiv:2407.08693, 2024. [41] W. Tang, D. Jing, J.-H. Pan, Z. Lu, Y.-H. Liu, L. E. Li, M. Ding, and C.-W. Fu, “Incentivizing multimodal reasoning in large models for direct robot manipulation,” arXiv preprint arXiv:2505.12744, 2025. [42] R. Zheng, Y. Liang, S. Huang, J. Gao, H. Daumé III, A. Kolobov, F. Huang, and J. Yang, “Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies,” arXiv preprint arXiv:2412.10345, 2024. [43] J. Cen, C. Yu, H. Yuan, Y. Jiang, S. Huang, J. Guo, X. Li, Y. Song, H. Luo, F. Wang et al., “Worldvla: Towards autoregressive action world model,” arXiv preprint arXiv:2506.21539, 2025. [44] L. Li, Q. Zhang, Y. Luo, S. Yang, R. Wang, F. Han, M. Yu, Z. Gao, N. Xue, X. Zhu et al., “Causal world modeling for robot control,” arXiv preprint arXiv:2601.21998, 2026. [45] T. Yuan, Z. Dong, Y. Liu, and H. Zhao, “Fast-wam: Do world action models need test-time future imagination?” arXiv preprint arXiv:2603.16666, 2026. [46] M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M.-Y. Liu, C. Finn et al., “Cosmos policy: Fine-tuning video models
14
for visuomotor control and planning,” arXiv preprint arXiv:2601.16163, 2026. [47] H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. HansenEstruch, A. W. He, V. Myers, M. J. Kim, M. Du et al., “Bridgedata v2: A dataset for robot learning at scale,” in Conference on Robot Learning. PMLR, 2023, pp. 1723–1736. [48] A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srinivasan, L. Y. Zhu, O. Lingelbach et al., “Droid: A large-scale in-the-wild robot manipulation dataset,” arXiv preprint arXiv:2403.12945, 2024. [49] Q. Bu, J. Cai, L. Chen, X. Cui, Y. Ding, S. Feng, S. Gao, X. He, X. Hu, X. Huang et al., “Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems,” arXiv preprint arXiv:2503.06669, 2025. [50] K. Wu, C. Zhao, J. Chen, W. Li, Y. Xu, J. Zhu et al., “Robomind: Benchmark on multi-embodiment intelligence normative data for robot manipulation,” arXiv preprint arXiv:2412.13877, 2024. [51] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, W. Deng, Y. Guo, T. Nian, X. Xie, Q. Chen, K. Su, T. Xu, G. Liu, M. Hu, H. ang Gao, K. Wang, Z. Liang, Y. Qin, X. Yang, P. Luo, and Y. Mu, “Robotwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation,” 2025. [52] B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “Libero: Benchmarking knowledge transfer for lifelong robot learning,” arXiv preprint arXiv:2306.03310, 2023. [53] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parber, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu, “Robocasa: Large-scale simulation of everyday tasks for generalist robots,” arXiv preprint arXiv:2406.02523, 2024. [54] O. M. Team, D. Ghosh, H. Walke, K. Pertsch, K. Black, O. Mees, S. Dasari, J. Hejna, T. Kreiman, C. Xu et al., “Octo: An open-source generalist robot policy,” arXiv preprint arXiv:2405.12213, 2024. [55] J. Zheng, J. Li, Z. Wang, D. Liu, X. Kang, Y. Feng, Y. Zheng, J. Zou, Y. Chen, J. Zeng et al., “X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model,” arXiv preprint arXiv:2510.10274, 2025. [56] J. Zheng, J. Li, D. Liu, Y. Zheng, Z. Wang, Z. Ou, Y. Liu, J. Liu, Y.-Q. Zhang, and X. Zhan, “Universal actions for enhanced embodied foundation models,” in Proceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 22 508–22 519. [57] K. Sohn, H. Lee, and X. Yan, “Learning structured output representation using deep conditional generative models,” Advances in neural information processing systems, vol. 28, 2015. [58] K. Xu, Z. Zhu, A. Chen, R. Xiong, and Y. Wang, “Apt: Action expert pretraining improves instruction generalization of vision-language-action policies,” arXiv preprint arXiv:2606.12366, 2026. [59] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin et al., “Latent action pretraining from videos,” arXiv preprint arXiv:2410.11758, 2024. [60] A. Van Den Oord, O. Vinyals et al., “Neural discrete representation learning,” Advances in neural information processing systems, vol. 30, 2017. [61] X. Chen, J. Guo, T. He, C. Zhang, P. Zhang, D. C. Yang, L. Zhao, and J. Bian, “Igor: Image-goal representations are the atomic control units for foundation models in embodied ai,” arXiv preprint arXiv:2411.00785, 2024. [62] Y. Chen, Y. Ge, Y. Li, Y. Ge, M. Ding, Y. Shan, and X. Liu, “Moto: Latent motion token as the bridging language for robot manipulation,” arXiv preprint arXiv:2412.04445, 2024. [63] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang, “Egodex: Learning dexterous manipulation from large-scale egocentric video,” 2025. [Online]. Available: https://arxiv.org/abs/2505.11709 [64] R. Punamiya, S. Kareer, Z. Liu, J. Citron, R.-Z. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu et al., “Egoverse: An egocentric human dataset for robot learning from around the world,” arXiv preprint arXiv:2604.07607, 2026. [65] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu et al., “Ego4d: Around the world in 3,000 hours of egocentric video,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 18 995– 19 012. [66] J. Ma, C. Wang, Y. Li, J. Liu, Y. Li, H. Huang, P. Xu, and H. Zhao, “Vla-jepa: Enhancing vision-language-action model with latent world model,” arXiv preprint arXiv:2602.10098, 2025. [67] D. Wu, Y. Cao, Y. Fu et al., “Latent action pretraining through world modeling,” arXiv preprint arXiv:2509.18428, 2025.
[68] S. Gao, S. Zhou, Y. Du, J. Zhang, and C. Gan, “Adaworld: Learning adaptable world models with latent actions,” in International Conference on Machine Learning (ICML), 2025. [69] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M.-Y. Liu, Y. Zhu, J. Jang, and L. J. Fan, “Dreamdojo: A generalist robot world model from large-scale human videos,” arXiv preprint arXiv:2602.06949, 2026. [70] Z. Cui, H. Pan, A. Iyer, S. Haldar, and L. Pinto, “Dynamo: In-domain dynamics pretraining for visuo,” Motor Control, 2024. [71] D. Jing, G. Wang, J. Liu, W. Tang, Z. Sun, Y. Yao, Z. Wei, Y. Liu, Z. Lu, and M. Ding, “Mixture of horizons in action chunking,” arXiv preprint arXiv:2511.19433, 2025. [72] P. Intelligence, “π0.5 : a vision-language-action model with open-world generalization,” 2025. [Online]. Available: https://arxiv.org/abs/2504. 16054 [73] S. Community, “Starvla: A lego-like codebase for vision-language-action model developing,” arXiv preprint arXiv:2604.05014, 2026. [74] J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in International conference on machine learning. pmlr, 2015, pp. 2256– 2265. [75] J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems, vol. 33, pp. 6840– 6851, 2020. [76] W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 4195–4205. [77] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei et al., “Qwen2. 5 technical report,” arXiv preprint arXiv:2412.15115, 2024. [78] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, W. Ge, Z. Guo, Q. Huang, J. Huang, F. Huang, B. Hui, S. Jiang, Z. Li, M. Li, M. Li, K. Li, Z. Lin, J. Lin, X. Liu, J. Liu, C. Liu, Y. Liu, D. Liu, S. Liu, D. Lu, R. Luo, C. Lv, R. Men, L. Meng, X. Ren, X. Ren, S. Song, Y. Sun, J. Tang, J. Tu, J. Wan, P. Wang, P. Wang, Q. Wang, Y. Wang, T. Xie, Y. Xu, H. Xu, J. Xu, Z. Yang, M. Yang, J. Yang, A. Yang, B. Yu, F. Zhang, H. Zhang, X. Zhang, B. Zheng, H. Zhong, J. Zhou, F. Zhou, J. Zhou, Y. Zhu, and K. Zhu, “Qwen3-vl technical report,” arXiv preprint arXiv:2511.21631, 2025.