ALAM: Algebraically Consistent Latent Transitions for Vision-Language-Action Models
arXiv:2605.10819v1 [cs.RO] 11 May 2026
Zuojin Tang1,2 Haoyun Liu3,4,2 Xinyuan Chang2 Changjie Wu2 Dongjie Huo5,2 Yandan Yang2 Bin Liu6 Zhejia Cai7 Feng Xiong2 Mu Xu2 Jiachen Luo8 De Ma1 Zhiheng Ma4∗ Gang Pan1∗ 1 Zhejiang University 2 Amap, Alibaba Group 3 Nanjing University 4 Shenzhen University of Advanced Technology 5 Beijing University of Chemical Technology 6 Embodied Intelligence General Platform Laboratory, Chery Auto 7 Tsinghua University 8 Queen Mary University of London ∗ Corresponding Author
Abstract Vision-language-action (VLA) models remain constrained by the scarcity of actionlabeled robot data, whereas action-free videos provide abundant evidence of how the physical world changes. Latent action models offer a promising way to extract such priors from videos, but reconstruction-trained latent codes are not necessarily suitable for policy generation: they may predict future observations while lacking the structure needed to be reused or generated coherently with robot actions. We introduce ALAM (Algebraic Latent Action Model), an Algebraically Consistent Latent Action Model that turns temporal relations in action-free video into structural supervision. Given frame triplets, ALAM learns latent transitions that are grounded by reconstruction while being regularized by composition and reversal consistency, encouraging a locally additive transition space. For downstream VLA learning, we freeze the pretrained encoder and use its latent transition sequences as auxiliary generative targets, co-generated with robot actions under a joint flow-matching objective. This couples structured latent transitions with flowbased policy generation, allowing the policy to exploit ALAM’s locally consistent transition geometry without requiring latent-to-action decoding. Representation probes show that ALAM reduces additivity and reversibility errors by 25-85× over unstructured latent-action baselines and improves long-horizon cumulative reconstruction. When transferred to VLA policies, ALAM raises the average success rate from 47.9% to 85.0% on MetaWorld MT50 and from 94.1% to 98.1% on LIBERO, with consistent gains on real-world manipulation tasks. Ablations further confirm that the strongest improvements arise from the synergy between algebraically structured latent transitions and joint flow matching.
1
Introduction
Vision-language-action (VLA) models have emerged as a promising paradigm for robot learning, integrating visual perception, language understanding, and action generation within a unified policy. Recent systems [6, 63, 25, 24, 14, 5, 21] show that large multimodal backbones, when fine-tuned on robot demonstrations, can perform a wide range of language-conditioned manipulation tasks. However, the scalability of this paradigm remains limited by the scarcity and cost of action-labeled robot data. In contrast, action-free videos contain abundant evidence of how the physical world changes: objects are pushed, grasped, opened, stacked, and rearranged across diverse scenes and embodiments. A central challenge is therefore to extract behavior-relevant priors from such videos and transfer them effectively to downstream VLA policies. Preprint.
Re-Ordered Pairs
a
c
a
b
La te nt Ac t 𝑧!"
𝑧"#
𝑧!#
𝑧"!
Spatial Decoder
c
c
Algebraic Latent Action Space
b
a'
𝒛𝒂𝒃 𝒛𝒂 𝒃
b' c'
a
b
c
𝒛𝒄𝒃
𝒛𝒂𝒄
a
c
Reconstruction Loss
Reversibility Loss
Additivity Loss
Other Latent Action Space
（No Algebraic Properties）
Latent Action Quantize
Vector Quantizer
b
b
Algebraic Latent Action Space (Ours)
Quantize Commit Loss
io n
a
b
Spatial-Temporal Encoder
a
Additivity
𝒛𝒄𝒂 = 𝒛𝒃𝒂 + 𝒛𝒄𝒃
✓
𝒛𝒄𝒂 ≠ 𝒛𝒃𝒂 + 𝒛𝒄𝒃
✕
Reversibility
𝒛𝒃𝒂 = −𝒛𝒂𝒃
✓
𝒛𝒃𝒂 ≠ −𝒛𝒂𝒃
✕
Observations Frames
a
b
c t
Figure 1: Left: Pretraining pipeline. From a sampled frame triplet (oa , ob , oc ), a relational encoder maps each pair (oi , oj ) to a continuous latent transition zij , and a decoder reconstructs ôj from oi and zij . Beyond conventional pair-based latent action models, ALAM further regularizes the latent space with additivity and reversal consistency losses. Right: Algebraically structured latent action space. Unlike conventional pair-based latent actions that only capture forward transitions, ALAM organizes transitions into two cycles representing forward and reverse temporal directions, with explicit additivity (zac ≈ zab +zbc ) and sign-reversibility (zba ≈ −zab ). Latent action models provide a natural route toward this goal by inferring transition variables from observation sequences without explicit action labels. Most existing methods [7, 11, 55, 12, 17] learn such variables through reconstruction or prediction objectives, encouraging the latent code to preserve information useful for recovering future observations. While reconstruction is an important grounding signal, it does not by itself guarantee that the learned latent is useful for policy generation. A reconstruction-trained code may capture appearance changes, camera motion, background dynamics, or other predictive factors that help recover nearby frames, while still lacking the structure needed to serve as a reliable auxiliary trajectory for control. In other words, a latent representation can be predictive without being easy for a generative policy to model jointly with actions. This limitation suggests that latent-action pretraining should not be viewed merely as learning a compact code for future-frame reconstruction. Even without action labels, videos provide more than paired examples of before-and-after observations; they also contain temporal relations among changes. A longer visual transition can be compared with the composition of shorter consecutive transitions, and a transition observed in reverse should be consistent with undoing the corresponding forward change. These relations offer a form of self-supervised structural signal that is independent of explicit robot actions. The key idea of our work is to use such temporal relations to shape the latent space, so that the learned transitions are not only predictive of visual change but also more consistent when reused over time. We propose ALAM (Algebraic Latent Action Model), an Algebraically Consistent Latent Action Model that learns structured latent transitions from action-free video. During pretraining, ALAM samples frame triplets from unlabeled videos and constructs both forward and reversed observation pairs. A relational encoder maps each pair to a continuous latent transition, while a decoder reconstructs the target observation from the source observation and the inferred transition. Reconstruction grounds the latent variable in visual change, whereas composition and reversal losses regularize the latent space: consecutive transitions are encouraged to agree with the longer transition they span, and a reversed transition is encouraged to cancel its forward counterpart. In this way, ALAM turns temporal relations in action-free video into structural supervision for learning reusable latent transitions. To transfer ALAM to downstream VLA policies, we freeze the pretrained encoder and use it to extract view-specific latent transition sequences from demonstration videos, temporally aligned with robot actions. Rather than decoding these latents into actions through a separate module, we train the policy to co-generate latent transition trajectories and robot action trajectories under a shared flow-matching objective. This coupling is important: ALAM turns reconstruction-grounded transition codes into locally consistent, approximately additive auxiliary trajectories, and flow matching can exploit this structure by modeling latents and actions along compatible interpolation paths. At inference time, the latent stream is generated jointly with the action stream but is not executed directly. It instead provides an internal auxiliary trajectory that helps structure action generation over the horizon. This allows the downstream policy to exploit the locally consistent transition space learned by ALAM within the same flow-matching process used to generate robot actions. 2
We evaluate ALAM at three levels. At the representation level, algebraic probes show that ALAM substantially reduces additivity and reversibility errors compared with unstructured latent-action baselines, indicating that the proposed constraints reshape the latent transition space rather than merely improving reconstruction. At the prediction level, ALAM improves reconstruction fidelity at the training horizon and degrades more slowly under long-horizon cumulative composition, supporting the claim that local consistency improves temporal reuse. At the policy level, transferring the frozen encoder to VLA training yields large gains on MetaWorld MT50 and consistent improvements on LIBERO and real-world robotic tasks. Ablations further show that algebraic regularization benefits multiple generative frameworks, while the joint flow-matching formulation produces the strongest improvement, confirming the synergistic effect between structured latent transitions and flow-based policy generation. In summary, our contributions are as follows: • We identify the lack of policy-friendly structure in reconstruction-trained latent actions as a key bottleneck for transferring action-free video pretraining to VLA control. • We propose ALAM, a pretraining framework that turns temporal relations in action-free video into structural supervision through composition and reversal consistency. • We introduce a joint flow-matching transfer mechanism that co-generates structured latent transition trajectories and robot actions, allowing the downstream policy to exploit the structure learned during pretraining. • We validate ALAM through representation-level algebraic probes, long-horizon cumulative reconstruction, and downstream VLA policy learning on simulation and real-world manipulation benchmarks.
2
Related Work
2.1
Vision-Language-Action Models
Driven by the rapid progress of Large Language Models [16, 1, 2, 58, 52, 30] and Multimodal Large Language Models [48, 3, 44, 45, 46], together with the emergence of large-scale robot datasets, VisionLanguage-Action (VLA) models have become a dominant paradigm in robot learning. Following the RT series [6, 63] that pioneered fine-tuning MLLMs on robot demonstrations, a line of work [25, 24, 38, 61, 42, 18, 32, 28] further improves scalability, controllability, and embodiment coverage. To better capture the multi-modal nature of robot actions, another line of work [14, 5, 21, 62, 36, 40, 53] replaces deterministic decoding with diffusion or flow matching, sampling actions from noise conditioned on observations, instructions, and proprioceptive states. Since directly mapping observations to actions lacks the explicit reasoning loop of LLM-style systems, recent methods [50, 19, 57, 20, 9, 41, 30, 60, 10, 54, 27, 51, 33] introduce an auxiliary forecasting step, either generating future frames externally or coupling pixel-level prediction with action prediction. However, such methods inherit the limitations of dense pixel reconstruction and are easily dominated by appearance details irrelevant to control. In contrast, our work bypasses pixel-level forecasting and instead learns algebraically structured latent transitions from action-free video, providing a compact, control-relevant prior for downstream policy learning. 2.2
Latent Action Models from Videos
Latent Action Models (LAMs) learn action-like representations directly from observation sequences to reduce the reliance on labeled robot actions. Early work [7, 11, 55] shows that unlabeled videos alone can supervise latent actions via inverse-dynamics or world-model objectives. Building on this, more recent methods couple forecasting with control by asking the policy to predict compact latent futures or latent actions alongside robot actions [12, 17, 8, 4, 15, 22, 56, 26, 23, 43], while a complementary line [13] grounds the latent space through a proprioceptive forward-dynamics module. Most of these methods do not impose explicit algebraic constraints on the latent space. A concurrent and independent preprint, AC-LAM [49], also studies algebraic structure in latent action spaces. While AC-LAM uses scene-wise additivity primarily to improve standalone latent-action labels, ALAM uses algebraically structured transitions as auxiliary generative trajectories within a flow-based VLA policy. 3
Future Sequence :
T
T+1
T+2
Wrist-view
T+H-1
Attention Mask
T+H State
···
Latent action token
“put the yellow and white mug in the microwave and close it.”
Tokenizer
···
Latent action
Gemma
State
Ground Action
+ Noise
Latent Action & Action Expert
Share Attention
xK
Joint Flow Matching
Third-view feature & latent action Wrist-view feature & latent action
···
···
State: T
T · · ·T+H-1 ···
Action token
Siglip
Action token
ALAM (Our Pretrained)
Latent action token
Current Time: T Third-view
Trainable
···
Frozen
K: Add/Denoise Steps
Denoise
H: Action Horizon
Figure 2: Downstream transfer of ALAM. The frozen ALAM encoder turns frame pairs over horizon T :T +H into latent transition tokens, interleaved with action tokens. Conditioned on visual and language context, a shared expert co-generates both streams via K-step joint flow matching with the Gemma backbone; only the action stream is executed on the robot.
3
Method
We introduce ALAM, a two-stage framework that learns latent transitions from action-free video and transfers them to vision-language-action (VLA) policy learning. In pretraining (Sec. 3.2), a relational encoder is trained with future-frame reconstruction together with two algebraic regularizers, composition consistency and reversal consistency, that encourage the latent transitions to be approximately additive and sign-reversible. At transfer (Sec. 3.3), the encoder is frozen and its latent transitions are jointly modeled with robot actions under a shared flow-matching objective on top of an off-the-shelf VLA backbone. 3.1
Structured Latent Transitions
A useful representation of how a scene changes between two observations should behave like a difference: composing two consecutive transitions should match the longer one they span, and reversing a transition in time should flip its sign. For any function f defined on a temporal index, the difference operator ∆(ti , tj ) = f (tj ) − f (ti ) satisfies ∆(ta , tc ) = ∆(ta , tb ) + ∆(tb , tc ) and ∆(tb , ta ) = −∆(ta , tb ). We use these two identities as soft constraints on a learned transition encoder. Given two observations Oi and Oj , we represent the transition from Oi to Oj by zij = E(Oi , Oj ),
(1)
where E(·, ·) is a relational encoder that takes both endpoints as input. For three temporally ordered observations (Oa , Ob , Oc ) with a < b < c, we impose zac ≈ zab + zbc ,
zba ≈ −zab ,
(2)
which we call composition consistency and reversal consistency. Both are imposed as soft regularizers, and we do not require any further structure on the latent space. See Fig. 1 for the pretraining pipeline. 3.2
Pretraining on Action-Free Video
The pretraining stage trains a latent transition encoder on unlabeled video so that its outputs are predictive of future frames and approximately respect the two consistency relations of Sec. 3.1. Concretely, given an unlabeled video corpus D, we draw a temporally ordered triplet and assemble four transition pairs that span both temporal directions and two horizons: (Oa , Ob , Oc ) ∼ D, a < b < c,
T = {(a, b), (b, c), (a, c)} ∪ {(b, a)}, | {z } | {z } Tfwd
(3)
Trev
where (a, b) and (b, c) are the two short-horizon legs, (a, c) provides the long-horizon target for composition consistency, and (b, a) provides the time-reversed counterpart for reversal consistency. 4
Each pair (Oi , Oj ) ∈ T is encoded into a latent transition by a relational encoder. A patch tokenizer fpatch first turns the two frames into patch sequences Xipatch , Xjpatch , which we concatenate with K learnable query tokens Q ∈ RK×d along the sequence axis (denoted [· ∥ ·]). A spatiotemporal Transformer fθ processes the joint sequence; we read out the query positions, (·)query , and project them through a linear head gϕ : query zij = gϕ fθ [ Q ∥ fpatch (Oi , Oj ) ] , (4) b c c a yielding the four latents za , zb , za , zb . Each latent is then passed through a learned codebook j q(·), with zq,i = q(zij ) and the standard commitment loss Lvq =
2
j zij − sg[zq,i ] 2,
(5)
where sg[·] is the stop-gradient operator and the codebook entries are updated by EMA [47]. For each forward pair (i, j) ∈ Tfwd , two linear heads fα , fβ , a cross-attention block hψ , and a pixel decoder uω produce the prediction j Ô i,j = uω hψ [ fα (zq,i ), fβ (Xipatch ) ] , (6) j i.e., the decoder reconstructs Oj from zq,i together with the source patches. Conditioning on Oi in this way encourages the latent to carry only what is needed to map Oi to Oj , rather than the full content of Oj .
The two algebraic regularizers are imposed on the continuous latents prior to quantization, so that gradients reach the encoder unimpeded by the codebook lookup: Ladd =
2
zac − (zab + zbc ) 2 ,
Lrev =
2
zab + zba 2 ,
(7)
which are squared relaxations of the two identities in Eq. (2). Pixel and perceptual reconstruction terms are averaged over Tfwd , X X 2 i,j 1 1 Ô − O , L = (8) Lrec = |Tfwd dperc Ô i,j , Oj , j perc | |Tfwd | 2 (i,j)∈Tfwd
(i,j)∈Tfwd
with dperc the LPIPS distance [59]. Combining the five terms gives the total pretraining objective, LALAM = λvq Lvq + λrec Lrec + λperc Lperc + λadd Ladd + λrev Lrev .
(9)
At transfer time, we discard the quantizer and the decoder, and freeze the encoder E = gϕ ◦fθ as a fixed transition extractor on robot trajectories (Sec. 3.3). 3.3
Transfer via Joint Flow Matching
At transfer time, the goal shifts from reconstructing video frames to generating robot action sequences. We reuse the pretrained latent transitions as auxiliary generation targets in a flow-matching policy, without any fine-tuning of the encoder. As shown in Fig. 2, we apply the frozen ALAM encoder to a m clip of H+1 consecutive frames V m = (I1m , . . . , IH+1 ) for each view m ∈ {th, wr}, where th and wr denote the third-person and wrist views. Applying the encoder to every adjacent frame pair gives H continuous latent transitions per view, time-aligned with the H ground-truth actions u1:H , m ztm = FALAM (Itm , It+1 ),
t = 1, . . . , H.
(10)
We take ztm to be the continuous output of the encoder, before the codebook lookup. The codebook in Sec. 3.2 is used only as a pretraining-time bottleneck; for joint flow matching we prefer a continuous, differentiable target with higher fidelity. Each ztm is then linearly projected to the dimension of ut , and we co-generate the two latent streams together with the action stream under a shared flow-matching objective. To let the latent transitions condition the action stream, we order the three modalities at every timestep so that the two latent tokens come first and the action token last; the full sequence concatenates these per-timestep blocks, th wr th wr z1 , z1 , u1 , . . . , zH , zH , uH . (11) 5
Table 1: Per-difficulty success rates on MetaWorld MT50. Avg. is the macro-average over the four tiers. AR: autoregressive, FM: flow matching, VA: video-augmented, LA: latent action. To our knowledge, no prior latent-action VLA reports MT50 results under this protocol. Method
Type
Size
Easy
Medium
Hard
Very Hard
Avg.
Diffusion Policy
DP
0.3B
23.1
10.7
1.9
6.1
10.5
TinyVLA-H RT-2∗ RoboTron Mani
AR AR AR
1.3B 7B 4B
77.6 75.5 85.5
21.5 35.3 67.7
11.4 30.7 76.7
15.8 15.2 81.0
31.6 39.2 77.7
GR-1∗ PAD Evo-1
VA VA VA
0.2B – 0.8B
76.6 81.8 89.2
35.3 65.1 76.8
46.0 56.7 77.2
44.0 87.2 79.2
50.5 72.7 80.6
π0.5 π0 SmolVLA
FM FM FM
3B 3B 2B
71.1 71.8 87.1
20.9 48.2 51.8
24.0 41.7 70.0
10.0 30.0 64.0
31.5 47.9 66.9
π0 + ALAM (ours)
LA
3B
89.3
83.6
85.0
82.0
85.0
∗
GR-1 and RT-2∗ scores are reproduced from PAD [39].
th wr Let y m denote the clean sequence of modality m ∈ {u, th, wr}, with y th = z1:H , y wr = z1:H , and u y = u1:H . All three modalities share the same linear data-to-noise interpolation, m m xm τ = τ ϵ + (1 − τ ) y ,
ϵm ∼ N (0, I),
m vtgt = ϵm − y m ,
(12)
sampled at a common time τ = 0.999 ξ + 0.001 with ξ ∼ Beta(1.5, 1) [5]. We further apply a structured attention mask over the interleaved sequence: at each timestep, the two view-specific latent tokens attend to each other but are isolated from action tokens, while each action token attends causally to all preceding tokens. Conditioned on context c (visual observations, language instructions, and proprioceptive states), the model predicts a per-modality velocity field v̂θm (xm τ | τ, c), trained with the per-modality flow-matching loss m Lm = E v̂θm (xm m ∈ {u, th, wr} (13) τ | τ, c) − vtgt 1 , and the total transfer objective is the weighted sum Ltransfer = λth Lth + λwr Lwr + λu Lu .
(14)
This design connects the two stages without retraining: the latent transitions used downstream come directly from the frozen encoder, and the policy is asked to generate them jointly with the actions, along the same interpolation schedule. The action stream is therefore not predicted in isolation, but co-denoised with a structured visual signal already shaped during pretraining. At inference time, we initialize all modalities from Gaussian noise, xm 1 ∼ N (0, I) for m ∈ {u, th, wr}, and jointly integrate from τ =1 to τ =0 with the learned velocity fields, m m m xm τ −∆τ = xτ − ∆τ v̂θ (xτ | τ, c),
(15)
following the same interleaved token order and attention mask as in training. Both streams are denoised jointly, but only u is executed on the robot.
4
Experimental Setup
Pretraining. ALAM is pretrained on 11 action-free video sources, mostly real-robot manipulation data from Open-X-Embodiment [37] and CALVIN [35]. We use AdamW on 128×H20 GPUs (90G) and run for 39 epochs (about 4 days), at which point the pretraining loss has plateaued. The full dataset configuration and hyperparameters are in Appendix A. Downstream control. The transfer stage is built on π0 [5]. The frozen ALAM encoder takes H+1 third-person and wrist frames and produces H latent transitions, which are linearly projected and interleaved with the action stream (Sec. 3.3). Policies are finetuned with flow matching on PaliGemma2B [45] and Gemma-300M [45] on 8×H20 GPUs. The full configuration is in Appendix A.2. 6
Table 2: Success rates on LIBERO. Avg. is the mean over four tasks. AR: autoregressive, FM: flow matching, VA: video-augmented, LA: latent action. For latent-action methods we annotate the underlying backbone in parentheses, so improvements over the corresponding backbone (gray row) directly reflect the contribution of the latent-action design. Type
Size
Spatial
Object
Goal
Long
Avg.
OpenVLA SpatialVLA CoT-VLA
AR AR AR
7B 4B 7B
84.7 88.2 87.5
88.4 89.9 91.6
79.2 78.6 87.6
53.7 55.5 69.0
76.5 78.1 81.1
WorldVLA DreamVLA
VA VA
7B 0.4B
87.6 97.5
96.2 94.0
83.4 89.5
60.0 89.5
81.8 92.6
SmolVLA GR00T-N1 π0 π0.5
FM FM FM FM
2B 2B 3B 3B
93.0 94.4 96.8 98.8
94.0 97.6 98.8 98.2
91.0 93.0 95.8 98.0
77.0 90.6 85.2 92.4
88.8 93.9 94.1 96.9
LAPA UniVLA JALA
LA LA LA
7B 9B 3B
87.4 96.5 96.0
91.2 96.8 98.2
90.0 95.6 97.4
65.4 92.0 96.0 80
83.5 95.2 96.9
π0 + ALAM (ours)
LA
3B
99.2
99.6
94.4
98.1
60
65 55
50
55 45
40
30
20 0 Insert Cylinder ω0
80
80 80 65
55 60 65 55 40 40 20
60 40 20
100
Success Rate (%)
75
70
100
80
99.0 60
65
55
70
50
0 0 0 Insert Cylinder Insert Cube Fold Stack Fold Cup TowelFold Towel Insert Cylinder Insert Cube Cup Stack Insert Cylinder Insert Stack Cube Cup Towel Insert Cube ω0.5
Stack Cup
Fold Towel
ω0
ω0
ω0.5ω0
ω0.5
ω0 + ALAM ω0ω0.5 + ALAM ω0 + ALAM
ω0 + ALAM
Figure 3: Real-world results on a Piper 6-DoF manipulator. ALAM is compared with π0 and π0.5 on four tasks: insert cylinder, insert cube, stack cup, and fold towel. Evaluation benchmarks. We use three settings. (i) MetaWorld MT50 [34]: a single policy is trained on all 50 tabletop tasks. Following Evo-1 [29], we report per-task success rates averaged over 10 trials per task and group tasks into Easy, Medium, Hard, and Very Hard tiers. All methods are trained for 30k steps. (ii) LIBERO [31]: four suites (Spatial, Object, Goal, Long), with the Long suite focusing on long-horizon and cross-scene generalization. We report success rates over 500 evaluation episodes per suite, again with 30k training steps. Training and inference details are in Appendix 9. (iii) Real-world: a Piper 6-DoF manipulator evaluated on four tasks (insert cylinder, insert cube, stack cup, fold towel); task descriptions and evaluation protocols are in Appendix D. Baselines. We use π0 [5] without latent actions or joint generation as the backbone reference, and compare against four families of VLA models: autoregressive (RT-2 [63], OpenVLA [25]), flow-matching (π0.5 [21]), video-augmented (Evo-1 [29], PAD [19], GR-1 [50], DreamVLA [60], WorldVLA [9]), and latent-action (UniVLA [8], JALA [33], LAPA [55]). We also include LAM [7] as an unstructured latent-action baseline at the same encoder capacity: it is trained on image pairs without any algebraic constraint, while ALAM uses consecutive triplets with the reversal (Lrev ) and additivity (Ladd ) losses. Per-benchmark protocols are in Appendix B. 4.1
75
55
Evaluating the algebraic structure of latent actions
We split the pretraining video corpus into 95%/5% train/test partitions, train ALAM and the LAM baseline on the 95% split, and evaluate the frozen encoder E and decoder D on the held-out 5%. All probes use anchor and target frames at horizons t ∈ {k, 2k, 3k, 4k, 5k} with stride k=5 frames; training only supervises t ∈ {k, 2k}, so t ∈ {3k, 4k, 5k} are not seen during pretraining. We write 7
65 55
45 40 40 30 80 80 75 75 75 80 70 70 70 65 20 65 65 65 65 55 55 55 55 55 55 55 55 60 50 50 50 45 45 45 0 40 40 40 Insert Cylinder Insert Cube Stack Cup Fold Towel 40 30 30 30 ω0 ω0.5 ω0 + ALAM 20
100
Success Rate (%)
Success Rate (%)
80 80
100
Success Rate (%)
100
Success Rate (%)
Method
Figure 4: LAM vs. ALAM on the 5% held-out split. (A) Algebraic probes (log y, shaded t≥3k not seen during training): (A.1) additivity error and (A.2) reversibility error, with LAM/ALAM ratios at t=5k shown in red. (B) Per-horizon ∆ from each model’s own k-step score, for direct (top, (a)–(c)) and cumulative (bottom, (d)–(f)) reconstruction. See Sec. 4.1 for definitions and k-step scores. Pt−1 (i+1)k zab = E(oa , ob ) for the latent transition between two frames, and Zt = i=0 zik for the sum of t consecutive short-horizon latents. We look at three things: how well each model reconstructs the target frame at long horizons, how closely the latent space respects the algebraic identities, and whether the same additivity carries over to videos from a different domain. Reconstruction quality across horizons. For each anchor and horizon we report PSNR, SSIM, and LPIPS between the ground-truth target otk and two reconstructions: the direct one, D(oa , zatk ) → ôtk , and the cumulative one, D(oa , Zt ) → ôtk . At the training horizon t=k, ALAM and LAM reach PSNR 28.87 vs. 26.74 dB, SSIM 0.91 vs. 0.90, and LPIPS 0.08 vs. 0.08. Fig. 4(B) plots the per-horizon ∆ relative to each model’s own k-step score. Under both regimes ALAM keeps a smaller ∆ than LAM at every t>k, and the PSNR gap at t=5k reaches +2.89 dB (direct) and +1.58 dB (cumulative). Algebraic consistency on unseen horizons. On the same latents we compute two consistency probes, a the additivity error Add(t) = E ∥zatk − Zt ∥ and the reversibility error Rev(t) = E ∥zatk + ztk ∥. Both are zero for an ideally additive encoder, and Add(1) is zero by construction. Fig. 4(A) shows that LAM’s errors grow with horizon, while ALAM stays one to two orders of magnitude lower across the whole range. The largest LAM/ALAM ratio is about 85× on reversibility at t=k, and at t=5k the ratio is 46× for additivity and 25× for reversibility. The same gap is preserved on the horizons t ∈ {3k, 4k, 5k} that were never used during training. Cross-domain additivity of latent actions. We also look at this additivity qualitatively, on samples from the pretraining mixture and on game videos that the encoder never saw during training. Given an anchor oa and two endpoints ob , oc , decoding zab + zbc gives a frame that looks similar to the direct long-horizon reconstruction D(oa , zac ), on both the test backgrounds and the unseen game footage. Additional examples are in Appendix E. 4.2
Downstream Policy Learning
Main results. We now turn to downstream control and evaluate whether the pretrained structured transitions translate into better policies. On MetaWorld MT50, adding ALAM latent transitions to the π0 backbone raises the average success rate from 47.9% to 85.0%, a +37.1-point absolute gain, and also exceeds the strongest video-augmented baseline Evo-1 [29] by 5.6 points on the same protocol. On LIBERO, ALAM lifts π0 from 94.1% to 98.1% averaged across the four suites, with the largest relative gain on the long-horizon LIBERO-Long split, the suite most sensitive to compositional reasoning. On the real-world Piper robot (Fig. 3), ALAM improves over π0 and π0.5 [21] on all four tasks, with the largest gain reaching 45 percentage points over π0 on the Stack Cup task. The three settings together indicate that the pretrained transitions transfer from action-free video to physical manipulation, rather than fitting a particular simulator. 8
Table 3: Generative framework ablation Table 4: Loss ablation on MetaWorld MT50. on MetaWorld MT50. LAM/ALAM: unstruc- Lrev and Ladd are added to the reconstruction loss tured/structured latents. J.: ✓ joint, ✗ two-stage. to form the full ALAM objective. All variants use DP: diffusion, FM: flow matching. the epoch-4 checkpoint for efficiency. Method
J. Gen. Easy Med. Hard V.H. Avg.
Variant
Easy Med. Hard V.H. Avg.
π0
– FM 71.8 48.2 41.7 30.0 47.9
π0
71.8
48.2
41.7 30.0 47.9
+ LAM + ALAM
✓ DP 70.0 36.6 20.0 26.0 38.2 ✓ DP 83.2 63.6 41.7 74.0 65.6
LAM
80.0
35.5
36.7 70.0 55.4
w/o Lrev +Ladd w/o Lrev w/o Ladd ALAM (ours)
77.6 83.9 82.9 86.1
47.3 48.2 51.8 78.2
38.3 48.3 50.0 71.7
+ LAM ✓ FM 82.5 44.6 55.0 78.0 65.0 + ALAM ✗ FM 83.2 53.3 56.3 82.0 68.7 + ALAM (ours) ✓ FM 89.3 83.6 85.0 82.0 85.0
70.0 74.0 84.0 76.0
58.3 63.6 67.2 78.0
ALAM and flow matching combine well. Tab. 3 varies the latent structure (LAM vs. ALAM) and the policy coupling (two-stage vs. joint), under both diffusion and flow matching, on MetaWorld MT50. Three patterns stand out. (i) Adding LAM with joint diffusion reaches 38.2%, below π0 alone (47.9%), so joint coupling does not help when the latent stream is unstructured. (ii) Replacing LAM with ALAM at the same coupling raises the average from 38.2% to 65.6% under diffusion and from 65.0% to 85.0% under flow matching, the largest single change in either column. (iii) With ALAM fixed under flow matching, switching from two-stage to joint coupling raises the average from 68.7% to 85.0% (+16.3 pts), and the joint flow-matching variant is also the best column overall. The pattern is consistent with the additivity and reversibility errors reported in Sec. 4.1. Ablation on the algebraic losses. A natural confound when comparing ALAM with LAM is that ALAM additionally consumes triplet inputs at pretraining, so its gain over LAM could in principle reflect added input capacity rather than the algebraic constraints. To control for this, we train five encoder variants on the same single-domain pretraining subset and transfer each to π0 with joint flow matching under matched downstream conditions (Tab. 4; epoch-4 checkpoints). Removing both algebraic losses (w/o Lrev +Ladd ) reaches 58.3%, only +2.9 pts over the LAM baseline (55.4%) and well below the full ALAM at 78.0% (+22.6 pts over LAM). Each loss on its own recovers most of this gap (w/o Lrev : 63.6%; w/o Ladd : 67.2%), and using both gives the highest score. Within this controlled comparison, the structured latent transitions, rather than the additional inputs, account for most of the downstream gain.
5
Conclusion and Discussion
We presented ALAM, a pretraining framework that learns structured latent transitions from action-free video by regularizing a relational encoder with two algebraic constraints, composition consistency and reversal consistency, motivated by elementary identities of difference operators. The resulting latents are trained to be approximately additive and sign-reversible, and transfer best when the latent is cogenerated with the robot action under a shared flow-matching objective rather than decoded through a separate module. On a held-out video split, ALAM reduces the additivity and reversibility errors by up to 25–85× relative to an unstructured latent-action baseline, and the cumulative reconstruction quality degrades more slowly with horizon. Transferring ALAM into a π0 backbone matches or exceeds the best previously reported numbers we are aware of on MetaWorld MT50, with positive transfer on LIBERO and on a four-task real-world Piper setup. Composition and reversal consistency each improve over an unstructured LAM under both diffusion and flow matching, and the joint flow-matching variant gives the largest improvement in our ablations. Limitations. ALAM provides a local and approximate algebraic regularizer rather than an exact group-structured latent space. Although the learned transitions reduce composition and reversal residuals and generalize to held-out temporal spans, the triplet-based constraints do not guarantee exact consistency or extrapolation to arbitrary horizons. In addition, joint flow matching introduces auxiliary latent tokens at inference time, increasing computation relative to action-only generation. However, these tokens are not decoded into future frames and are not executed by the robot, so the additional cost is confined to the denoising network. 9
References [1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. [2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [4] Hongzhe Bi, Hengkai Tan, Shenghao Xie, Zeyuan Wang, Shuhe Huang, Haitian Liu, Ruowen Zhao, Yao Feng, Chendong Xiang, Yinze Rong, et al. Motus: A unified latent action world model. arXiv preprint arXiv:2512.13030, 2025. [5] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. π0 : A vision-language-action flow model for general robot control, 2026. [6] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022. [7] Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024. [8] Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025. [9] Jun Cen, Chaohui Yu, Hangjie Yuan, Yuming Jiang, Siteng Huang, Jiayan Guo, Xin Li, Yibing Song, Hao Luo, Fan Wang, et al. Worldvla: Towards autoregressive action world model. arXiv preprint arXiv:2506.21539, 2025. [10] Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, et al. Gr-2: A generative video-language-action model with web-scale knowledge for robot manipulation. arXiv preprint arXiv:2410.06158, 2024. [11] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. Lapo: Latent-variable advantage-weighted policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:36902–36913, 2022. [12] Xiaoyu Chen, Junliang Guo, Tianyu He, Chuheng Zhang, Pushi Zhang, Derek Cathera Yang, Li Zhao, and Jiang Bian. Igor: Image-goal representations are the atomic control units for foundation models in embodied ai. arXiv preprint arXiv:2411.00785, 2024. [13] Xiaoyu Chen, Hangxing Wei, Pushi Zhang, Chuheng Zhang, Kaixin Wang, Yanjiang Guo, Rushuai Yang, Yucen Wang, Xinquan Xiao, Li Zhao, et al. Villa-x: enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025. [14] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025. [15] Zichen J Cui, Hengkai Pan, Aadhithya Iyer, Siddhant Haldar, and Lerrel Pinto. Dynamo: In-domain dynamics pretraining for visuo-motor control. Advances in Neural Information Processing Systems, 37:33933–33961, 2024. 10
[16] Luciano Floridi and Massimo Chiriatti. Gpt-3: Its nature, scope, limits, and consequences. Minds and machines, 30(4):681–694, 2020. [17] Shenyuan Gao, Siyuan Zhou, Yilun Du, Jun Zhang, and Chuang Gan. Adaworld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025. [18] Ankit Goyal, Hugo Hadfield, Xuning Yang, Valts Blukis, and Fabio Ramos. Vla-0: Building state-of-the-art vlas with zero modification. arXiv preprint arXiv:2510.13054, 2025. [19] Yanjiang Guo, Yucheng Hu, Jianke Zhang, Yen-Jen Wang, Xiaoyu Chen, Chaochao Lu, and Jianyu Chen. Prediction with action: Visual policy learning via joint denoising process. Advances in Neural Information Processing Systems, 37:112386–112410, 2024. [20] Yucheng Hu, Yanjiang Guo, Pengchao Wang, Xiaoyu Chen, Yen-Jen Wang, Jianke Zhang, Koushil Sreenath, Chaochao Lu, and Jianyu Chen. Video prediction policy: A generalist robot policy with predictive visual representations. arXiv preprint arXiv:2412.14803, 2024. [21] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5 : a visionlanguage-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025. [22] Joel Jang, Seonghyeon Ye, Zongyu Lin, Jiannan Xiang, Johan Bjorck, Yu Fang, Fengyuan Hu, Spencer Huang, Kaushil Kundalia, Yen-Chen Lin, et al. Dreamgen: Unlocking generalization in robot learning through video world models. arXiv preprint arXiv:2505.12705, 2025. [23] Hanjung Kim, Lerrel Pinto, and Seon Joo Kim. Hierarchical latent action model. arXiv preprint arXiv:2603.05815, 2026. [24] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. arXiv preprint arXiv:2502.19645, 2025. [25] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246, 2024. [26] Seungjae Lee, Yibin Wang, Haritheja Etukuru, H Jin Kim, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto. Behavior generation with latent actions. arXiv preprint arXiv:2403.03181, 2024. [27] Lin Li, Qihang Zhang, Yiming Luo, Shuai Yang, Ruilin Wang, Fei Han, Mingrui Yu, Zelin Gao, Nan Xue, Xing Zhu, et al. Causal world modeling for robot control. arXiv preprint arXiv:2601.21998, 2026. [28] Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, et al. Cogact: A foundational vision-language-action model for synergizing cognition and action in robotic manipulation. arXiv preprint arXiv:2411.19650, 2024. [29] Tao Lin, Yilei Zhong, Yuxin Du, Jingjing Zhang, Jiting Liu, Yinxinyu Chen, Encheng Gu, Ziyan Liu, Hongyi Cai, Yanwen Zou, et al. Evo-1: Lightweight vision-language-action model with preserved semantic alignment. arXiv preprint arXiv:2511.04555, 2025. [30] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [31] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023. [32] Haoyun Liu, Jianzhuang Zhao, Xinyuan Chang, Tianle Shi, Chuanzhang Meng, Jiayuan Tan, Feng Xiong, Tong Lin, Dongjie Huo, Mu Xu, SongLin Dong, Zhiheng Ma, Yihong Gong, and Sheng Zhong. Neural implicit action fields: From discrete waypoints to continuous functions for vision-language-action models, 2026. 11
[33] Hao Luo, Ye Wang, Wanpeng Zhang, Haoqi Yuan, Yicheng Feng, Haiweng Xu, Sipeng Zheng, and Zongqing Lu. Joint-aligned latent action: Towards scalable vla pretraining in the wild, 2026. [34] Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, K.R. Zentner, Ryan Julian, J K Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Meta-world+: An improved, standardized, RL benchmark. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. [35] Oier Mees, Lukas Hermann, Erick Rosete-Beas, and Wolfram Burgard. Calvin: A benchmark for language-conditioned policy learning for long-horizon robot manipulation tasks. IEEE Robotics and Automation Letters, 7(3):7327–7334, 2022. [36] Chaojun Ni, Cheng Chen, Xiaofeng Wang, Zheng Zhu, Wenzhao Zheng, Boyuan Wang, Tianrun Chen, Guosheng Zhao, Haoyun Li, Zhehao Dong, et al. Swiftvla: Unlocking spatiotemporal dynamics for lightweight vla models at minimal overhead. arXiv preprint arXiv:2512.00903, 2025. [37] Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open xembodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024. [38] Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. Spatialvla: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025. [39] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning, pages 1332–1344. PMLR, 2023. [40] Mustafa Shukor, Dana Aubakirova, Francesco Capuano, Pepijn Kooijmans, Steven Palma, Adil Zouitine, Michel Aractingi, Caroline Pascal, Martino Russi, Andres Marafioti, et al. Smolvla: A vision-language-action model for affordable and efficient robotics. arXiv preprint arXiv:2506.01844, 2025. [41] Jingwen Sun, Wenyao Zhang, Zekun Qi, Shaojie Ren, Zezhi Liu, Hanxin Zhu, Guangzhong Sun, Xin Jin, and Zhibo Chen. Vla-jepa: Enhancing vision-language-action model with latent world model. arXiv preprint arXiv:2602.10098, 2026. [42] Zuojin Tang, Bin Hu, Chenyang Zhao, De Ma, Gang Pan, and Bin Liu. Vlascd: A visual language action model for simultaneous chatting and decision making. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 9223–9243, 2025. [43] Zuojin Tang, Shengchao Yuan, Xiaoxin Bai, Zhiyuan Jin, De Ma, Gang Pan, and Bin Liu. One token per frame: Reconsidering visual bandwidth in world models for vla policy, 2026. [44] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [45] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024. [46] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. 12
[47] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. [48] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. [49] Hangxing Wei, Xiaoyu Chen, Chuheng Zhang, Tim Pearce, Jianyu Chen, Alex Lamb, Li Zhao, and Jiang Bian. Learning additively compositional latent actions for embodied ai. arXiv preprint arXiv:2604.03340, 2026. [50] Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong. Unleashing large-scale video generative pre-training for visual robot manipulation. arXiv preprint arXiv:2312.13139, 2023. [51] Junjin Xiao, Yandan Yang, Xinyuan Chang, Ronghan Chen, Feng Xiong, Mu Xu, Wei-Shi Zheng, and Qing Zhang. World-env: Leveraging world model as a virtual environment for vla post-training. arXiv preprint arXiv:2509.24948, 2025. [52] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [53] Yandan Yang, Shuang Zeng, Tong Lin, Xinyuan Chang, Dekang Qi, Junjin Xiao, Haoyun Liu, Ronghan Chen, Yuzhi Chen, Dongjie Huo, et al. Abot-m0: Vla foundation model for robotic manipulation with action manifold learning. arXiv preprint arXiv:2602.11236, 2026. [54] Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Sihyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026. [55] Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. [56] Chuheng Zhang, Tim Pearce, Pushi Zhang, Kaixin Wang, Xiaoyu Chen, Wei Shen, Li Zhao, and Jiang Bian. What do latent action models actually learn? arXiv preprint arXiv:2506.15691, 2025. [57] Jianke Zhang, Yanjiang Guo, Yucheng Hu, Xiaoyu Chen, Xiang Zhu, and Jianyu Chen. Up-vla: A unified understanding and prediction model for embodied agent. arXiv preprint arXiv:2501.18867, 2025. [58] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. [59] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018. [60] Wenyao Zhang, Hongsi Liu, Zekun Qi, Yunnan Wang, Xinqiang Yu, Jiazhao Zhang, Runpei Dong, Jiawei He, Fan Lu, He Wang, et al. Dreamvla: a vision-language-action model dreamed with comprehensive world knowledge. arXiv preprint arXiv:2507.04447, 2025. [61] Tony Z Zhao, Vikash Kumar, Sergey Levine, and Chelsea Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023. [62] Jinliang Zheng, Jianxiong Li, Zhihao Wang, Dongxiu Liu, Xirui Kang, Yuchun Feng, Yinan Zheng, Jiayin Zou, Yilun Chen, Jia Zeng, et al. X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. arXiv preprint arXiv:2510.10274, 2025. [63] Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. 13
A
Implementation Details
A.1
ALAM pretraining
The relational encoder E maps a frame pair (Oi , Oj ) to a latent transition zij ∈ R128 . Frames are tokenized with a 14×14 patch embedding, and the patch sequence is prepended with 256 learnable latent queries and processed by a 12-layer ViT. The query outputs are linearly projected to the latent transition space. During pretraining, zij passes through a small VQ bottleneck and is fed to a symmetric 12-layer decoder that reconstructs Ôj from Oi and the quantized latent. Architectural hyperparameters are listed in Tab. 5. Table 5: ALAM pretraining architecture. Component
Value
Input resolution Patch size Hidden size / heads Encoder / decoder depth Latent queries Latent dimension VQ codebook size
200×200×3 (RGB) 14 768 / 12 12 / 12 256 128 7
The pretraining loss combines pixel reconstruction, an LPIPS perceptual term [59], the two algebraic regularizers (composition and reversal consistency), and the standard VQ commitment loss. All terms are weighted equally at 1.0 (Tab. 6). For each video, we sample a temporally ordered triplet (Oa , Ob , Oc ) with a < b < c, where the gaps b − a and c − b are drawn uniformly from {1, . . . , 16} at the effective frame rate (frame-skip 5 by default, 10 for CALVIN). The four pairs {(a, b), (b, c), (a, c), (b, a)} are encoded independently and supervised as in Sec. 3.2. We optimize with AdamW (lr 10−4 , weight decay 10−4 , β1 =0.9, β2 =0.95) and per-GPU batch size 32 for 39 epochs, on 128×H20 (90 GB) GPUs (about 4–5 days). Table 6: Pretraining loss weights. Term
Weight
λrec λperc λadd λrev λcommit
1.0 1.0 1.0 1.0 1.0
Role Pixel reconstruction of Ôj LPIPS perceptual similarity Composition consistency Reversal consistency VQ commitment
The 11 video sources from Sec. 4 are mixed with the sampling weights in Tab. 7, and clips are drawn with probability proportional to their weight. Frames are decoded at the corresponding stride and resized to 200×200. No action, language, or proprioceptive labels are used at this stage. Table 7: Pretraining data mixture and sampling weights. Stride is the frame-skip used for decoding. Dataset fractal20220817 (RT-1) BridgeData-V2 TACO-Play JaCo-Play Berkeley Cable Routing RoboTurk NYU Door-Opening VIOLA Berkeley AutoLab UR5 TOTO CALVIN (task_ABC_D)
14
Stride
Weight
5 5 5 5 5 5 5 5 5 5 10
150 50 5 20 20 10 5 3 5 5 200
A.2
Downstream adaptation
The transfer stage is built on π0 [5], with PaliGemma-2B [45] as the vision-language encoder and Gemma-300M as the action expert. We do not add a latent-to-action decoder; latent transitions enter the policy through the input pathway in Sec. 3.3. For each view, the frozen ALAM encoder turns H+1 consecutive frames into H continuous latent transitions, which are linearly projected to the action dimension and interleaved with the action stream. The encoder E is frozen throughout, and only the π0 backbone, action expert, and the linear projection are updated. Policies are trained with flow matching on 8×H20 GPUs; default hyperparameters are in Tab. 8, with per-task overrides (e.g., training steps for short vs. long horizons) noted in the corresponding benchmark protocol. Table 8: Downstream finetuning hyperparameters (defaults).
B
Hyperparameter
Value
Optimizer Learning rate Weight decay Per-GPU batch size GPUs Action chunk H Vision resolution Frozen modules Trainable modules
AdamW (β1 =0.9, β2 =0.95) 5×10−5 10−4 32 8×H20 (90 GB) 16 224×224 ALAM encoder E PaliGemma-2B, Gemma-300M, linear projection
Simulation Benchmark Protocols
On MetaWorld MT50 [34], we train a single multi-task policy on all 50 tasks and follow the Evo1 [29] protocol: each task is evaluated with 10 rollouts of up to 200 environment steps, with success defined by the task-specific success flag. We report per-tier means (Easy / Medium / Hard / Very Hard, with the same partitioning as Evo-1) and the overall average; the full task split is in App. C. The action horizon is 5 at both training and inference for π0 and π0 +ALAM. On LIBERO [31], we use the four suites (Spatial, Object, Goal, Long) and follow the original protocol: each suite is evaluated with 500 episodes from unseen initial states, and we report per-suite success and the overall average. All four suites share a single trained checkpoint with action horizon H=20, and only the inference-time action horizon and replan interval differ across suites (Tab. 9). Table 9: LIBERO inference configuration. All suites share the same trained checkpoint (training H=20); only the inference action horizon and replan step differ.
C
Suite
Inference H
Replan step
Spatial Object Goal Long
14 14 18 18
5 10 7 12
Task Difficulty Definitions in MetaWorld MT50
We follow [39] and partition the MetaWorld MT50 tasks into four difficulty tiers (Tab. 10).
D
Real-World Tasks: Setup and Protocol
All real-world experiments use an Agile-X Piper 6-DoF arm with a parallel-jaw gripper. The robot is observed by two synchronized RGB cameras: a third-person Intel RealSense D435 on a tripod and a wrist-mounted RealSense D405. Both streams are resized to 224×224 and synchronized at the control rate. We collect 50 teleoperated demonstrations per task (200 in total), used only for downstream finetuning; no real-world data is used during ALAM pretraining. 15
Table 10: Task difficulty partitioning for MetaWorld MT50. Difficulty
Task Names
Easy (28 tasks)
button-press, button-press-topdown, button-press-topdown-wall, button-press-wall, coffee-button, dial-turn, door-close, door-lock, door-open, door-unlock, drawer-close, drawer-open, faucet-close, faucet-open, handle-press, handle-press-side, handle-pull, handle-pull-side, lever-pull, plate-slide, plate-slide-back, plate-slide-back-side, plate-slide-side, reach, reach-wall, window-close, window-open, peg-unplug-side
Medium (11 tasks)
basketball, bin-picking, box-close, coffee-pull, peg-insert-side, push-wall, soccer, sweep, sweep-into
Hard (6 tasks)
assembly, hand-insert, pick-out-of-hole, pick-place, push, push-back
Very Hard tasks)
(5
coffee-push,
hammer,
shelf-place, disassemble, stick-pull, stick-push, pick-place-wall
Insert Cylinder: insert the cylinder into the matching base.
Insert Cube: insert the cube into the matching base.
Stack Cup: stack the left and right cups onto the center cup.
Fold Towel: fold the towel in half twice.
Figure 5: Key frames of real-world rollouts from ALAM.
We evaluate on four tasks. Insert Cylinder. The robot grasps a cylinder and inserts it into a matching socket base, with one object and its base on the tabletop per trial; we run 20 trials, and a trial counts as successful if the cylinder is correctly inserted and remains stably seated after release. The success rate is reported on a 0–100 scale, equivalent to 5 points per successful trial. Insert Cube. Same setup as Insert Cylinder, but with a cube; the geometry requires more accurate orientation alignment, making the task more sensitive to execution precision. Stack Cup. Three cups are placed upside down on the table, and the robot stacks the left and right cups onto the center cup in sequence. We run 10 trials; in each trial, stacking one cup gives 5 points and both cups give 10, so the total over 10 trials is on a 0–100 scale. Fold Towel. Starting from a flat towel, the robot folds it in half twice. We run 10 trials; each successful fold gives 5 points, so completing both folds gives 10 per trial, and the sum across trials is on a 0–100 scale. 16
E
More Transfer Results
Figure 6: Cross-domain additivity (ground-truth target reference). Each panel has two rows. Source is a trajectory in the source domain; Target is the reconstruction in a different target domain obtained by transferring the inferred latent transitions. Columns: anchor oa ; forward state ôab ; forward state ôac ; and additive composition ôab + ôbc , which should align with ôac if the latent transitions are additive. Hatted symbols denote reconstructions; un-hatted denote ground truth.
Figure 7: Cross-domain additivity (transferred target). Same layout as Fig. 6, but the target rows are produced solely from transferred latent transitions, with no ground-truth target frames as reference. We pick two domains from the pretraining mixture and use one as source and the other as target. Fig. 6 uses ground-truth target-domain frames as the rendering reference, which gives an upper bound on visual fidelity that depends only on the decoder. Fig. 7 uses transferred latent transitions as the only source of target-domain dynamics, with no ground-truth target frames as reference. In both settings, the composed reconstruction ôab + ôbc is close to the direct ôac rollout across the four source/target pairs, including pairs that differ in viewpoint, embodiment, and scene appearance. We 17
do not claim domain-invariance in general; the qualitative match supports using the latent transitions as an auxiliary signal for downstream policies (Sec. 3.3).
F
Inference-time intervention on the latent stream
As an additional check, we apply three test-time interventions to the frozen π0 +ALAM model (no retraining or finetuning). Freeze-at-noise keeps the latent input fixed at the initial Gaussian noise sample throughout the flow-matching trajectory. Block attention masks the action-to-latent crossattention. Temporal shuffle permutes the temporal order of the generated latents within each rollout while keeping their values. On MetaWorld MT50, all three interventions lower the average success rate from 85.0% to between 75 and 77 points (Tab. 11); we include this as a sanity check that the latent stream is used at inference, and do not draw further conclusions from the relative ordering of the three interventions. Table 11: Test-time interventions on the latent stream (MetaWorld MT50, per-tier success rate %). All rows share the same trained weights.
G
Variant
Easy Med. Hard V.Hard Avg.
π0 (no-latent baseline) ALAM (default)
71.8 89.3
48.2 83.6
41.7 85.0
30.0 82.0
47.9 85.0
Freeze latent Block A→L attention Shuffle latent
86.4 83.9 84.3
62.7 68.2 65.5
65.0 63.3 63.3
90.0 90.0 88.0
76.3 76.4 75.3
Broader Impacts
This work may help reduce reliance on expensive action-labeled robot data and improve the data efficiency of robot policy learning, which can make robot learning research more accessible. As with other learned policies, real-world deployment still requires task-specific evaluation and system-level safety measures.
18