arXiv CS open access

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos

Zhiyuan Xiang et al. · arxiv_cs

arXiv CS · Papers

Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and dive

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

LAPT-Scale: Scaling Embodied Policies with Latent Action Pre-Training on Human Videos Zhiyuan Xiang1,2,* Peidong Liu1,*,† Mingyang Li1,3,* Jiawei Li1,† Wenhow Li1,4 Wenbo Ding3 Xiang Deng2,† 1

Joy Future Academy, JD School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 3 Tsinghua SIGS, Tsinghua University 4 Artificial Intelligence, The Hong Kong University of Science and Technology (Guangzhou) * Equal contribution † Corresponding authors lpd19@tsinghua.org.cn li-jw15@tsinghua.org.cn dengxiang@hit.edu.cn

arXiv:2610.13732v1 [cs.CV] 9 Oct 2026

Abstract: Scaling generalist robotic policies is limited by the high cost of collecting embodied interaction data. Large-scale egocentric human videos offer a rich alternative, but they lack robot action labels, and existing action annotation methods are often noisy under real-world visual complexity and diverse manipulation dynamics. We introduce LAPT-Scale, a scalable latent action pretraining paradigm with two components: LAPT-Scale-LAM learns continuous latent actions by explaining frame-to-frame visual transitions in human videos and uses them as human-derived supervision, while LAPT-Scale-Policy trains a VLA policy through human-video latent-action pre-training, target-robot embodiment alignment, and task-specific post-training. Unlike prior vector-quantized LAMs, LAPT-Scale-LAM uses a continuous latent space to preserve fine-grained motion and interaction dynamics. We further combine perceptual, semantic, and opticalflow alignment signals to improve transition-level supervision beyond pixel reconstruction. Trained on 23.3K hours of egocentric video, LAPT-Scale-LAM provides implicit action supervision for policy pre-training, yielding consistent scaling gains and improving real-robot success from 63.3% to 85.0% as annotated human video scales from 1K to 5K hours. The learned policy also generalizes to unseen visual conditions. Keywords: Latent action models, human-video pre-training, robot policy scaling

Introduction

Scaling data drives progress in vision and language, but embodied intelligence faces a sharper bottleneck: interaction data requires physical robots or simulators, making it far more expensive than text or static images. Large-scale human videos, especially egocentric ones, offer a promising way to relieve this bottleneck [1]. They naturally record diverse everyday interactions at a scale beyond robot-collected trajectories, and transforming such videos into action-like supervision could complement robot-only datasets such as Open X-Embodiment [2]. The central challenge is that human videos lack robot action labels, and human motion cannot be directly mapped to robot control spaces. Existing methods, such as VITRA [3], often convert human videos into pseudo-actions through explicit motion estimation, where estimation errors can lead to noisy action annotations; simply adding more human video does not necessarily improve policies. Latent Action Models (LAMs) offer an alternative: they infer implicit actions between consecutive frames end-to-end, without predefined robot action labels [4, 5, 6, 7]. Because these implicit actions are optimized to explain visual transitions, they provide compact supervision for unlabeled videos. 10th Conference on Robot Learning (CoRL 2026), Austin, Texas, USA.

Despite this potential, existing LAMs are limited by their architectural priors and alignment signals. Many recent methods use Vector Quantized (VQ) latent spaces [8, 5, 6]. While compact, a fixed discrete codebook can over-compress the continuous changes that define egocentric manipulation, weakening the supervision needed for fine-grained policy learning. In addition, pixel-wise reconstruction signals such as MSE are often dominated by large static regions, providing weak signals for small but important hand-object interactions. As a result, learned latent actions may miss transition-relevant structure needed for downstream embodied policies. To address these limitations, we introduce LAPT-Scale, a scalable latent action pre-training paradigm for embodied policy learning. LAPT-Scale contains two components: LAPT-Scale-LAM, which constructs human-derived supervision with a continuous Latent Action Model trained on large-scale human videos, and LAPT-Scale-Policy, which uses this supervision to train a VLA policy. The policy training follows three stages: human-video latent-action pre-training, embodiment alignment with target-robot data, and task-specific post-training in simulation and the real world. Unlike discrete LAMs, LAPT-Scale-LAM parameterizes implicit actions in a continuous space to preserve fine-grained transition dynamics. To make this space perceptually grounded and temporally consistent, we train the LAM with alignment signals beyond pixel reconstruction: perceptual alignment for local contact fidelity [9], semantic alignment with self-supervised features for hand-object structure [10], and optical-flow alignment for observed dynamics [11]. We train LAPT-Scale-LAM on 23.3K hours of diverse egocentric videos and use it to annotate a 5.1K-hour cleaned human-video subset for policy pre-training. With 250 hours of self-collected robot data for embodiment alignment, LAPT-Scale reaches 90.0%/89.3% success on RoboTwin Easy/Hard, 64.8% on RoboCasa-GR1. Scaling LAM-annotated human video from 1K to 5K hours further improves real-robot success from 63.3% to 85.0%, suggesting that continuous latent action annotation is a promising route for benefiting from larger human-video corpora. On four harder real-robot tasks, it retains 70.0–75.0% success under unseen position, illumination, and background shifts. Our contributions are summarized as follows: • LAPT-Scale-LAM: a continuous latent action model with perceptual, self-supervised feature, and optical-flow alignment signals for learning a more expressive latent action space from human videos. • LAPT-Scale-Policy: a three-stage framework for human-video latent-action pre-training, target-robot embodiment alignment, and downstream post-training. • Real-robot scaling: increasing only Stage-1 human-video pre-training data improves downstream real-robot success from 63.3% to 85.0%.

Related Work

Vision-language-action policy learning. Vision-language-action (VLA) models unify visual perception, language conditioning, and action generation for generalist manipulation. Recent systems scale VLA policies with web knowledge, robot data, flow-based action modeling, and crossembodiment training [12, 13, 14, 15, 16, 17, 18, 19, 20], while other work studies backbone, adaptation, spatial-representation, and action-space choices [21, 22, 23, 24]. However, VLA training remains constrained by action-labeled robot trajectories. Egocentric VLA methods such as VITRA and EgoScale reduce this bottleneck by converting human videos into action-like targets before robot alignment [3, 25], but such targets depend on explicit motion estimation or retargeting signals. Simulation-mediated pipelines also convert reconstructed human demonstrations into physically checked robot-centered trajectories [26]. LAPT-Scale instead annotates human videos with LAM-derived implicit actions. Human video data for embodied learning. Human video datasets provide broad coverage of physical interaction, from object manipulation in Something-Something and EPIC-KITCHENS to 2

LAM Pre-Training

Policy Pre-Training

Policy Mid-Training

Policy Post-Training

23K hours human video

5K hours human video

250 hours robot data

Real world

Policy Post-Training RoboCasa-GR1

LAM Annotation 5K hours human video

Policy Post-Training RoboTwin 2.0

Figure 1: Overview of LAPT-Scale. LAPT-Scale-LAM annotates egocentric human videos with continuous implicit actions. LAPT-Scale-Policy uses these annotations for human-video pretraining, followed by target-robot embodiment alignment and downstream post-training. daily-life egocentric video in Ego4D [27, 28, 1]. Recent resources expand scale and annotation density with paired ego-exo views, factory videos, 3D/4D annotations, dexterous demonstrations, multimodal interaction labels, and filtered Ego4D text/action annotations [29, 30, 31, 32, 33, 34]. These datasets are useful visual experience, but their human observations and motion annotations do not directly specify robot-executable actions. LAPT-Scale learns latent actions to bridge this missing-label gap before robot embodiment alignment. Latent action models from videos. Latent Action Models infer action-like variables from observation transitions, making unlabeled videos useful for behavior learning. Related work learns compact transition variables through inverse-dynamics, world-model, or latent-action objectives and integrates them into VLA or world-model pipelines [4, 5, 35, 36, 37, 6, 7, 38, 39, 40]. However, many LAMs rely on discrete or vector-quantized latents, which can over-compress continuous egocentric manipulation dynamics [8, 5, 6]. Reconstruction-only training can be dominated by static background regions. LAOF augments reconstruction with optical-flow constraints to improve robustness to visual distractors [41]. LAPT-Scale-LAM therefore uses continuous latents together with perceptual, self-supervised semantic, and optical-flow alignment signals to preserve fine-grained implicit actions for human-video policy pre-training [9, 10, 11].

Method

3.1

Problem Formulation & Data

Pipeline. As shown in Fig. 1, LAPT-Scale contains two modules. LAPT-Scale-LAM learns con(i) tinuous latent actions from an egocentric human-video corpus Dh = {x1:T } and annotates a cleaned h h h subset as Dz = {(ct , z̄t )}. LAPT-Scale-Policy trains a VLA policy in three stages: human-video latent-action pre-training on Dzh , target-robot embodiment alignment on Dralign , and downstream post-training on Drtask . Human-video data and filtering. Our LAM is trained on 23.3K hours of diverse egocentric videos, while policy pre-training uses a 5.1K-hour cleaned subset with more reliable manipulation content. Fig. 2 summarizes the corpus diversity. Following EgoVid-5M [34], we remove corrupted or blurry clips and discard segments dominated by camera motion or weak hand-object interaction.

Action verbs

Manipulated objects

(a) Scene overview

(b) Action-object semantic diversity

Figure 2: Human-video data overview. The corpus spans everyday egocentric manipulation across varied environments and interactions, providing broad coverage for latent-action learning. This lets the LAM learn from broad visual diversity while policy scaling uses cleaner action-like supervision. 3.2

LAPT-Scale-LAM: Constructing Human Latent-Action Supervision

LAPT-Scale-LAM explains visual transitions in human videos. Given adjacent frames (xt , xt+1 ) from a clip x1:T , the encoder Eϕ : (xt , xt+1 ) 7→ zt produces a transition-level latent action zt ∈ Rd that captures hand-object dynamics from xt to xt+1 . Applying the same transition encoder over the clip yields z1:T −1 . We implement the LAM as a spatio-temporal VAE [42]: for each transition, a learnable action query q0 is prepended to the two-frame patch embeddings and processed by a Transformer encoder [43]. The query output parameterizes a diagonal Gaussian posterior qϕ (zt | xt , xt+1 ), from which the continuous latent action is sampled. A low-dimensional bottleneck (e.g., d = 32), together with a decoder that separately receives current visual context, encourages zt to capture frame-to-frame change rather than static appearance.

x 𝒕"𝟏

x𝒕

Latent action encoder

GT Frame LPIPS

RAFT

Alignment

z!𝒕

Latent action decoder

DINO

LPIPS

DINO

RAFT

Pred Frame

x! 𝒕"𝟏

Figure 3: LAPT-Scale-LAM training alignments. The latent action explains the transition from the current frame to the next frame. Multilevel alignments encourage it to capture handobject motion rather than static appearance.

To reconstruct the future frame, we inject the latent action into current-frame patch embeddings, dt,n = Wp pt,n + Wz zt , and use a spatial Transformer decoder Dθ to predict x̂t+1 . The LAM is aligned with visual fidelity, semantic structure, and observed motion signals, as illustrated in Fig. 3. The latent action encoder infers continuous transition variables, and the decoder is trained with reconstruction, perceptual, semantic, and optical-flow alignment terms. These signals are complementary: pixel and perceptual alignment keep the predicted frame visually faithful, semantic features preserve object-centric structure under appearance changes, and optical-flow alignment encourages the predicted transition to match the observed motion field. We therefore supervise the decoder at multiple levels rather than relying on reconstruction alone. Concretely, the training alignment contains the following terms. • Perceptual alignment. Pixel reconstruction and LPIPS [9] preserve local structure: Lperc = λmse ∥x̂t+1 − xt+1 ∥22 + λlpips LPIPS(x̂t+1 , xt+1 ). 4

(1)

• Semantic alignment. Frozen DINOv2 [10] features preserve hand-object structure: 1 1 ∥S(P (x̂t+1 )) − S(P (xt+1 ))∥22 + ∥c(x̂t+1 ) − c(xt+1 )∥22 , (2) 2 2 where P and c denote patch and CLS features, and S(·) denotes spatial pooling over patch features. Ldino =

• Motion alignment. A frozen RAFT [11] estimator matches reconstructed and observed optical-flow fields: Lflow = ∥R(xt , x̂t+1 ) − sg[R(xt , xt+1 )]∥1 ,

(3)

where sg[·] stops gradients through the target flow. The full LAM training objective combines these alignment terms: LLAM = Lperc + λdino Ldino + λflow Lflow + βLKL ,

(4)

where LKL keeps the continuous latent space close to the prior. The frozen encoder then generates the latent-action labels used in Stage 1 of VLA training. 3.3

LAPT-Scale-Policy: Three-Stage VLA Policy Training

LAPT-Scale-Policy contains a VLM backbone Fω that encodes the policy context c and uses the JoyAI-RA policy architecture [44], with a Perceiver-style action expert Aη trained by Conditional Flow Matching [45]. We keep the same action expert across three stages: Stage 1 predicts chunks h on Dzh , where each z̄τh is the posterior mean for one of LAM-produced latent actions y = z̄t:t+H−1 human-video transition and h indicates the human-video source; Stage 2 switches the target to robot action chunks y = art:t+H−1 on Dralign for embodiment alignment; Stage 3 continues on Drtask for downstream post-training. No separate human-video head, adapter, or reinitialization is introduced. For each target y and context c, we sample noise ξ ∼ N (0, I) and flow time τ ∈ [0, 1] to construct y τ = τ y + (1 − τ )ξ. The flow-matching objective is 2

LFM (c, y) = ∥Aη (c, y τ , τ ) − (y − ξ)∥2 .

(5)

Targets are padded to the action expert’s maximum dimension and masked in the loss when needed. At inference time, the policy predicts executable robot actions; LAM-encoded actions are used only as scalable Stage-1 supervision.

Experiments

We evaluate LAPT-Scale along three axes: downstream VLA performance in simulation and the real world, scaling with LAM-annotated human-video data, and standalone LAM quality. The downstream benchmarks test executable robot behavior after embodiment alignment, while the LAM diagnostics test whether the annotation model preserves transition-relevant visual and semantic information. 4.1

Experimental Setup

LAPT-Scale-LAM is trained on a 23.3K-hour egocentric human-video corpus. The main policy run uses a 5.1K-hour cleaned subset annotated by the frozen LAM, and the scaling study uses nested 1K-, 3K-, and 5K-hour subsets with all robot data held fixed. For downstream evaluation, we test on RoboTwin 2.0 [46], RoboCasa-GR1 Tabletop [47, 16], and seven real-robot tasks on an AgiBot G1 robot after 250 hours of target-robot embodiment alignment. We report task success rate for policy evaluation and use human-video reconstruction metrics plus LARY linear-probe accuracy for LAM diagnostics. Data quantities and evaluation details are in Appendix D. 5

4.2

Main Results

We compare the full LAPT-Scale pipeline against strong VLA baselines after Stage-1 pre-training on the 5.1K-hour LAM-annotated subset, target-robot embodiment alignment, and downstream posttraining. Baseline sources and evaluation details are summarized in the appendix. Table 1: Main downstream policy results on RoboTwin 2.0. Results are average success rates in the Easy and Hard settings. Method

π0 [14] π0.5 [15] Motus [7] LingBot-VLA [18] JoyAI-RA [44] LAPT-Scale Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard Easy Hard

Success Rate (%) 65.9 58.4 82.7 76.8 88.7 87.0 88.6

86.7

88.6

87.2

90.0

89.3

On RoboTwin 2.0, LAPT-Scale reaches 90.0% in the Easy setting and 89.3% in the Hard setting, slightly outperforming the strongest listed baselines. This indicates that the benefit is not limited to fixed initial configurations, since RoboTwin Hard varies object poses and scene layouts. Table 2: Main downstream policy results on RoboCasa-GR1. Results are average success rates. Method

GR00T-N1.6 [16]

Qwen3PI [48]

ABot-M0 [19]

JoyAI-RA [44]

LAPT-Scale

47.6

43.9

58.3

63.2

64.8

Success Rate (%)

On RoboCasa-GR1, LAPT-Scale reaches 64.8% average success, showing that the gain also transfers to household tabletop manipulation. Table 3: Main downstream policy results on the three original real-robot tasks, with 20 trials per task. Diffusion Policy uses the same 250-hour robot corpus. Method

π0.5 [15] JoyAI-RA [44] LAPT-Scale Mouse Head. Cup Avg. Mouse Head. Cup Avg. Mouse Head. Cup Avg.

Success Rate (%)

55.0

40.0

75.0 56.7

80.0

95.0

65.0 80.0

85.0

95.0

75.0 85.0

Diffusion Policy [49]: Mouse 50.0, Head. 40.0, Cup 45.0, Avg. 45.0.

The real-robot results test whether human-video latent supervision remains useful after embodiment alignment. LAPT-Scale achieves 85.0% average success over the three tasks, improving over π0.5 and JoyAI-RA. At the task level, it improves over JoyAI-RA on Mouse Pad and Cup while matching it on Headphones. Overall, Tables 1–3 suggest that Stage-1 latent-action pre-training can benefit both simulation and physical robot deployment. We view these results as evidence that LAM-based human-video pre-training remains compatible with strong downstream policy training; the scaling behavior is analyzed separately below. 4.3

Scaling with Annotated Human Video

We next study the central scaling question: whether increasing only the amount of LAM-annotated human video improves downstream robot performance. The LAM is always trained on the full 23.3K-hour corpus; we vary only the amount of cleaned, filtered, and annotated human video used for Stage-1 VLA pre-training, while keeping robot data, policy architecture, and evaluation protocol fixed. We compare three forms of human-video supervision. Hand-motion annotation uses explicit handderived pseudo-actions, the DreamDojo LAM [37] uses its official pretrained reconstruction-based LAM, and LAPT-Scale-LAM uses the full perceptual, semantic, and motion-aware alignment signals. This comparison tests whether scale helps from more video alone or from higher-quality action-like supervision. The pretrained DreamDojo LAM differs from the MSE-only diagnostic trained on our data with its reconstruction-plus-KL recipe.

1K hours

3K hours

(b) Real-robot performance scaling

5K hours

Average success rate (%)

Policy pre-training loss

(a) Policy pre-training loss 0.10 0.08 0.06 0.04 0.02

Training steps (K)

100

Hand-motion annotation

DreamDojo LAM

LAPT-Scale-LAM

80 70 60 50

Human-video pre-training data

Figure 4: Scaling behavior of LAPT-Scale. Left: policy pre-training loss for different amounts of human-video pre-training data. Right: real-robot average success rate as the amount of human-video pre-training data increases, with all robot data held fixed. Fig. 4 shows that LAPT-Scale-LAM benefits most consistently from additional human video, improving from 63.3% at 1K hours to 80.0% at 3K hours and 85.0% at 5K hours. Larger LAMannotated datasets also reach lower Stage-1 pre-training losses. The baselines show why annotation quality matters. Explicit hand-motion pipelines can accumulate source-dependent hand-estimation noise under camera motion, occlusion, hand-visibility changes, and viewpoint shifts. A hand tracker can produce plausible but wrong pseudo-actions in these cases, and downstream policy training does not correct the labels. Such noise may explain why hand motion improves from 1K to 3K hours but degrades to 55.0% at 5K hours. DreamDojo LAM scales more reliably than hand motion and reaches 76.7% at 5K hours, but remains below LAPT-Scale-LAM. This suggests that pixel-level future-frame reconstruction alone may not fully capture the semantic and motion structure needed for moving hand-object interactions, motivating the controlled alignment ablation below. The full per-task breakdown and shared 0-hour control (55.0%) are in Appendix E. 4.4

Controlled Stage-1 and Alignment Ablations

We remove Stage 1 with initialization, robot data, Stage-2/3 schedules, and evaluation fixed, and extend Stage-2 training in a compute-matched control (Table 4; Appendix E.1). Both remain below the full model. Table 4: Stage-1 controls: average success on the three original real-robot tasks. Stage-1 supervision

Stage-2 training

Success (%)

None None LAPT-Scale, 5K hours

Standard Extended, compute-matched Standard

55.0 53.3 85.0

We retrain the annotation-to-policy pipeline after removing each LAM alignment term (Table 5). Table 5: Controlled LAM alignment ablation. RT-H is RoboTwin Hard success. LAM objective

PSNR

LARY

RT-H (%)

MSE-only (Recon.+KL) Full w/o LPIPS Full w/o DINO Full w/o flow Full objective

33.9844 34.2105 34.1500 34.3012 34.4217

0.675 0.691 0.698 0.685 0.713

74.2 82.5 84.1 77.8 89.3

Removing flow yields the largest RoboTwin Hard drop (11.5 points), despite high PSNR, illustrating the limits of pixel fidelity as a diagnostic. 7

4.5

Harder Real-Robot Tasks and Generalization

Four harder tasks cover paper-ball disposal, pen and self-occluded eraser grasping, and multi-stage book stowing. One joint policy uses 200 demonstrations per task, also used to post-train π0.5 . We test seen conditions and unseen position, illumination, and background shifts with 20 trials per task per domain (protocols and domain images in Appendix H). Table 6: Four-task generalization success (%); Average includes all four domains. Method

Seen

Position

Illumination

Background

Average

Diffusion Policy π0.5 LAPT-Scale

60.0 85.0 85.0

20.0 50.0 75.0

15.0 45.0 70.0

15.0 40.0 70.0

27.5 55.0 75.0

LAPT-Scale and π0.5 match at 85.0% in seen conditions, but reach 70.0–75.0% and 40.0–50.0%, respectively, under the tested shifts (Table 6). 4.6

Human-Video LAM Evaluation

We evaluate the LAM before policy training with two diagnostics: future-frame reconstruction on human-video transitions and first-person action recognition on LARY [50] using frozen latent actions with a linear probe. We compare three LAM variants under the same transition-prediction protocol. VQ-LAM uses a discrete codebook [8]; MSE-only LAM keeps the continuous latent space but removes the LPIPS, DINO, and optical-flow alignment terms; LAPT-Scale-LAM uses the full continuous latent space and action-aware alignment signals. This tests latent-space and alignment choices. Table 7: Human-video LAM evaluation. Reconstruction metrics are measured on human-video transitions; LARY reports first-person human-action classification accuracy from frozen latent actions. LAM Variant

PSNR ↑

SSIM ↑

LPIPS ↓

LARY Acc. ↑

VQ-LAM MSE-only LAM LAPT-Scale-LAM

18.3416 33.9844 34.4217

0.6741 0.9234 0.9326

0.3443 0.0769 0.0546

0.232 0.675 0.713

Table 7 shows that LAPT-Scale-LAM improves both transition reconstruction and latent-action semantics. The gap to VQ-LAM suggests that discrete codes are too coarse for fine-grained egocentric hand-object dynamics, while the gain over MSE-only LAM shows the value of perceptual, semantic, and motion alignment.

Figure 5: Human-video transition reconstruction. Given the previous frame, VQ-LAM, MSEonly LAM, and LAPT-Scale-LAM predict the next frame, shown against the ground truth. Additional visualizations are provided in the appendix. The qualitative example in Fig. 5 illustrates the same trend: VQ-LAM produces blurry reconstructions with noticeable hand-position errors, and MSE-only LAM remains visually blurred. LAPTScale-LAM reconstructs the transition most similarly to the ground truth, especially around the moving hand and object.

These diagnostics complement downstream evaluation by showing that Stage-1 annotations retain transition information before robot embodiment alignment. 4.7

Interpreting the Learned Latent Action Space

To probe what the learned latent action token represents, we perform counterfactual latent-action replacement on human ego-view videos, which match the training source of LAPT-Scale-LAM. For each donor-target pair, we keep the target visual context fixed but replace its latent action with one inferred from a donor video. We then decode three short-horizon predictions: the donor self-rollout, the target base rollout, and the target rollout under donor-latent replacement. This isolates the latent action from scene context: if the token carries transition information, the replacement should alter the target rollout without simply copying donor appearance. Donor latent source

Target base prediction

Target + donor latent

Figure 6: Human ego-view latent-action replacement across three donor-target pairs. Rows show donor self-rollout frames, the target base prediction, and the target prediction after replacing the target latent action with the donor latent action. Across all three examples, the replacement changes the short-horizon motion pattern while preserving the overall target scene layout. Across the examples in Figure 6, replacement changes how the target sequence evolves without globally rewriting the target scene. The clearest effects appear as camera-relative hand-object motion changes: the donor latent action redirects the short-horizon movement tendency, while the scene layout remains largely target-specific. These results suggest that LAM tokens encode transferable information about how the scene changes, rather than only what the scene contains.

Conclusion and Limitations

We presented LAPT-Scale, a latent action pre-training paradigm that turns large-scale egocentric human videos into supervision for embodied policy learning. By learning continuous transition variables directly from video and using them for Stage-1 VLA pre-training, LAPT-Scale provides a scalable way to exploit human videos without explicit robot action labels. Across simulation, real-robot, scaling, and LAM diagnostic experiments, our results suggest that LAM-based humanvideo supervision improves downstream manipulation and becomes more useful as the amount of annotated human video increases, supporting latent actions as scalable human-video annotations. Four harder real-robot tasks also show robustness under the tested position and appearance shifts. The main limiting assumption is that visual latent actions learned from human videos can become useful robot-policy supervision after embodiment alignment; they are not guaranteed to align with independently controllable robot degrees of freedom. As a result, target-robot embodiment alignment remains necessary. The approach may also fail when videos contain weak manipulation signal, large embodiment gaps, heavy occlusion, or visual changes dominated by ego-camera motion. Our experiments are limited in robot and task coverage: the real-robot study uses seven tasks and one target platform, with 20 trials per condition. This limits the precision of success estimates, and broader validation is needed before claiming deployment robustness. Future work should study stronger embodiment alignment, longer-horizon latent-action composition, broader robot coverage, and more systematic scaling laws across human-video data, model size, and robot demonstrations.

References [1] K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, et al. Ego4D: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18995–19012, 2022. [2] A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In IEEE International Conference on Robotics and Automation, pages 6892–6903. IEEE, 2024. [3] Q. Li, Y. Deng, Y. Liang, L. Luo, L. Zhou, C. Yao, L. Zeng, Z. Feng, H. Liang, S. Xu, Y. Zhang, X. Chen, H. Chen, L. Sun, D. Chen, J. Yang, and B. Guo. Scalable vision-language-action model pretraining for robotic manipulation with real-life human activity videos. arXiv preprint arXiv:2510.21571, 2025. [4] J. Bruce, M. D. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalankar, R. Steigerwald, C. Apps, et al. Genie: Generative interactive environments. In International Conference on Machine Learning, 2024. [5] S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin, et al. Latent action pretraining from videos. arXiv preprint arXiv:2410.11758, 2024. [6] Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. UniVLA: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025. [7] H. Bi, H. Tan, S. Xie, Z. Wang, S. Huang, H. Liu, R. Zhao, Y. Feng, C. Xiang, Y. Rong, et al. Motus: A unified latent action world model. arXiv preprint arXiv:2512.13030, 2025. [8] A. van den Oord, O. Vinyals, and K. Kavukcuoglu. Neural discrete representation learning. Advances in Neural Information Processing Systems, 30, 2017. [9] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. URL https://arxiv.org/abs/1801.03924. [10] M. Oquab, T. Darcet, T. Moutakanni, H. V. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y. Huang, S.-W. Li, I. Misra, M. Rabbat, V. Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. Joulin, and P. Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. [11] Z. Teed and J. Deng. RAFT: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, 2020. [12] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, et al. RT-2: Vision-language-action models transfer web knowledge to robotic control. In Conference on Robot Learning, pages 2165–2183. PMLR, 2023. [13] M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kollar, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246. [14] K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al. π0 : A vision-language-action flow model for general robot control, 2024. URL https://arxiv.org/abs/2410.24164.

[15] Physical Intelligence, K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, M. Y. Galliker, D. Ghosh, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, D. LeBlanc, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, A. Z. Ren, L. X. Shi, L. Smith, J. T. Springenberg, K. Stachowicz, J. Tanner, Q. Vuong, H. Walke, A. Walling, H. Wang, L. Yu, and U. Zhilinsky. π0.5 : a vision-languageaction model with open-world generalization, 2025. URL https://arxiv.org/abs/2504. 16054. [16] NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, et al. GR00T N1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734, 2025. [17] Gemini Robotics Team, S. Abeyruwan, J. Ainslie, J.-B. Alayrac, M. G. Arenas, T. Armstrong, A. Balakrishna, R. Baruch, M. Bauza, M. Blokzijl, et al. Gemini Robotics: Bringing AI into the physical world. arXiv preprint arXiv:2503.20020, 2025. [18] W. Wu, F. Lu, Y. Wang, S. Yang, S. Liu, F. Wang, Q. Zhu, H. Sun, Y. Wang, S. Ma, et al. A pragmatic VLA foundation model. arXiv preprint arXiv:2601.18692, 2026. [19] Y. Yang, S. Zeng, T. Lin, X. Chang, D. Qi, J. Xiao, H. Liu, R. Chen, Y. Chen, D. Huo, et al. ABot-M0: VLA foundation model for robotic manipulation with action manifold learning. arXiv preprint arXiv:2602.11236, 2026. [20] JoyAI-RA Team. JoyAI-RA 0.5: Scaling robot manipulation learning via dual action alignment, 2026. URL https://arxiv.org/abs/2608.05674. [21] X. Li, P. Li, L. Qian, M. Liu, D. Wang, J. Liu, B. Kang, X. Ma, X. Wang, D. Guo, T. Kong, H. Zhang, and H. Liu. What matters in building vision-language-action models for generalist robots. arXiv preprint arXiv:2412.14058, 2024. [22] Y. Wang, P. Ding, L. Li, C. Cui, Z. Ge, X. Tong, W. Song, H. Zhao, W. Zhao, P. Hou, et al. VLA-Adapter: An effective paradigm for tiny-scale vision-language-action model. arXiv preprint arXiv:2509.09372, 2025. [23] D. Qu, H. Song, Q. Chen, Y. Yao, X. Ye, Y. Ding, Z. Wang, J. Gu, B. Zhao, D. Wang, and X. Li. SpatialVLA: Exploring spatial representations for visual-language-action model. arXiv preprint arXiv:2501.15830, 2025. [24] J. Zheng, J. Li, D. Liu, Y. Zheng, Z. Wang, Z. Ou, Y. Liu, J. Liu, Y.-Q. Zhang, and X. Zhan. Universal actions for enhanced embodied foundation models. arXiv preprint arXiv:2501.10105, 2025. [25] R. Zheng, D. Niu, Y. Xie, J. Wang, M. Xu, Y. Jiang, F. Castañeda, F. Hu, Y. L. Tan, L. Fu, T. Darrell, F. Huang, Y. Zhu, D. Xu, and L. Fan. EgoScale: Scaling dexterous manipulation with diverse egocentric human data, 2026. URL https://arxiv.org/abs/2602.16710. [26] P. Liu, Y. Liu, S. Guo, F. Ma, Z. Yuan, A. Li, Z. Chen, W. Li, T. Zhang, M. Li, J. Zhang, J. Xiong, Z. Xiang, D. Chi, Y. Zhuang, L. Luo, W. Tan, D. Li, N. Jiang, Y. Li, Q. He, J. Liang, C. Cai, M. Luo, H. Zhang, P. Hao, S. Wang, N. Qiao, Y. Gao, L. Kang, J. Xiong, R. Li, J. Li, H. Shen, Y. Gong, N. Duan, and L. Lin. JoyAI-Sim: A simulation-enabled interconversion toolchain for the embodied data pyramid, 2026. URL https://arxiv.org/abs/2606. 16776. [27] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, I. Fruend, P. Yianilos, M. Mueller-Freitag, F. Hoppe, C. Thurau, I. Bax, and R. Memisevic. The something something video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, 2017.

[28] D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, et al. Scaling egocentric vision: The EPIC-KITCHENS dataset. In European Conference on Computer Vision, pages 720–736, 2018. [29] K. Grauman, A. Westbury, L. Torresani, K. Kitani, J. Malik, T. Afouras, K. Ashutosh, V. Baiyya, S. Bansal, B. Boote, et al. Ego-Exo4D: Understanding skilled human activity from first- and third-person perspectives. International Journal of Computer Vision, 133(12): 8356–8435, 2025. doi:10.1007/s11263-025-02557-6. [30] Build AI. Egocentric-10K, 2025. builddotai/Egocentric-10K.

URL https://huggingface.co/datasets/

[31] Ropedia. Xperience-10M: A large-scale egocentric multimodal dataset with structured 3D/4D annotations, 2026. URL https://huggingface.co/datasets/ropedia-ai/ xperience-10m. Dataset. [32] R. Hoque, P. Huang, D. J. Yoon, M. Sivapurapu, and J. Zhang. EgoDex: Learning dexterous manipulation from large-scale egocentric video. arXiv preprint arXiv:2505.11709, 2025. [33] R. Punamiya, S. Kareer, Z. Liu, J. Citron, R.-Z. Qiu, X. Cai, A. Gavryushin, J. Chen, D. Liconti, L. Y. Zhu, P. Aphiwetsa, B. Li, A. Cheluva, P. Kuppili, Y. Liu, D. Patel, A. Gao, H.-Y. Chung, R. Co, R. Zbizika, J. Liu, X. Xu, H. Xiong, G. Chen, S. Oliani, W. Xuan, C. Yang, X. Wang, J. Fort, R. Newcombe, J. Gao, J. Chong, G. Matsuda, A. Doriwala, M. Pollefeys, R. Katzschmann, X. Wang, S. Song, J. Hoffman, and D. Xu. EgoVerse: An egocentric human dataset for robot learning from around the world, 2026. URL https: //arxiv.org/abs/2604.07607. [34] X. Wang, K. Zhao, F. Liu, J. Wang, G. Zhao, X. Bao, Z. Zhu, Y. Zhang, and X. Wang. EgoVid5M: A large-scale video-action dataset for egocentric video generation, 2024. URL https: //arxiv.org/abs/2411.08380. [35] X. Chen, J. Guo, T. He, C. Zhang, P. Zhang, D. C. Yang, L. Zhao, and J. Bian. IGOR: Imagegoal representations are the atomic control units for foundation models in embodied AI. arXiv preprint arXiv:2411.00785, 2024. [36] S. Gao, S. Zhou, Y. Du, J. Zhang, and C. Gan. AdaWorld: Learning adaptable world models with latent actions. arXiv preprint arXiv:2503.18938, 2025. [37] S. Gao, W. Liang, K. Zheng, A. Malik, S. Ye, S. Yu, W.-C. Tseng, Y. Dong, K. Mo, C.-H. Lin, Q. Ma, S. Nah, L. Magne, J. Xiang, Y. Xie, R. Zheng, D. Niu, Y. L. Tan, K. R. Zentner, G. Kurian, S. Indupuru, P. Jannaty, J. Gu, J. Zhang, J. Malik, P. Abbeel, M.-Y. Liu, Y. Zhu, J. Jang, and L. Fan. DreamDojo: A generalist robot world model from large-scale human videos, 2026. URL https://arxiv.org/abs/2602.06949. [38] X. Chen, H. Wei, P. Zhang, C. Zhang, K. Wang, Y. Guo, R. Yang, Y. Wang, X. Xiao, L. Zhao, et al. Villa-X: Enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025. [39] C. Zhang, J. Wang, Z. Gao, Y. Su, T. Dai, C. Zhou, J. Lu, and Y. Tang. CLAP: Contrastive latent action pretraining for learning vision-language-action models from human videos. arXiv preprint arXiv:2601.04061, 2026. [40] P. Liu, Z. Xiang, M. Li, W. Li, J. Zhang, J. Sun, and J. Li. WLA3 : World latent action modeling for semantics, dynamics, and kinematics, 2026. URL https://arxiv.org/abs/ 2609.15870. [41] X. Bu, J. Lyu, F. Sun, R. Yang, Z. Ma, and W. Li. LAOF: Robust latent action learning with optical flow constraints. arXiv preprint arXiv:2511.16407, 2025.

[42] D. P. Kingma and M. Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014. [43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017. [44] T. Zhang, Z. Yuan, D. Chi, P. Liu, D. Li, K. Hu, L. Zhang, J. Nie, Z. Wei, Z. Chen, Y. Tang, J. Li, Z. Xiang, M. Li, T. Luo, H. Wan, A. Li, L. Zhai, Z. Zhan, X. Bai, J. Cai, P. Cao, K. Chen, S. Chen, Y. Dai, S. Di, Y. Gong, C. Gui, Y. Guo, P. Hao, Q. He, H. Huang, K. Huang, Z. Huang, S. Jin, Y. Jin, A. Li, D. Li, J. Li, R. Li, Y. Li, Y. Li, J. Liang, F. Liu, J. Long, M. Luo, X. Pan, H. Shen, X. Tian, D. Wang, S. Wang, J. Xiong, H. Xu, W. Xu, Z. Yu, H. Zhang, J. Zhang, L. Zhao, C. Zhou, N. Duan, Y. Zhuang, and L. Lin. JoyAI-RA 0.1: A foundation model for robotic autonomy, 2026. URL https://arxiv.org/abs/2604.20100. [45] Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. [46] T. Chen, Z. Chen, B. Chen, Z. Cai, Y. Liu, Z. Li, Q. Liang, X. Lin, Y. Ge, Z. Gu, et al. RoboTwin 2.0: A scalable data generator and benchmark with strong domain randomization for robust bimanual robotic manipulation. arXiv preprint arXiv:2506.18088, 2025. [47] S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems, 2024. [48] StarVLA Community. StarVLA: A lego-like codebase for vision-language-action model developing. arXiv preprint arXiv:2604.05014, 2026. [49] C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. C. Burchfiel, and S. Song. Diffusion Policy: Visuomotor Policy Learning via Action Diffusion. In Proceedings of Robotics: Science and Systems, Daegu, Republic of Korea, 2023. doi:10.15607/RSS.2023.XIX.026. [50] D. Nie, F. Chen, Q. Lv, J. Kuang, X. Li, X. Cao, and X. Cai. LARY: A latent action representation yielding benchmark for generalizable vision-to-action alignment. arXiv preprint arXiv:2604.11689, 2026.

Human-Video Data Construction

A.1

Source Composition

LAPT-Scale-LAM is trained on a 23.3K-hour egocentric human-video corpus collected from six egocentric human-video sources: Egocentric-10K [30], EgoDex [32], Ego4D [1] after EgoVid-style filtering [34], Xperience [31], EgoVerse [33], and self-collected human videos. The corpus is built by merging complementary sources rather than relying on a single dataset. Broad daily-activity datasets provide scene and object diversity, manipulation-focused datasets increase the density of hand-object interaction, and self-collected videos improve coverage of target-relevant activities and camera viewpoints. Table 8: Human-video source composition. Raw hours denote the source mixture used to form the 23.3K-hour LAM training corpus. The cleaned subset column denotes the portion retained after manipulation-focused filtering for Stage-1 policy pre-training. Source

Primary Content

Role in Corpus

Egocentric10K [30]

Filtered egocentric activity clips

EgoDex [32]

Dexterous egocentric manipulation Ego4D clips retained by EgoVid-style filtering Large-scale egocentric multimodal activity data Diverse egocentric robot-learning videos Egocentric human manipulation videos collected by us

Ego4D filtered by EgoVid [1, 34] Xperience [31] EgoVerse [33] Self-collected human videos

A.2

Raw Hours

Cleaned Stage-1 Hours

Additional LAM coverage; excluded from Stage-1 scaling because it lacks hand-pose annotations needed by the hand-motion baseline Fine-grained hand-object motion

10.0K

0.0K

0.7K

0.5K

Broad daily activity with stronger manipulation relevance Scene, object, and motion diversity

1.8K

0.8K

1.1K

0.4K

Cross-scene and cross-task diversity

1.2K

0.5K

Target-relevant activities and camera viewpoints

8.5K

2.9K

Filtering, Mixing, and Annotation

We first standardize all videos to a common decoding format, normalize frame rates and resolutions for training, remove clips with severe corruption or missing frames, and segment long videos into short temporal clips. We also remove near-duplicate clips after segmentation to avoid overrepresenting repeated scenes or repeated camera trajectories. For policy pre-training, we apply an additional manipulation-focused filtering stage. A clip is retained when it contains visible hand-object interaction and temporal visual change that can plausibly correspond to an action. We discard segments dominated by strong blur, extreme ego-camera motion, missing hands, long static intervals, severe occlusion, or weak manipulation content. This produces a 5.1K-hour cleaned subset used for the main Stage-1 policy pre-training run. The source mixture is constructed in two stages. First, all candidate videos are standardized and pooled into the 23.3K-hour corpus used to train LAPT-Scale-LAM. This stage favors coverage: even clips with moderate manipulation density can help the LAM learn generic egocentric visual dynamics, camera-relative motion, and object appearance variation. Second, we apply the stricter manipulation filter described above to build the 5.1K-hour Stage-1 policy pre-training subset. This stage favors supervision quality: clips are kept when they contain visible hands, object interaction, and meaningful temporal change. The 1K-, 3K-, and 5K-hour scaling subsets are sampled from the cleaned pool and are nested when possible: the 3K-hour subset contains the 1K-hour subset plus additional clips, and the 5K-hour subset contains the 3K-hour subset plus additional clips whenever the source manifests allow it. We also cap or rebalance very large sources during sampling so that scaling primarily increases the amount of action-like supervision rather than simply increasing the dominance of one source.

In practice, this preserves a mixture of household manipulation, dexterous-hand interaction, and general egocentric daily activity across all scales. For fair comparison with the hand-motion annotation baseline, the cleaned Stage-1 pool only includes sources where the same filtering and hand-pose-based pseudo-action pipeline can be applied. Egocentric-10K remains useful for broadening LAM training coverage, but it does not provide the hand-pose annotations required by the hand-motion baseline. We therefore exclude it from the cleaned Stage-1 scaling subsets, which keeps the comparison focused on the form of human-video supervision rather than source-specific annotation availability. We also audit the semantic diversity of the available human-video metadata. The fine-grained taxonomy identifies 12 scene categories, 18 task categories, 77 scene-task cells, 21 action labels, 54 object labels, and 272 action-object pairs. We use these statistics as a metadata diversity audit rather than as exact training-hour composition, because weakly labeled or unlabeled sources remain part of the large-scale video pool. After filtering, the frozen LAPT-Scale-LAM encoder annotates each retained clip with transitionlevel latent actions. For a clip x1:T , the encoder is applied to adjacent frames to produce posterior means z̄1:T −1 , which are used as deterministic Stage-1 policy targets. The latent annotations are stored together with the corresponding visual context, language or clip-level text when available, source metadata, and masks for valid temporal positions. Invalid transitions caused by decoding failure or clip boundaries are excluded from the Stage-1 loss. This annotation format lets the same policy training code consume human-video latent-action chunks in Stage 1 and executable robot action chunks in Stages 2–3.

Training and Implementation Details

B.1

LAM and Policy Training

The controlled LAM diagnostic variants are trained on the same human-video corpus with the same clip sampling and optimization protocol. LAPT-Scale-LAM encodes each video transition into a continuous latent action and reconstructs the next frame from the current visual context and latent action. The full model uses the reconstruction, LPIPS, DINO feature, optical-flow, and KL terms described in the main paper. The MSE-only diagnostic uses DreamDojo’s reconstruction-plus-KL training recipe on our data, omitting the LPIPS, DINO, and optical-flow alignment terms. In contrast, the DreamDojo baseline used for Stage-1 scaling uses its official pretrained LAM. It is not the same checkpoint as the controlled MSE-only diagnostic. The VQ-LAM baseline replaces the continuous latent action with a discrete codebook while keeping the transition prediction protocol fixed. The LAM uses a patch size of 16, model dimension 1024, 24 Transformer encoder blocks, 24 Transformer decoder blocks, and 16 attention heads. The latent action is parameterized as a 32dimensional continuous Gaussian posterior. The training objective uses MSE reconstruction as the base term, KL weight 1 × 10−6 , LPIPS weight 0.01, DINO feature weight 0.003, and optical-flow weight 3.3 × 10−5 . DINO alignment uses a frozen DINOv2 ViT-B/14 backbone, and flow alignment uses RAFT-large from the torchvision optical-flow models. The LAM training configuration also uses paired augmentation. For each paired sample, the configuration uses four repeats and resamples the latent action under augmentation. The color augmentation ranges are 0.3 for brightness, contrast, and saturation, 0.08 for hue, gamma in [0.7, 1.4], Gaussian noise standard deviation 0.035, blur probability 0.2, and minimum crop scale 0.9. LAPT-Scale-Policy uses the same VLM backbone and action expert across all experiments. Stage 1 trains the policy on LAM-annotated human-video clips, where the target is the continuous latent action produced by the frozen LAM encoder. Stage 2 continues training the same action expert on 250 hours of target-robot embodiment-alignment data, replacing the Stage-1 latent target with executable robot action chunks without introducing a separate action head or adapter. The action

expert supports up to 32 target dimensions; lower-dimensional targets are padded and excluded from the loss with a mask. Stage 3 further adapts the same policy on task-specific robot data. Across scaling experiments, the embodiment-alignment data, downstream task data, policy architecture, and training schedule are fixed; only the amount and source of human-video pre-training supervision are changed. B.2

Compute and Hyperparameters

We separate the computational cost into three parts: LAM training, offline human-video annotation, and policy training. LAM training is a one-time cost on the 23.3K-hour human-video corpus. After training, the frozen LAM encoder is used offline to annotate the cleaned 5.1K-hour subset and the nested scaling subsets. Policy training then consumes the cached latent-action annotations in Stage 1, so downstream scaling experiments do not require rerunning the LAM encoder except when changing the annotated human-video pool. Table 9 summarizes the implementation, training, and compute configuration used in our experiments. Table 9: Implementation, training, and compute details for LAPT-Scale-LAM training, offline annotation, and policy training. Component

Item

Value

Human-video corpus Human-video corpus Human-video corpus LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM LAM losses LAM augmentation LAM compute LAM compute LAM compute LAM annotation LAM annotation LAM annotation Policy Stage 1 Policy Stage 1 Policy Stage 2 Policy Stage 2 Policy Stage 3 Policy Policy Policy Policy Policy Policy compute Policy compute

LAM training data Cleaned Stage-1 subset Scaling subsets Latent action dimension Latent type Prediction target Visual losses Patch size / model dimension Encoder / decoder depth Attention heads Image resolution Input clip length / frame stride Optimizer / learning rate Weight decay / gradient clipping Batch size / training steps Max epochs Samples per epoch Precision / distributed strategy Checkpoint / log interval Padding / sampling strategy KL / LPIPS / DINO / flow weights Paired augmentation Hardware Training time Total GPU hours Annotated subset size Annotation throughput Total annotation cost Supervision Human-video scale Embodiment-alignment data Supervision Supervision Action expert capacity Action chunk horizon Optimizer / scheduler Learning-rate groups Gradient accumulation Hardware Detailed training schedule

23.3K hours 5.1K hours 1K, 3K, and 5K hours 32 Continuous Gaussian posterior Next-frame reconstruction MSE, LPIPS, DINO, optical-flow, KL 16 / 1024 24 / 24 blocks 16 320 × 240 2 frames / random 1–4-frame gap AdamW / 2.5 × 10−5 10−2 / 0.3 16 per GPU / 1,000,000 1000 10,000,000 bf16 / DDP 5000 / 1000 steps Repeat / manual 10−6 / 0.01 / 0.003 / 3.3 × 10−5 Enabled, 4 repeats, resample latent 32 H200 GPUs 154.3 hours at 1.8 steps/s 4,938 H200 GPU-hours 5.1K hours 3 video hours / H200-hour 1,700 H200 GPU-hours LAM posterior-mean latent-action chunks 1K / 3K / 5K / 5.1K hours 250 hours Target-robot action chunks Task-specific robot action chunks Up to 32 target dimensions 30 AdamW / cosine with minimum learning rate base / Qwen-VL interface / action model 1 32 H200 GPUs Tables 10 and 11

Table 10: Policy pre-training and embodiment-alignment configuration. Learning rates are reported for the base, Qwen-VL interface, and action-model parameter groups. Stage

Frozen modules

Batch size

Base LR

Qwen-VL LR Action LR Time

−5

Stage 1 None 1024 (32/GPU × 32) 2.5×10 Stage 2 Qwen-VL LM backbone 1024 (32/GPU × 32) 2.5×10−5 Both stages are reported at 100K training steps.

1×10−5 1×10−5

1×10−4 1×10−4

43h 50h

Table 11: Real-robot post-training configuration. All three tasks use full fine-tuning, task-specific robot demonstrations, and 30K training steps. Task

Batch size

Mouse Pad Headphones Cup

64 (8/GPU × 8) 64 (8/GPU × 8) 64 (8/GPU × 8)

Base LR −5

3×10 3×10−5 3×10−5

Qwen-VL LR

Action LR

−5

1×10 1×10−5 1×10−5

−4

1×10 1×10−4 1×10−4

Time 7h 7h 8h

Table 12: Implementation comparison of human-video supervision baselines. All policy baselines share the same VLA backbone, action expert, robot alignment data, and downstream training schedule. Method

Target / Latent Type

Training Signal

Used For

Hand-motion annotation

Reconstructed hand trajectory; explicit pseudo-action with hand pose Implicit transition action; latent action without hand pose Implicit transition action; discrete codebook without hand pose Implicit transition action; continuous Gaussian without hand pose Implicit transition action; continuous Gaussian without hand pose

Hand-pose trajectory fitting

Stage-1 policy scaling

Pixel-level future-frame reconstruction

Stage-1 policy scaling

Future-frame reconstruction with vector quantization

LAM diagnostic baseline

MSE future-frame reconstruction and KL

LAM diagnostic baseline

DreamDojo LAM

VQ-LAM

MSE-only LAM

LAPT-Scale-LAM

MSE, LPIPS, DINO, optical-flow, and Stage-1 policy KL alignment scaling and LAM diagnostics

Baseline Implementation Details

Table 12 summarizes the main implementation differences among the human-video supervision baselines. For fair comparison, all variants use the same cleaned human-video subsets, the same Stage-1 policy architecture, the same 250-hour target-robot embodiment-alignment data, and the same downstream post-training protocol. The comparison therefore isolates the form and quality of the supervision extracted from human videos rather than changes in robot data or policy capacity. For the explicit annotation baseline in Table 15, we convert egocentric human videos into action-like supervision through reconstructed hand motion. Specifically, this baseline treats the human hand as a dexterous end-effector and uses hand-pose trajectories as pseudo-action targets for human-video pretraining. It uses the same clip sampling, filtering, policy architecture, robot data, and downstream training schedule as LAPT-Scale-LAM. This comparison is intended to isolate the effect of replacing explicit hand-motion supervision with LAM-derived implicit actions. The hand-motion baseline depends on explicit hand reconstruction, which can produce noisy pseudo-actions under occlusion, motion blur, unusual viewpoints, or contact-rich manipulation. The LAM baselines avoid explicit hand-pose estimation by learning transition variables directly from video. DreamDojo LAM and MSE-only LAM primarily test reconstruction-driven latent actions, 17

VQ-LAM tests whether a discrete bottleneck is sufficient for egocentric manipulation, and LAPTScale-LAM tests whether continuous latents with perceptual, semantic, and motion-aware alignment produce better action-like supervision.

Evaluation Protocol Details

For all downstream robot evaluations, we report task success rate. A rollout is counted as successful only when the task-specific goal condition is satisfied within the evaluation horizon; otherwise it is counted as a failure. For simulation benchmarks, we follow the benchmark-provided success detectors and episode termination rules. For real-robot tasks, success is judged by the final task state after execution and uses the same criteria for all compared methods. D.1

Robot-Data Quantities and Evaluation Accounting

The 250-hour Stage-2 corpus is heterogeneous embodiment-alignment data, comprising 162,767 episodes and 26.7M frames rather than per-task data. For Stage 3, we use 50 demonstrations per original real-robot task, 200 per added hard task, 50 clean plus 500 randomized demonstrations per RoboTwin task, and 1,000 demonstrations per RoboCasa task. The original tasks use 20 evaluation trials per task. The four harder tasks use 20 trials per task per domain, or 320 trials per method across the seen domain and three shifted domains. D.2

Simulation Benchmarks

RoboTwin 2.0 evaluation follows the benchmark Easy and Hard settings. For RoboTwin posttraining, each task uses 50 clean demonstrations and 500 heavily randomized demonstrations. The Easy setting evaluates policies under the standard benchmark initialization distribution, while the Hard setting introduces stronger variation in object poses and scene layouts. We report the average success rate across the benchmark task suite for each setting; the main LAPT-Scale policy reaches 90.0% in Easy and 89.3% in Hard. All compared methods are evaluated with the same task split, observation interface, action interface, and success definition. RoboCasa-GR1 evaluation uses the tabletop manipulation setting with 24 tasks and 50 rollouts per task. We follow the benchmark-provided simulator reset distribution and task success checks, and report the average success rate across all RoboCasa rollouts. The reported RoboCasa-GR1 number averages over the evaluated task instances rather than selecting the best task-specific checkpoint; the main LAPT-Scale policy reaches 64.8% average success. D.3

Real-Robot Tasks

Real-robot evaluation is conducted on an AgiBot G1 robot after target-robot embodiment alignment. We evaluate three tasks: Mouse Pad, Headphones, and Cup. Each task is evaluated over 20 trials per method. Between trials, the scene is reset to the task-specific initial distribution, and all methods are evaluated under the same reset and success criteria. A Mouse Pad trial succeeds if the robot completes the requested manipulation of the mouse pad; a Headphones trial succeeds if the headphones are manipulated to the target state; and a Cup trial succeeds if the cup task reaches the specified final configuration. Table 13: Summary of downstream evaluation protocols. Benchmark

Setting

Metric

Evaluation scope

Success Criterion

RoboTwin 2.0 Easy Success rate Official task suite Benchmark detector RoboTwin 2.0 Hard Success rate Official task suite Benchmark detector RoboCasa-GR1 Tabletop Success rate 24 tasks × 50 rollouts Benchmark detector Real robot Mouse Pad Success rate 20 Task completion Real robot Headphones Success rate 20 Task completion Real robot Cup Success rate 20 Task completion

Additional Real-Robot Scaling Results

Table 14: Stage-1 scaling on the original real-robot tasks. Each entry is an average success rate (%) over three tasks with 20 trials per task. The shared 0-hour column is the same robot-only control. Stage-1 supervision

0 hours

1K hours

3K hours

5K hours

Hand motion DreamDojo LAM LAPT-Scale-LAM

55.0 55.0 55.0

58.3 61.7 63.3

75.0 75.0 80.0

55.0 76.7 85.0

Table 15: Per-task breakdown for the real-robot scaling study. Human-video scale denotes the amount of annotated human video used for VLA pre-training; all robot data is held fixed. Each task is evaluated over 20 trials. Human Supervision

Human Video

Mouse Pad

Headphones

Cup Task

Avg.

1K hours 3K hours 5K hours 1K hours 3K hours 5K hours 1K hours 3K hours 5K hours

45.0 80.0 75.0 60.0 80.0 65.0 50.0 70.0 85.0

80.0 45.0 50.0 55.0 85.0 75.0 70.0 75.0 95.0

50.0 100.0 40.0 70.0 60.0 90.0 70.0 95.0 75.0

58.3 75.0 55.0 61.7 75.0 76.7 63.3 80.0 85.0

Hand-motion annotation Hand-motion annotation Hand-motion annotation DreamDojo LAM DreamDojo LAM DreamDojo LAM LAPT-Scale-LAM LAPT-Scale-LAM LAPT-Scale-LAM E.1

Stage-1 Removal Control

The robot-only control removes Stage 1 while retaining the same initialization, 250-hour embodiment-alignment corpus, Stage-2/3 data, and evaluation protocol. The compute-matched control extends Stage-2 training to match the training compute budget. Unlike the Diffusion Policy comparison, Stage-1 removal tests human-video supervision within the same policy architecture. Table 16: Stage-1 removal and compute-matched controls on real-robot tasks. Stage-1 supervision

Stage-2 training

Average success (%)

None (robot-only) None (robot-only) LAPT-Scale, 5K hours

Standard Extended, compute-matched Standard

55.0 53.3 85.0

Additional LAM Diagnostics

F.1

Reconstruction Protocols

In addition to the single-step reconstruction results reported in the main paper, we evaluate each LAM variant under two rollout protocols. The first is a teacher-forced next-frame reconstruction protocol: for every time step, the model receives the ground-truth current frame xt and predicts x̂t+1 . This measures whether the latent action extracted from a local transition preserves enough information to reconstruct the next visual state without accumulating temporal errors. The second is an autoregressive rollout protocol: the model receives the ground-truth first frame x1 , predicts x̂2 , and then conditions on its own prediction to generate subsequent frames. This evaluates temporal stability and whether reconstruction errors compound over multiple latent-action steps. For this quantitative comparison, we include LAPA [5] and VILLA-X [38] as additional latent-action model baselines. LAPA represents a discrete latent-action annotation approach, while VILLA-X represents a video latent-action baseline trained for egocentric transition prediction. Together with MSE-only LAM, these baselines test whether the gains come from the LAPT-Scale alignment design rather than only from using a latent transition model.

The metrics in Table 17 are computed over 3500 frames sampled from a randomly held-out 100hour evaluation split of our mixed human-video dataset. Qualitative examples for both protocols are shown in Figs. 7 and 8. Table 17: LAM reconstruction comparison under teacher-forced and autoregressive rollout protocols. Teacher-forced evaluation conditions each prediction on the ground-truth previous frame, while autoregressive evaluation starts from the ground-truth first frame and then conditions on the model’s previous prediction. Higher PSNR and SSIM are better; lower LPIPS is better. Protocol

LAM Variant

PSNR ↑ SSIM ↑ LPIPS ↓

Teacher-forced Teacher-forced Teacher-forced Teacher-forced

LAPA VILLA-X MSE-only LAM LAPT-Scale-LAM

18.3416 26.9595 33.9844 34.4217

0.6741 0.7973 0.9234 0.9326

0.3443 0.1685 0.0769 0.0546

Autoregressive Autoregressive Autoregressive Autoregressive

LAPA VILLA-X MSE-only LAM LAPT-Scale-LAM

15.8607 22.5951 28.3290 28.8291

0.5551 0.7008 0.8426 0.8510

0.4593 0.3228 0.1768 0.1347

Figure 7: Teacher-forced LAM reconstruction comparison. Each prediction conditions on the ground-truth previous frame. LAPA and VILLA-X show weaker local transition fidelity, MSEonly LAM is smoother around hand contacts, and LAPT-Scale-LAM preserves sharper hand-object structure. These two protocols expose the differences progressively. In the teacher-forced setting, LAPA and VILLA-X already show weak local transition fidelity: hand-object boundaries are less stable and the predicted motion is often visually inconsistent with the ground-truth next frame. MSE-only LAM is substantially stronger under teacher forcing because every prediction is conditioned on a clean

ground-truth frame, but it still tends to produce smoother hands and object contacts. LAPT-ScaleLAM gives the sharpest teacher-forced reconstructions, while DreamDojo-style reconstruction remains slightly blurrier around the moving hand and contacted objects. Fig. 7 illustrates the easier teacher-forced case. Because every prediction receives a clean previous frame, the comparison mainly reflects how well each latent action preserves the local transition. The strongest differences appear around the hand and the object being manipulated, where reconstruction-only objectives tend to smooth contact boundaries.

Figure 8: Autoregressive LAM rollout comparison. Later predictions condition on the model’s own generated frames. Error accumulation makes the differences more visible: MSE-only LAM gradually loses fine hand structure, while LAPT-Scale-LAM preserves a visible hand region and more coherent short-horizon object motion. The autoregressive setting makes this gap much more visible. Once each model must condition on its own previous prediction, small contact and hand-position errors compound over time. LAPA and VILLA-X drift quickly, while MSE-only LAM gradually loses fine hand structure; by the final generated frames, the hand is often no longer clearly recognizable. LAPT-Scale-LAM remains more stable under the same rollout protocol, preserving a visible hand region and coherent short-horizon object motion for longer. This supports the view that the additional perceptual, semantic, and motion alignment terms improve temporal latent-action structure rather than only single-step reconstruction quality. Fig. 8 evaluates the same models under a stricter rollout protocol. Here, the model must reuse its own generated frames, so small hand-position errors and blurred contacts become part of the next input. This is why the gap between MSE-only LAM and LAPT-Scale-LAM is more visible in later frames than in the teacher-forced comparison.

F.2

Similar Motions Across Different Scenes

To test whether the learned latent action is tied to visual appearance or to transition-level motion, we compare clips that contain similar short-horizon actions in different scenes. For each query transition, we retrieve nearest neighbors in the LAPT-Scale-LAM latent-action space while excluding clips from the same source video. This produces groups where the background, object identity, lighting, and hand appearance can differ substantially, but the short-horizon motion pattern is similar. For the cluster visualization, we sample 50K motion-filtered windows from 2.36M candidate windows across 11,981 episodes. Each window contains 16 frames with stride 16. We retain windows above the 70th percentile of the motion score (threshold 4.805), reduce latent-action features to 64 PCA dimensions (explained-variance sum 0.7756), and cluster the resulting features into 50 groups. The selected clusters in Fig. 9 illustrate repeated hand-object motion patterns across visually different clips.

Figure 9: Latent-action clusters with shared motion patterns across different data sources. Each panel shows examples grouped by the learned latent-action representation. Clusters are computed from 50K motion-filtered 16-frame windows after PCA reduction to 64 dimensions and 50way clustering. Although the visual scenes and objects vary, samples within a cluster exhibit similar short-horizon motion patterns, suggesting that LAPT-Scale-LAM captures transition-level structure beyond dataset-specific appearance. This analysis complements the latent-action replacement experiment reported in the main paper. Nearest-neighbor retrieval asks whether similar motions naturally cluster together across scenes, while replacement asks whether a latent action inferred from one scene can change the rollout in another scene without copying the donor appearance. Together, these diagnostics support the inter-

pretation that the LAM token encodes how the scene changes rather than only what the scene looks like. The selected clusters in Fig. 9 are not meant to show near-duplicate frames. Instead, they show that clips with different backgrounds, objects, and hand appearances can still be close in latentaction space when their short-horizon motion is similar. This is the desired behavior for Stage1 supervision: the policy should learn reusable transition patterns rather than memorize sourcespecific visual appearance.

Stage-1 Latent Rollout Diagnostics

To further diagnose Stage-1 human-video pre-training, we evaluate whether the learned latent-action space is meaningful enough for decoder rollout and whether the VLA policy can predict actionrelevant latent variables from context. We compare two rollout settings: using the ground-truth latent action encoded by LAPT-Scale-LAM and using the latent action sampled from the Stage-1 VLA policy. The ground-truth-latent rollout tests whether the compressed latent space preserves enough transition information for decoding, while the VLA rollout tests whether Stage-1 pre-training learns to predict latent actions that still produce coherent human motion. Table 18: Stage-1 latent-action rollout quality on human videos. Higher PSNR and SSIM are better; lower LPIPS is better. Rollout Target

PSNR ↑

SSIM ↑

LPIPS ↓

VLA-predicted latent rollout Ground-truth latent rollout

25.5371 30.7705

0.8137 0.8982

0.3033 0.2271

Table 18 shows that ground-truth latent actions can reconstruct future visual transitions with strong fidelity, indicating that the compressed LAM latent space retains meaningful motion information. More importantly, the VLA-predicted latent rollout also achieves reasonable reconstruction quality, showing that Stage-1 pre-training learns action-related latent structure rather than only static appearance cues. The two examples below visualize this diagnostic at different manipulation scales. The dice example emphasizes short, fast hand-object motion, while the sandwich example contains longer contact sequences and larger appearance changes. In both cases, the Stage-1 policy-predicted latents produce rollouts that preserve the dominant motion pattern even though they are not used as executable robot actions.

Gather the four dice with one hand and roll them on the brown background while sitting.

Figure 10: Stage-1 latent-action rollout on human videos: dice rolling. Frames are sampled every six video frames and arranged from left to right within each row. The first row shows the groundtruth video, the second row shows decoder rollout using the ground-truth latent action encoded by LAPT-Scale-LAM, and the third row shows decoder rollout using the latent action predicted by the Stage-1 VLA policy.

The sandwich sequence is visually harder because multiple ingredients and hand contacts change across the clip. The rollout remains useful as a diagnostic because it tests whether predicted latent actions encode the transition tendency rather than merely reconstructing a static scene.

Make a sandwich using breads, cheese, onion, ketchup, mayo, tomato, lettuce, and ham while sitting at a table.

Figure 11: Stage-1 latent-action rollout on human videos: sandwich making. Frames are sampled every six video frames and arranged from left to right within each row. The Stage-1 predictions produce rollouts with motion patterns close to the ground-truth-latent rollout and the observed video, indicating that human-video pre-training learns meaningful human motion representations rather than only static visual appearance.

Additional Robustness Evaluation

H.1

Harder Real-Robot Tasks

To test whether the gains extend beyond the three original tabletop tasks, we add paper-ball disposal, slender-pen grasping, small eraser grasping under self-occlusion, and multi-stage book stowing. The first task requires disposal into a target receptacle; pen and eraser require precision grasping despite small contact regions and partial occlusion; book stowing requires a controlled edge push, grasp, vertical reorientation, and shelving. We jointly train one policy with 200 demonstrations per task. The same downstream demonstrations are used to post-train π0.5 from its official checkpoint. H.2

Generalization to Unseen Conditions

We evaluate the four harder tasks under the training distribution and three unseen shifts: object position, illumination, and background/tablecloth appearance. Each entry averages 20 trials per task and domain. LAPT-Scale and π0.5 are tied in-distribution, whereas LAPT-Scale degrades less under all three shifts. Origin

Spatial

Illumination

Background

Figure 12: Original evaluation domain and the three unseen generalization shifts. Table 19: Four-task real-robot generalization (20 trials per task per domain). Method

Seen Position Illumination Background Average

Diffusion Policy 60.0 π0.5 85.0 LAPT-Scale 85.0