Latent Actions from Factorized Transition Effects under Agent Ambiguity
Heejeong Nam 1 Chandradithya S Jonnalagadda 1 Harshit Aggarwal 1 Eric Xu 1 Randall Balestriero 1 https://hazel-heejeong-nam.github.io/LAM/
arXiv:2606.30544v1 [cs.AI] 29 Jun 2026
Abstract Latent Action Models (LAMs) learn action-like proxies from observation transitions. However, in multi-object or distractor-rich scenes, these visual effects mix agent motion with distractors, camera dynamics, and background changes, making the underlying action source ambiguous without supervision. Structuring this mixture as reusable transition effects provides an intermediate representation from which action-like latents can be more robustly formed. We introduce Observed Transition Factorization (OTF), which decomposes each transition into a sparse set of observedtransition primitives. Using these primitives as the transition interface, we propose OTF-LAM, which abstracts motion primitives into actionlike latents within the standard inverse–forward dynamics framework, and OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINOv2 representation space. Empirically, OTF primitives transfer zero-shot across controlled carrier and morphology shifts, showing reusability. Furthermore, downstream policy learning results match or outperform baselines under complex transition ambiguity.
Figure 1. Motivation for agent ambiguity in observation-only latent action learning. Unlike action-conditioned world models, which receive the actor and action as part of the input, LAMs infer actionlike latents only from visual transitions.
(LAMs) (Schmidt & Jiang, 2024; Ye et al., 2025) have been proposed to learn action-like proxies directly from observation transitions. A typical LAM encodes an observation pair (xt , xt+1 ) into a latent action and uses it to predict xt+1 from xt . This formulation is appealing because it removes the need for action annotations, but it also changes what the model can identify. However, a LAM does not observe pure actions themselves. It observes the visual effects that actions, dynamics, rendering, and other transition sources produce. This distinction is often hidden in controlled settings where the agent is visually dominant and non-agent elements are static (Yu et al., 2020; Mandlekar et al., 2021). In more general scenes, however, a single transition may mix agent motion with passive objects, articulated parts, camera/background dynamics, and distractors (Nikulin et al., 2025; Wang et al., 2025; Adnan et al., 2026; Klepach et al., 2026). From the model’s perspective, all of these changes arrive through the same channel: the pixels at time t and t + 1. Without additional supervision or inductive biases, the model has no direct basis for deciding which observed changes should be attributed to the controlled agent, or
1. Introduction World models (Ha & Schmidhuber, 2018) for planning and control have been evaluated in settings where the actor and the action space are known, such as a robotic manipulator (Hansen et al., 2024; Hafner et al., 2025). In this regime, the model predicts the next state from the current state, and can therefore, separate the controlled source of variation from autonomous background dynamics or moving distractors at least implicitly. Recently, motivated by the fact that most online videos do not provide frame-level action labels or proprioceptive signals, Latent Action Models 1 Brown University. Correspondence to: Heejeong Nam <[email protected]>.
Preprint. June 30, 2026.
1
Latent Actions from Factorized Transition Effects under Agent Ambiguity e.g., joint torque
which changes should be selected as relevant to the desired action abstraction, as depicted in Figure 1. A natural response to this ambiguity is to treat it as an object selection problem (Wang et al., 2025; Adnan et al., 2026; Klepach et al., 2026), identifying the controllable entity in order to learn its latent action. However, relying on segmentation or object-centric priors limits scalability and imposes rigid assumptions. More importantly, objects are not always the appropriate unit of abstraction, as motion effects can manifest within, across, or independently of strict object boundaries. Recognizing these limitations, we step back from the standard operational view of LAMs as simple transition bottlenecks (Schmidt & Jiang, 2024) and ask what can be identified from observation-only transitions, and what latent actions should represent under agent ambiguity.
e.g., optical flow
e.g., proprioceptive signals
forward generation
forward generation
inverse learning
inverse learning
(a) Action / Cause
Observed in pixels, e.g.,
(b) State Change
(c) Visual Change
Figure 2. From causal generation to inverse inference. Blue arrows denote the causal pathway from action to visual change, while red arrows denote the inverse problem faced by observation-only latent action learning.
2. From Actions to Observed Effects
We interpret the pathway from action to observation at three levels as in Figure 2: the true underlying action induces a physical or environment state change, which then produces visual transition effects in observation space. Under observation-only learning, the most reliable target is not necessarily the true action itself, but the structure of these observed transition effects. We therefore take a bottom-up approach to latent action learning under ambiguity. We first introduce Observed Transition Factorization (OTF), which represents each transition as a sparse composition of reusable observed-transition primitives. Rather than compressing all changes into a monolithic embedding, OTF decomposes the transition into localized observed effects using a compositional VQ-style codebook. The resulting representation records which transition effects occur and where they are expressed. Using this factorized transition interface, we instantiate OTF-LAM, which forms state-aware action-like latents within the standard inverse–forward dynamics framework. A frozen OTF module first extracts reusable observed-transition primitives, and a state-aware aggregator abstracts these primitives into a compact latent action for prediction. We also introduce OTF-LAM-Dino, a decoder-free variant that predicts future states in a frozen DINO representation space, combining a reusable transition vocabulary with a reusable visual state representation. Empirically, OTF primitives transfer zero-shot across heldout visual carriers and cross-morphology shifts on Moving MNIST (Srivastava et al., 2015) and DCS (Stone et al., 2021), indicating reusable transition structure rather than environment- or embodiment-specific templates. For downstream control, we map learned latent actions to true actions using a small action-labeled dataset. The resulting latent action interfaces match or outperform LAM baselines under complex transition ambiguity, demonstrating the effectiveness of factorized observed transitions for policy learning. OTF-LAM-Dino further improves over decoder-based OTFLAM, showing that reusable transition vocabularies complement frozen visual state representations.
LAMs trained from observation-only transitions do not recover actions directly, but representations of their visible consequences after being mediated by state, embodiment, dynamics, and rendering. As a result, the learned latent space is not necessarily action-equivalent. 2.1. The Forward Generative Chain We view a transition as a chain of three stages: at , ξt | {z }
trainsition sources
F
E −−→
∆s |{z}t
R
E −−→
physical state change
∆x |{z}t
visual transition effect
as in Figure 2. Here, at denotes the controlled action, while ξt denotes uncontrolled transition sources such as camera motion, distractor motion, background dynamics, or passive environment changes. The environment E includes the embodiment, dynamics, camera, and other environmentspecific factors. Let F E be the state transition function and RE be the observation or rendering map. The physical state change may be written as ∆st = FE (st , at , ξt ), while the visual transition effect may be written as ∆xt = RE (st , ∆st , ξt ). Equivalently, the observed transition is generated by the composition ∆xt = RE st , FE (st , at , ξt ), ξt . (1) Intuition. This setup clarifies why a latent action cannot generally recover the true action. The observed transition is generated by Eq. 1. For fixed observations, the map from at to ∆xt is generally not injective, and ∆xt may also contain changes not caused by at at all. Therefore, an observation-trained latent action can at best identify an equivalence class of causes and observed effects. 2
Latent Actions from Factorized Transition Effects under Agent Ambiguity
2.2. Observed Effects as a Transferable Intermediate Actions and physical state changes are often environmentspecific. Actions depend on the control interface, such as torques, joint commands, or button inputs, while physical changes depend on state, embodiment, contact, and dynamics. Thus, neither level provides an obvious reusable primitive across heterogeneous embodiments. Observed visual effects provide a weaker but more transferable abstraction. They describe how change appears in observation space, such as local displacement, edge shifts, rotation-like residuals, contact deformation, or background drift. For instance, Mario moves right
Figure 3. Observed-transition vocabulary learning with a VQVAE. Motion-centered transition inputs are patchified, quantized into a shared codebook of reusable observed-transition primitives, and decoded with reference-frame conditioning to reconstruct the observed-transition.
Kirby moves right Patchwise vector quantization. Given ot ∈ RC×H×W , we partition it into P non-overlapping spatial patches and encode each patch with a shallow MLP into a token ft,i ∈ RD . Each token is then assigned to its nearest codebook entry using top-1 nearest-neighbor quantization
may correspond to different bodies, controls, and dynamics, yet both can induce similar rightward displacement patterns in the observation space. This motivates using observed effects as the intermediate target for latent action learning, rather than requiring the representation to immediately identify the controlled source.
∗ kt,i = arg
3. From Observed Effects to Latent Actions
and
∥ft,i − c(k) ∥22 ,
∗
qt,i = c(kt,i ) . (3)
Note that this quantization should be interpreted as discretizing local observed effects rather than pure motion like optical flow. Even when ot suppresses object-level semantics, patch tokens may retain carrier-specific cues such as texture, contrast, local geometry, and occlusion.
Latent action learning can be viewed as an inverse problem of the causal process: construct latent actions from observations. Unlike a standard LAM (Schmidt & Jiang, 2024) learning zt = q(xt , xt+1 ) directly, we decompose the inverse pathway into observed-transition factorization followed by state-aware abstraction: Et = qϕ (xt , xt+1 ),
min k∈{1,...,K}
Intuition. Patch size controls the locality of the observed-transition primitives. Smaller patches reduce appearance and object-identity leakage, but may not contain enough context to infer a stable motion effect. Larger patches provide richer motion evidence, but can mix multiple transition sources and become tied to local carrier appearance. A more detailed discussion is provided in Appendix E.2.
ztact = hψ (Et , xt ), (2)
where Et denotes a structured set of observed-transition factors, and ztact is the compact latent action used for prediction. Our framework is trained in two stages. In Sec.3.1, we pretrain the Observed-Transition Factorizer qϕ with a motion-space autoencoding objective. In Sec.3.2, we freeze qϕ and train OTF-LAM on top of the extracted factors.
From the patch-level assignments, we derive for each code (k) k an occupancy map Mt , denoting the spatial support or occupancy of code k in the transition, and an activation (k) strength wt . Together these form the transition (k) (k) K Et = {et,k }k=1 , et,k = c(k) , Mt , wt . (4)
3.1. Observed Transition Factorization (OTF) We construct the motion input ot as a frame-level transition signal over a temporal stride τ , chosen to emphasize observable change while reducing dependence on RGB appearance, object identity, and environment-specific semantics. Unless otherwise specified, we construct ot by applying a gradient transform to both frames and subtracting the transformed current frame from the transformed future frame. The vocabulary stage does not infer latent actions or assign changes to action-level causes. It instead learns a shared codebook of K reusable observed-transition primitives for representing ot , capturing recurring visual change patterns such as local displacement, edge shifts, rotation-like residuals, contact deformation, background drift, or articulated part motion.
Thus, Et records which observed-transition primitives are active in an observation transition, where they appear, and how strongly they are expressed, without yet determining which of them are action-relevant. In later stages, this observed-transition vocabulary can be compressed or selected into action-like latents for downstream LAM training. Training. Our training scheme follows Figure 3. We instantiate the observed-transition factorizer as a VQ-VAE trained to reconstruct the motion observation ot rather than the future RGB frame. Each transition factor et,k ∈ Et 3
quantized local motion
Latent Actions from Factorized Transition Effects under Agent Ambiguity frame t
frame t
current state frame t+1
FDM
𝑟!
𝛼! 𝜎(𝑔! (,))
frame t+1
IDM
IDM
frame t+1
𝐸!
frame t+1
𝑟! FDM
current state
𝑃!
⨂
Action Abstraction Module (b) LAM with Factorized Motion
(a) LAPO
Figure 4. Latent-action learning on top of the pretrained observed-transition vocabulary. Given the extracted observed-transition factors Et and the current frame xt , the model builds state-aware factor tokens, scores them with a relevance gate, aggregates the selected factors into a compact action-like latent ztact , and uses a reference-conditioned decoder to predict the next observation. "
"
"
IDM
(𝑐 , 𝑀# , 𝜔# , 𝑥# ) contains the informationmotion associated with code k.Γ!We map Details of architecture and training schemes are provided in Environment-agnostic atoms each active factor to an aggregated factor embedding Appendix A. Remark 1 (Local Quantization vs. Discrete Latent Acht,k = ρη (et,k ) tions.). Recent work (Garrido et al., 2026a) suggests that Γ! ( , 𝑥# ) , where ρη is a small shared network and k indexes a code using vector quantization as the final latent action bottleneck observed in the current transition. Let It denote the set of can be too restrictive for complex in-the-wild videos. OTF active codes. The resulting set of factor embeddings is uses quantization differently. The codebook represents local observed-transition primitives, while the final action-like set Ht = {ht,k }k∈It . latent is obtained by aggregating these primitives into a continuous representation. Thus, the discrete bottleneck is used To preserve spatial support, we place these embeddings back to structure reusable local effects, not to represent the full onto the patch grid using the occupancy maps action as a single code. K X (k) Ht (u, v) = Mt (u, v) ht,k . 3.2. OTF-LAM k=1
The decoder then combines the spatial factor map Ht with the current frame xt , which provides appearance and support information, to reconstruct the motion observation
After learning the OTF module, we freeze the factorizer and train a LAM on the extracted factor set Et , as shown in Fig. 4. Our goal is to learn an action-like latent
ôt = Dη (Ht , xt ).
ztact = hψ (Et , xt )
This encourages the codebook to represent local transition effects rather than object-specific appearance templates. OTF is trained with a motion reconstruction objective, standard VQ losses, and an orthogonality regularizer:
by selecting or compressing observed-transition factors for next-frame prediction. Action abstraction. We first convert each factor in Et into a state-aware factor token
Lvocab = Lrec + λ1 Lcode + λ2 Lcommit + λ3 Lorth ,
(k)
(k)
rt,k = Γθ (c(k) , Mt , wt , xt ),
where Lrec = ∥ôt − ot ∥22 , Lcode =
P X
where Γθ embeds the code identity, occupancy, activation strength, and current-frame context. A gate network then assigns each factor a relevance weight αt,k = σ(gθ (rt,k )) ∈ [0, 1], with inactive codes masked out. The factor-level latent is obtained by a normalized gated average
(5)
∥sg(ft,i ) − qt,i ∥2
2
(6)
2
(7)
i=1
Lcommit =
P X
∥ft,i − sg(qt,i )∥2 ,
PK fac k=1 αt,k rt,k zt = PK . k=1 αt,k + ϵ
i=1
Lorth =
X
′
ĉ(k)⊤ ĉ(k )
2
.
(8)
Finally, a linear projection produces the action-like latent ztact = Pθ (ztfac ). This module does not assume that the
′
k,k ∈It k̸=k′
4
Latent Actions from Factorized Transition Effects under Agent Ambiguity
predictor
> ℒ!
>
Action Abstraction Module
frozen DINOv2
frozen DINOv2
frozen IDM
Figure 6. Policy Training. The latent motion space is distilled into a policy via behavioral cloning (left). An action decoder then maps these primitives to ground-truth actions at , utilizing the pixel frame to associate motions with specific objects (right).
motion input
frame t
environment-specific. OTF-LAM-Dino pushes this separafrozen object-centric tion further by replacing pixel-space prediction with preencoder diction in a frozen DINOv2 representation space. In this variant, both the observed-transition vocabulary and the visual state encoder are fixed and reusable, leaving the learned model to focus on abstracting transition information into ztact and predicting its effect in representation space. OTFLAM appears architecturally similar to the concurrent work TDV (Daithankar et al., 2026), the two methods are fundamentally different in purpose: TDV aims to learn a strong encoder, or scene representation, whereas our goal is to learn latent actions. Appendix C describes details of OTFLAM-Dino architecture and training objective.
frame t+1
Figure 5. OTF-LAM-Dino uses frozen DINOv2 as an encoder and frozen OTF module as a motion primitive extractor.
factorizer has identified the controlled agent. It only learns which observed-transition factors are useful for forming a compact latent for prediction. Forward dynamics. We predict the next observation from the current frame and the latent action x̂t+τ = pθ (xt , ztact ). Features from xt provide appearance and layout information, while ztact modulates the decoder to provide transition information. The decoder may predict either the next frame directly or a residual update to the current frame, we use residual prediction by default.
3.4. Policy Learning with Latent Actions The final step is to connect the learned latent action space back to real actions. Since the visual transition alone is not invertible to the true action, we use a small amount of action supervision to resolve this inverse ambiguity. As described in Figure 6, we first distill the learned latent action space into a policy π(ztact | xt ) via behavioral cloning. We then train an action decoder on a small action-labeled dataset to map the predicted latent action to the true environment action. The decoder also receives the current observation xt , allowing it to associate latent motion with the relevant object or state context. In our experiments, this supervised stage uses 32 action-labeled trajectories for cheetah-run and walker-run from DCS (Stone et al., 2021).
Training. A full latent action model is trained with a nextframe prediction loss 2
LOTF−LAM = ∥x̂t+τ − xt+τ ∥2 . The observed-transition vocabulary encoder and codebook remain frozen, and gradients are applied only to the action abstraction module and the forward dynamics model. Architectural details of the factor token construction, gating network, and decoder are provided in Appendix B. 3.3. OTF-LAM-Dino We explore a decoder-free JEPA-style variant, OTF-LAMDino. As described in Figure 5, we map both the current and future observations into a latent representation space using a frozen DINOv2 encoder (Oquab et al., 2024), and train the model to predict the future representation with ztact similar as OTF-LAM.
4. Experiments 4.1. Dataset We mainly evaluate our method on the Distracting Control Suite (DCS) (Stone et al., 2021), where we focus on cheetah-run and walker-run environment, which combine articulated embodied motion with complex visual distractors in the background. This dataset is challenging
OTF-LAM separates reusable transition primitives from the image decoder, but the decoder itself can still become 5
Latent Actions from Factorized Transition Effects under Agent Ambiguity
because the visual transition contains both agent-induced motion and distractor-induced changes, making it a natural testbed for evaluating whether a latent action model can separate reusable transition structure from environmentspecific visual variation. Trajectories are collected using the pretrained expert policies from Nikulin et al. (2025). For each environment, we collect 2000 trajectories, corresponding to roughly 2M transitions. We use 1000 trajectories to train the latent action models, and reserve the remaining trajectories for downstream training and evaluation.
Figure 7. Example of learned OTF codes assigned to the motion input. More visualization results are in Appendix D.2
Moving MNIST (Srivastava et al., 2015). We further introduce controlled Moving MNIST benchmark, where digit identity and observed-transition primitive can be independently specified. This setting serves as a clean diagnostic environment for testing whether the learned codebook captures reusable motion/effect primitives rather than digitspecific visual templates. In particular, we use held-out digit classes to evaluate transfer across visual carriers while keeping the underlying observed-transition primitives shared. We construct a training set of 3,000 sequences containing digits from {0, 1, 2, 3, 4} and a test set of 300 sequences containing digits from {5, 6, 7, 8, 9}. Figure 8. Cross-morphology transfer in DCS using only the pretrained observed-transition factorizer. The vocabulary is trained on walker-run and evaluated zero-shot on cheetah-run.
4.2. Baseline Latent Action Models We evaluate three existing LAM baselines and two OTFLAM-specific variants. The OTF-LAM-specific variants are designed to isolate the contributions of motion processing and patchwise latent quantization.
4.3. Reusability of Latent Action Primitives In this section, we evaluate the observed-transition primitives learned by the pretrained OTF tokenizer. A useful motion vocabulary should transfer across visual carriers and embodied morphologies, rather than overfitting to a single agent. To test this, the vocabulary is trained on walker-run and applied zero-shot to cheetah-run environment, with the codebook kept fixed. We also conduct a similar transfer study on Moving MNIST (Srivastava et al., 2015), trained on digits 0,1,2,3,4 and evaluated zero-shot on held-out digits 5,6,7,8,9 with the same observed-transition primitives. To evaluate transferability, we compare against a monolithic VQ-VAE that encodes each transition into a single global latent bottleneck with a comparable latent budget. We additionally vary the motion input ot used for vocabulary learning. In particular, we compare grayscale frames with a Sobel transform against our default gradient-based transform. Details of the motion input construction are provided in Appendix D.1.
• LAPO (Schmidt & Jiang, 2024): A standard monolithic latent action model with inverse dynamics model (IDM) and forward dynamics model (FDM). • FLAM (Wang et al., 2025): A factorized latent action baseline that decomposes scene dynamics into separate factors, addressing the limitations of a single monolithic latent action. We train two different models with differnt number of slots (4 and 8). • HiLAM (Kim et al., 2026): A hierarchical latent action baseline that uses a pretrained low-level LAM to extract latent action and aggregates them into higher-level latent skills. • Motion-LAPO: a LAPO variant that receives motion inputs together with pixel frames, isolating the effect of explicit motion processing.
Results on transferability. We provide qualitative and quantitative results on transferability. Figure 8 shows that OTF reconstructs motion signals under a test-time embodiment shift across different DCS environments, and Figure 9 shows results on controlled carrier transfer on Moving
• F-LAPO: a LAPO variant with an OTF-LAM-like patch encoder and patchwise quantization, isolating the effect of patch-level latent quantization. 6
Latent Actions from Factorized Transition Effects under Agent Ambiguity
where Esource and Etarget denote reconstruction error on the training and transfer sets, respectively. A smaller drop indicates better reusability. As shown in Table 1, the OTF vocabulary shows substantially smaller degradation than the monolithic global embedding with the same input type and transform. These results indicate that the factorized codebook is less sensitive to the morphology shift, while the global monolithic bottleneck more strongly entangles motion reconstruction with source-domain visual structure. 4.4. Policy Learning Results In Figure 10 (a), we report the mean and standard deviation of downstream returns. For each seed, the policy is evaluated over 10 trajectories, and each model is run with three seeds. The performance of OTF-LAM and OTF-LAM-Dino is reported using the best-performing K.
Figure 9. Controlled carrier transfer on Moving MNIST. Table 1. Quantitative evaluation of cross-morphology zero-shot transfer degradation in DCS environments under different motion inputs and frame-level transforms. Motion inputs are described in Appendix D.1.
OTF
Motion Input type transform
MSE transfered
Drop (%)
train
gradient sobel gradient sobel
0.142 0.088 0.037 0.024
0.171 0.107 0.055 0.036
20.49% 20.98% 49.34% 51.58%
gradient sobel gradient sobel
0.172 0.120 0.048 0.034
0.271 0.196 0.080 0.058
58.18% 64.06% 64.78% 71.60%
acc. vel.
monolithic
acc. vel.
The results highlight three observations. First, OTF-LAM and OTF-LAM-Dino remain competitive with, and in some cases outperform, monolithic LAM baselines, despite using a reusable latent action interface, or even reusable visual encoder. Second, these results support our hypothesis that factorizing observed transitions can provide a useful latent action interface under distractor-induced ambiguity, while also showing that explicit scene-level factorization remains a strong advantage when available. Third, OTFLAM-Dino improves the mean return over decoder-based OTF-LAM in both environments. A possible explanation for the stronger performance of OTF-LAM-Dino is that it changes the prediction target rather than simply changing the encoder. Decoder-based OTF-LAM is trained to predict future pixels, which can pressure the latent action and forward model to account for texture, background motion, lighting, and other environment-specific details. OTF-LAMDino instead predicts future states in a frozen DINO representation space, where many of these pixel-level nuisance factors are suppressed while control-relevant state information is preserved. In this sense, the frozen representation may act as a regularizer, encouraging action aggregator to encode transition structure that is useful for control rather than appearance variation that is useful for reconstruction. Remark 2 (Toward reusable latent action models.). DINOWM (Zhou et al., 2025) suggests that a frozen visual representation can provide an effective state space for planning and control across environments. OTF-LAM-Dino applies this premise to observation-only latent action learning by fixing the visual prediction space and learning the actionconditioning variable from unlabeled transitions. The results indicate that this provides a viable latent-action interface, while identifying the learned aggregator and predictor as the remaining sources of environment dependence. The observed-transition vocabulary may transfer across visual carriers and morphologies, while the action aggregator and predictor remain learned components that can still absorb
MNIST dataset. Larger versions of these figures are provided in Appendix. From the reconstructed motion signals, we observe that OTF remains better aligned with the target transition. Note this difference can be visually subtle, since the reference and future frames are often close in pixel space. A model that copies the reference frame may therefore look plausible, even when it fails to incorporate the actual future transition. For this reason, the reconstructions should be compared against whether they reflect the target motion, not only whether they preserve a sharp reference appearance. The monolithic VQ-VAE baseline often shows this failure mode, producing reconstructions closer to the reference frame than to the future frame. This suggests that its latent bottleneck entangles appearance or morphologyspecific structure with motion information and falls back toward the static reference appearance under transfer. By contrast, OTF maintains a more usable reconstruction of the observed motion, indicating that its codebook captures observed-transition primitives that are not tied to a single digit class or embodiment. Quantitatively, we measure the relative transfer degradaE −Esource tion from the source to the target Drop = target , Esource 7
Latent Actions from Factorized Transition Effects under Agent Ambiguity FLAM (8)
HiLAM
OTF-LAM-Dino
OTF-LAM
OTF-LAM-Dino
LAPO
Motion LAPO
cheetah-run
F-LAPO
walker-run
OTF-LAM
OTF-LAM-Dino
cheetah-run
OTF-LAM
FLAM (4)
walker-run
(a) Downstream policy learning results
(b) Codebook size ablation study
Figure 10. (a) Bar chart comparing the relative mean return of latent action model variants on the DCS cheetah-run downstream policy evaluation task. (b) Downstream evaluation of OTF-LAM with different motion vocabulary sizes. Error bar is computed as the RMS of standard deviations per seed.
environment-specific regularities. This suggests a broader direction for pushing the generalization boundary of latent action models under minimal assumptions about the actor, action space, and environment.
poral motion from spatial information, which can be used for video reconstruction and generation (Shen et al., 2025; Yin et al., 2025; Liu et al., 2025). However, their primary objective is video reconstruction or generation and not the decomposition of multiple simultaneously moving sources into an abstraction for action-relevant factors. Other prior works have approached motion factorization through independent motion segmentation where the goal is to identify parts of a scene that move as either coherent or separately moving components (Meunier & Bouthemy, 2024; Wang et al., 2024; Choudhury et al., 2022). Ai et al. (2026) further show that motion structure can exist below the object level through dynamic-static disentanglement to separate independently moving parts from static geometry. Our focus is not on 3D reconstruction, but instead to learn reusable observedtransition primitives, which capture localized patterns of visual change caused by motions.
Effect of Motion Vocabulary Size. We study how the size of the motion primitives affects downstream policy performance. Figure 10 (b) reports evaluation returns for OTF-LAM and OTF-LAM-Dino with different codebook sizes K on cheetah-run and walker-run, averaged over 3 seeds, where each seed evaluates 10 trajectories. For decoder-based OTF-LAM, cheetah-run performance improves consistently as K increases. This suggests that a larger vocabulary provides a more expressive set of transition primitives for this environment. On walker-run, however, performance is relatively stable across codebook sizes, with all variants achieving similar returns, indicating that the required transition structure in walker-run is captured even by smaller vocabularies, or that downstream control is less sensitive to additional motion primitives in this setting. OTF-LAM-DINO shows a different pattern. It improves over decoder-based OTF-LAM for every tested K in both environments, suggesting that prediction in a frozen DINO representation space provides a useful downstream interface for the learned latent actions. However, its performance is not monotonic in K. Overall, codebook size should be viewed as an environmentand architecture-dependent capacity parameter rather than a monotonic source of improvement. A more systematic study of how vocabulary size interacts with motion complexity, representation-space prediction, and downstream control remains future work.
Latent Action Learning and Distractors. LAPO (Schmidt & Jiang, 2024) introduced this actionfree learning framework for control from videos, while LAPA (Ye et al., 2025) scaled this idea by pretraining latent actions from large video datasets for VLA modeling. However, these models compress each transition into a single compact latent action. This can perform well in controlled settings where the agent is the main source of change, but becomes ambiguous in scenes with distractors where background/camera motion or other exogenous motions may be entangled with the latent action (Nikulin et al., 2025; Stone et al., 2021). Nikulin et al. (2025) shows that latent action learning can degrade under distractors, and that limited action supervision can improve performance. Other methods improve robustness by constraining or focusing the visual transition: LAOF (Bu et al., 2025) uses optical-flow constraints, Klepach et al. (2026) uses object-centric scene decomposition, and Adnan et al. (2026) uses pretrained segmentation masks to suppress background distractors. Recent work also studies latent action world models in more diverse
5. Related Works Factorized Motion and observed-transition Representations. Recent works feature video autoencoding methods to learn decoupled representations that can separate tem8
Latent Actions from Factorized Transition Effects under Agent Ambiguity
in-the-wild videos (Garrido et al., 2026b). In contrast, we factorize the observed transition into multiple motion-effect factors and learn to gate them with current-frame context. Several recent methods introduce structured latent action spaces. AC-LAM (Wei et al., 2026) uses additive component vectors, FLAM (Wang et al., 2025) decomposes dynamics into entity-level IDM/FDM modules, and LPWM (Daniel et al., 2026) models object-centric stochastic dynamics with particle-level latents. Our work instead factorizes transitions through patch-level activations of a shared observedtransition codebook.
tending it to richer video-game domains, such as Mario-like environments with scrolling cameras, enemies, particles, and diverse non-agent dynamics, remains future work.
Acknowledgement We thank Chen Sun and Dan Haramati for helpful and meaningful discussions.
References Adnan, H., Jackson, M. T., and Zakharov, A. Segment to focus: Guiding latent action models in the presence of distractors, 2026. URL https://arxiv.org/abs/ 2602.02259.
Compositional and Quantized Latent Representations. VQ-VAE introduced learned discrete codebooks as reusable latent units, and LAPA uses a VQ-VAE-style objective to learn discrete latent actions from transitions (Van Den Oord et al., 2017; Ye et al., 2025). Compositional VQ-VAE-style representations have also been used to decompose complex human motion into reusable discrete components, for example in speech-driven holistic 3D motion generation (Yi et al., 2023) and music-conditioned dance video generation (Chen et al., 2025). Product quantization has also been used to learn compact discrete representations in visual and multimodal domains (Ma et al., 2024; El-Nouby et al., 2023). A complementary line of work studies compositionality through object-centric representations, where objects or slots serve as the basic units of decomposition (Locatello et al., 2020; Nam et al., 2026; Daniel & Tamar, 2022). In contrast, we build on the idea that codebooks can provide reusable components. This provides a compositional latent interface alternative to compact latent actions, where individual codes can be analyzed through their localized effects on the reconstructed transition.
Ai, H., Chang, W., Jiao, J., Leonardis, A., and Ofek, E. Articulation in motion: Prior-free part mobility analysis for articulated objects by dynamic-static disentanglement. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview. net/forum?id=mvKM40zDyn. Bu, X., Lyu, J., Sun, F., Yang, R., Ma, Z., and Li, W. Laof: Robust latent action learning with optical flow constraints. ArXiv, abs/2511.16407, 2025. URL https://api.semanticscholar. org/CorpusID:283110282. Chen, Z., Xu, H., Song, G., Xie, Y., Zhang, C., Chen, X., Wang, C., Chang, D., and Luo, L. X-dancer: Expressive music to human dance video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10602–10611, 2025. Choudhury, S., Karazija, L., Laina, I., Vedaldi, A., and Rupprecht, C. Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motion. In British Machine Vision Conference (BMVC), 2022.
6. Conclusion We presented OTF-LAM, a bottom-up approach to latent action learning under agent ambiguity. Since observation transitions are visual effects rather than actions themselves, a monolithic latent action can entangle controllable motion with distractors, camera dynamics, and other spurious transition sources. OTF addresses this by factorizing transitions into reusable observed-transition primitives before composing them into state-aware action-like latents. Our motion primitives transfer across held-out carriers and morphology shifts, and the resulting latent action model, OTF-LAM, support downstream policy learning in the conventional inverse–forward dynamics framework. OTF-LAM-Dino further shows that this interface can be paired with a frozen DINO representation space, improving over pixel-space prediction. These results support factorized observed effects as a useful intermediate representation for observation-only latent action learning. This work evaluates the approach on controlled settings and DCS distractor environments. Ex-
Daithankar, N., Gladstone, A., LeCun, Y., and Ji, H. You don’t need strong assumptions: Visual representation learning via temporal differences, 2026. URL https: //arxiv.org/abs/2606.15956. Daniel, T. and Tamar, A. Unsupervised image representation learning with deep latent particles, 2022. URL https: //arxiv.org/abs/2205.15821. Daniel, T., Qi, C., Haramati, D., Zadeh, A., Li, C., Tamar, A., Pathak, D., and Held, D. Latent particle world models: Self-supervised object-centric stochastic dynamics modeling. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=lTaPtGiUUc. 9
Latent Actions from Factorized Transition Effects under Agent Ambiguity
El-Nouby, A., Muckley, M. J., Ullrich, K., Laptev, I., Verbeek, J., and Jegou, H. Image compression with product quantized masked image modeling. Transactions on Machine Learning Research, 2023. ISSN 28358856. URL https://openreview.net/forum? id=Z2L5d9ay4B.
Liu, H., Sun, W., Zhang, Q., Di, D., Gong, B., Li, H., Wei, C., and Zou, C. Hi-vae: Efficient video autoencoding with global and detailed motion, 2025. URL https: //arxiv.org/abs/2506.07136. Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-centric learning with slot attention. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Garrido, Q., Nagarajan, T., Terver, B., Ballas, N., LeCun, Y., and Rabbat, M. Learning latent action world models in the wild, 2026a. URL https://arxiv.org/abs/ 2601.05230. Garrido, Q., Nagarajan, T., Terver, B., Ballas, N., LeCun, Y., and Rabbat, M. Learning latent action world models in the wild. ArXiv, abs/2601.05230, 2026b. URL https://api.semanticscholar. org/CorpusID:284543722.
Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview. net/forum?id=Bkg6RiCqY7.
Lucas, B. D. and Kanade, T. An iterative image registraHa, D. and Schmidhuber, J. Recurrent world models tion technique with an application to stereo vision. In facilitate policy evolution. In Bengio, S., Wallach, H., Proceedings of the 7th International Joint Conference on Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Artificial Intelligence - Volume 2, IJCAI’81, pp. 674–679, Garnett, R. (eds.), Advances in Neural Information San Francisco, CA, USA, 1981. Morgan Kaufmann PubProcessing Systems, volume 31. Curran Associates, Inc., lishers Inc. 2018. URL https://proceedings.neurips. cc/paper_files/paper/2018/file/ Ma, L., Ye, Y., Hong, F., Guzov, V., Jiang, Y., Postyeni, R., 2de5d16682c3c35007e4e92982f1a2ba-Paper. Pesqueira, L., Gamino, A., Baiyya, V., Kim, H. J., et al. pdf. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. In European Conference on Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering Computer Vision, pp. 445–465. Springer, 2024. diverse control tasks through world models. Nature, 640 (8059):647–653, 2025.
Mandlekar, A., Xu, D., Wong, J., Nasiriany, S., Wang, C., Kulkarni, R., Fei-Fei, L., Savarese, S., Zhu, Y., and Martín-Martín, R. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning (CoRL), 2021.
Hansen, N., Su, H., and Wang, X. TD-MPC2: Scalable, robust world models for continuous control. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum? id=Oxh5CstDJU.
Meunier, E. and Bouthemy, P. Segmenting the motion components of a video: A long-term unsupervised model. arXiv preprint arXiv:2310.01040, 2024. URL https: //arxiv.org/abs/2310.01040.
Horn, B. K. and Schunck, B. G. Determining optical flow. Artificial Intelligence, 17 (1):185–203, 1981. ISSN 0004-3702. doi: https://doi.org/10.1016/0004-3702(81)90024-2. URL https://www.sciencedirect.com/ science/article/pii/0004370281900242.
Nam, H., Lidec, Q. L., Maes, L., LeCun, Y., and Balestriero, R. Causal-jepa: Learning world models through objectlevel latent interventions, 2026. URL https:// arxiv.org/abs/2602.11389.
Kim, H., Pinto, L., and Kim, S. J. Hierarchical latent action model. In ICLR 2026 the 2nd Workshop on World Models: Understanding, Modelling and Scaling, 2026. URL https://openreview.net/forum? id=IUaiouMYXp.
Nikulin, A., Zisman, I., Tarasov, D., Lyubaykin, N., Polubarov, A., Kiselev, I., and Kurenkov, V. Latent action learning requires supervision in the presence of distractors. In Proceedings of the 42nd International Conference on Machine Learning, 2025. URL https: //icml.cc/virtual/2025/poster/46574.
Klepach, A., Nikulin, A., Zisman, I., Tarasov, D., Derevyagin, A., Polubarov, A., Lyubaykin, N., Kiselev, I., and Kurenkov, V. Object-centric latent action learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pp. 22626–22634, 2026.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov, V., Fernandez, P., HAZIZA, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., 10
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Galuba, W., Howes, R., Huang, P.-Y., Li, S.-W., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:// openreview.net/forum?id=a68SUt6zFt. Featured Certification.
Wei, H., Chen, X., Zhang, C., Pearce, T., Chen, J., Lamb, A., Zhao, L., and Bian, J. Learning additively compositional latent actions for embodied ai, 2026. URL https:// arxiv.org/abs/2604.03340. Ye, S., Jang, J., Jeon, B., Joo, S. J., Yang, J., Peng, B., Mandlekar, A., Tan, R., Chao, Y.-W., Lin, B. Y., Liden, L., Lee, K., Gao, J., Zettlemoyer, L., Fox, D., and Seo, M. Latent action pretraining from videos. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum? id=VYOe2eBQeh.
Perez, E., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. Schmidt, D. and Jiang, M. Learning to act without actions. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview. net/forum?id=rvUq3cxpDF.
Yi, H., Liang, H., Liu, Y., Cao, Q., Wen, Y., Bolkart, T., Tao, D., and Black, M. J. Generating holistic 3d human motion from speech. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 469–480, 2023.
Shen, C., Xu, L., Zhu, X., and Liu, G. Autoregressive video autoencoder with decoupled temporal and spatial context, 2025. URL https://arxiv.org/abs/2512. 11293.
Yin, X., Yuan, J., Hu, Z., Sun, W., Chen, J., Qiao, X., Li, H., and Sun, X. Deco-vae: Learning compact latents for video reconstruction via decoupled representation, 2025. URL https://arxiv.org/abs/2511.14530.
Sobel, I. and Feldman, G. M. An isotropic 3×3 image gradient operator. 1990. URL https://api. semanticscholar.org/CorpusID:59909525.
Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1094– 1100. PMLR, 30 Oct–01 Nov 2020. URL https:// proceedings.mlr.press/v100/yu20a.html.
Srivastava, N., Mansimov, E., and Salakhutdinov, R. Unsupervised learning of video representations using lstms. In Proceedings of the 32nd International Conference on Machine Learning, pp. 843–852. PMLR, 2015. URL https://proceedings.mlr.press/v37/ srivastava15.html.
Zhou, G., Pan, H., LeCun, Y., and Pinto, L. DINOWM: World models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, 2025. URL https: //openreview.net/forum?id=D5RNACOZEI.
Stone, A., Ramirez, O., Konolige, K., and Jonschkowski, R. The distracting control suite – a challenging benchmark for reinforcement learning from pixels, 2021. URL https://arxiv.org/abs/2101.02722. Van Den Oord, A., Vinyals, O., et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017. van der Maaten, L. and Hinton, G. Visualizing data using tsne. Journal of Machine Learning Research, 9(86):2579– 2605, 2008. URL http://jmlr.org/papers/v9/ vandermaaten08a.html. Wang, Z., Guo, J., and Daniilidis, K. Un-evimo: Unsupervised event-based independent motion segmentation. In European Conference on Computer Vision, pp. 228–245. Springer, 2024. Wang, Z., Shi, C., Hu, J., Martín-Martín, R., and Stone, P. Flam: Scaling latent action world models with factorization. In NeurIPS 2025 Workshop on Embodied World Models for Decision Making, 2025. 11
Latent Actions from Factorized Transition Effects under Agent Ambiguity
A. OTF Implementation Detail Encoder. The input is a motion observation ot ∈ RC×H×W . The encoder partitions ot into non-overlapping spatial patches and embeds each patch independently using a lightweight MLP patch encoder. We concatenate normalized spatial coordinates to each flattened patch representation before the MLP, so that the encoder has access to coarse patch location. The output is a grid of latent patch embeddings {ft,i }P i=1 , where each embedding corresponds to a local transition region. Local code assignment. Each patch embedding is discretized using a learned codebook C = {c(1) , . . . , c(K) }, where patch i is assigned to its nearest codebook entry. To stabilize training, we use a short warmup phase in which quantization is disabled and the model behaves as a continuous autoencoder. After warmup, the codebook is initialized with k-means clustering over encoder patch embeddings collected from the training data, and discrete quantization is enabled. (k)
Decoder input construction. For each code k, we compute an occupancy map Mt indicating which patches are assigned (k) to the code, and a normalized usage weight wt equal to the fraction of assigned patches. Together, these define the (k) (k) observed-transition factor et,k = (c(k) , Mt , wt ). Each factor is mapped to a descriptor ht,k = ρη (et,k ) using a shared MLP. The descriptors are then placed back onto the patch grid using the occupancy maps Ht (u, v) =
K X
(k)
Mt (u, v)ht,k .
k=1
Decoder. The decoder reconstructs the motion observation from the spatial feature map Ht . It uses a convolutional decoding network with bilinear upsampling to produce an output at the original resolution. A separate convolutional encoder extracts spatial context features from the current frame xt , and these features are combined with Ht before decoding. The reference pathway provides appearance and support information, while the codebook pathway provides the factorized transition information. Training. The objective combines a reconstruction loss with the standard VQ-VAE (Van Den Oord et al., 2017) codebook, commitment, and orthogonal losses L = Lrec + λcode Lcode + λcommit Lcommit + λorth Lorth .
(9)
The codebook is updated by exponential moving average updates, where orthogonality primarily serves as a diagnostic for codebook diversity. To reduce code collapse, rarely used codebook entries are periodically reinitialized using randomly sampled encoder embeddings. We train with AdamW (Loshchilov & Hutter, 2019) and gradient clipping. All OTF tokenizers are trained for 10 epochs, with batch size 512, learning rate 1e-4, codebook vector dimension 32. For all DCS (Stone et al., 2021) environment, we use gradient transform to each frame and subtract each other to construct motion input.
B. OTF-LAM Implementation Detail Let B denote the batch size, K the number of VQ codes, (h, w) the VQ-VAE patch grid, C the number of image channels, and (H, W ) the image resolution. For convenience, we denote the stacked occupancy maps by Ot ∈ RB×K×h×w , where (k) Ot,k = Mt is the occupancy map of code k. B.1. Encoders State Encoder The state encoder extracts a spatial representation of the current observation conditioned on the observedtransition usage pattern. It takes the current frame xt ∈ RB×C×H×W and the code usage vector wt ∈ RB×K after projected into a compact conditioning vector cocc = Wocc wt . t The image branch is a convolutional encoder with strided downsampling and residual blocks. Rather than concatenating the code usage vector to the image, we use cocc to modulate intermediate visual features through FiLM conditioning t (Perez et al., 2018) after each encoder stage. This allows the encoder to produce state features that are aware of the active observed-transition vocabulary while preserving the spatial structure of the current frame. The encoder outputs a spatial state feature map st ∈ RB×ds ×h×w . 12
Latent Actions from Factorized Transition Effects under Agent Ambiguity
When necessary, a 1 × 1 projection adjusts the channel dimension to ds , and bilinear interpolation is used toP match the target 1 spatial resolution (h, w). We also compute a global state summary by spatial average pooling s̄t = hw i,j st,:,i,j . The spatial feature map st is used by the decoder to preserve layout information, while the pooled feature s̄t provides a compact state representation for later conditioning modules. Occupancy Encoder The occupancy encoder converts the spatial support of each observed-transition code into a compact representation. Given the occupancy maps Ot , the encoder processes each code occupancy map independently and produces ut ∈ RB×K×do . Each vector ut,k summarizes where code k is active in the current transition. By default, we use an MLP occupancy encoder. For each code k, the corresponding occupancy map is flattened and passed through a small network ut,k = focc (Ot,k ; s̄t ), where s̄t is the global state feature from the state encoder. We condition the hidden representation on s̄t using FiLM, allowing the same occupancy pattern to be interpreted in the context of the current frame. This yields a compact state-aware embedding of each code’s spatial support. B.2. Action Aggregator (k)
(k)
We first get et,k = (c(k) , Mt , wt ) from OTF module. We map this descriptor to a factor token (k)
rt,k = ffac (et,k ; s̄t , Mt ), (k)
where the embedding network is conditioned on the global state feature s̄t and the occupancy embedding Mt using FiLM. This conditioning allows the same codebook primitive to be interpreted differently depending on the current visual state and its spatial support. We then calculate occupancy-pooled local state. For code k, we compute this vector by occupancy-weighted average pooling over the spatial state feature map P i,j Ot,k,i,j st,:,i,j P ℓt,k = . max( i,j Ot,k,i,j , ϵ) Now we summarize information. For each code k, the gate network takes the factor token, local state summary, and global in state summary gt,k = [rt,k , ℓt,k , s̄t ]. The gate network Gθ maps each factor representation to a scalar logit, which is converted into a continuous gating weight with a sigmoid activation in αt,k = σ(Gθ (gt,k ; cgate t,k )) ∈ [0, 1],
where cgate t,k = [s̄t , ut,k ] is for FiLM-conditioning. Codes with zero occupancy are masked out before aggregation. Finally, the model forms a factor-level latent by a normalized gated average over factor tokens PK fac k=1 αt,k rt,k zt = , PK max( k=1 αt,k , ϵ) and the final action-like latent is obtained by a projection ztact = Pθ (ztfac ). The projection is the identity when the factor-token dimension already matches the desired latent dimension; otherwise, it is a linear layer. B.3. Decoder The decoder predicts the next observation from the current-state features and the selected latent action. It receives the spatial state feature map st from the state encoder and the latent action ztact , and produces a residual update added to the current frame. To preserve spatial structure, we first project ztact into a low-resolution spatial grid ztgrid ∈ RB×dg ×h×w . 13
Latent Actions from Factorized Transition Effects under Agent Ambiguity
This grid is concatenated with the state feature map st , allowing the decoder to combine the current visual layout with the transition information encoded in the latent action. The concatenated representation is then processed by a convolutional decoder with progressive upsampling. The latent action also modulates the decoder features through FiLM conditioning. After each upsampling stage, channel-wise scale and shift parameters are predicted from ztact and applied to the intermediate feature map a ← (1 + γ(ztact )) ⊙ a + β(ztact ). This gives the latent action a direct mechanism for controlling the predicted transition while the spatial state features preserve appearance and layout.
C. OTF-LAM-DINO Implementation Detail Here we describe a decoder-free variant, OTF-LAM-DINO, which replaces pixel-space prediction with prediction in a frozen visual representation space. The purpose of this variant is to test whether the latent action learned from observed-transition primitives can also condition a JEPA-style latent dynamics model, and furthur maintain downstream performance with frozen DINOv2 (Oquab et al., 2024) encoder. We use the same pretrained OTF module as OTF-LAM. Latent visual state.
Given an observation xt , we encode it with a frozen DINOv2 encoder: St = DINO(xt ),
St+τ = sg(DINO(xt+τ )),
where St denotes the patch-token representation and sg(·) stops gradients through the future target. The DINO encoder is kept fixed throughout training. Thus, unlike the main OTF-LAM model, both the state representation and the prediction target are defined by a reusable pretrained visual encoder. Latent action. Following OTF module described in Appendix A, we first get the set of assigned code for each patch, Ct . An action abstraction module first abstract local transition primitives, conditioned on the current DINO state: (i)
(i)
ui = ϕ([St , Ct , pi ]) where pt is learnable positional encoding. Next, since our state encoder is frozen DINO encoder, we need stronger aggregator compared to OTF-LAM. Here we pool four action tokens with Cross Attention, then feed that into small Transformer (depth=2) with query residual. At = Transformer (Q + CrossAttn(Q, Ut , Ut )) , ztact = ϕout (Flatten(At )). The default action embedding dimension is 256 and ϕ is a linear layer with layernorm. Forward dynamics. Instead of reconstructing the future frame, OTF-LAM-DINO predicts the future DINO representation: Ŝt+τ = Pθ (St , ztact ). The latent action token is injected into the predictor at multiple layers, to prevent model ignoring it after the input layer. Training objective.
The model is trained with a latent prediction loss in frozen DINO feature space: 2
LOTF-LAM-DINO = Ŝt+τ − St+τ
. 2
Discussion. OTF-LAM-DINO follows the same latent action principle as OTF-LAM, but evaluates it in a different prediction space. The decoder-based model asks whether ztact can support future observation reconstruction, while OTFLAM-DINO asks whether the same type of latent action can support future prediction in a reusable representation space. This variant is therefore complementary to the main model: it removes the pixel decoder, but still relies on the OTF vocabulary to construct the latent action from local observed-transition primitives. 14
Latent Actions from Factorized Transition Effects under Agent Ambiguity
D. Additional Visualizations D.1. Motion-Centered Transition Inputs To learn reusable observed-transition primitives effectively, the input should reduce semantic appearance information present in RGB frames while preserving the visual evidence of motion. We therefore consider several simple motion-centered input spaces. Specifically, we construct motion inputs by combining two temporal difference operators, which define the order of motion, with three frame-level transforms, which change the visual carrier through which motion is represented. Let ϕ denote a transformation applied independently to each frame. For first-order velocity inputs, we define ovel,ϕ = ϕ(xt+τ ) − ϕ(xt ). t
(10)
oacc,ϕ = ϕ(xt+τ ) − 2ϕ(xt ) + ϕ(xt−τ ). t
(11)
For second-order acceleration inputs, we define
In all cases, the transform is applied before temporal differencing. We evaluate choices of ϕ: Sobel (Sobel & Feldman, 1990) features after grayscale transformation, and image-gradient features. Sobel and gradient transforms emphasize edge changes and local directional variation, respectively. Figure A1 shows examples.
Figure A1. Examples of motion-centered inputs produced by different frame-level transforms: Sobel, gradient, and none (identity).
D.2. Visualized Examples of OTF Module We provide additional qualitative visualizations of the learned observed-transition vocabulary. Distracting Control Suite (Stone et al., 2021). We train the observed-transition factorizer using the Sobel-transformed velocity input. Similarly to the Moving MNIST visualization, Fig. A2(b) shows the learned codebook after training the observed-transition factorizer on DCS samples. Fig. A3 shows example assignments on individual walker-run transitions, including the current frame, the motion-centered input, the local patch embeddings, and the assigned codebook entries. MovingMNIST (Srivastava et al., 2015). We train the observed-transition factorizer using the Sobel-transformed acceleration input. To make the motion signal more pronounced, we use a temporal stride of two frames when constructing the transition input. Fig. A2 (a) visualizes the learned codebook after training the factorizer described in Sec. 3.1. This visualization provides a qualitative view of how the discrete observed-transition primitives are organized in the learned vocabulary. Fig. A6 shows example assignments on individual Moving MNIST samples. For each sample, we visualize the current frame, the motion-centered input, the t-SNE (van der Maaten & Hinton, 2008) projection of patch embeddings, and the corresponding assigned codebook entries on the patch grid or assigned observed-transition primitives on the patch grid. These examples illustrate how local transition patterns are mapped to discrete codebook primitives across different spatial regions. Then we use Moving MNIST (Srivastava et al., 2015) as a controlled diagnostic for motion reusability. The VQ-VAE factorizer is trained on digits {0,1,2,3,4} and evaluated on held-out digits {5,6,7,8,9}, while the set of observedtransition primitives is shared across the two splits. Figure A4 shows result. 15
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Figure A2. Visualization of the learned codebook of observed-transition primitives on controlled Moving MNIST. Each point denotes one learned codebook vector, and the two-dimensional coordinates are obtained by applying t-SNE (van der Maaten & Hinton, 2008) to the code embeddings.
Figure A3. Example code assignments for walker-run transitions from Distracting Control Suite.
16
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Figure A4. Zero-shot transfer experiment results on MovingMNIST.
E. Observed Motion Signals and Patch Scale E.1. Observed Motion Signals Are Not Pure Motion The motion input used in this work is computed from image observations, for example by first- or second-order temporal differencing. Although this suppresses static appearance, it does not expose physical motion directly. A pixel-level velocity or acceleration signal is an observed effect of motion through the local visual content of the scene. More formally, let mt,p ∈ M denote the underlying local motion effect at patch p, such as translation, rotation, deformation, or camera-induced displacement. Let ξt,p ∈ Ξ denote the local visual carrier through which this motion is observed. This carrier may include local texture, contrast, edge orientation, color, depth discontinuities, occlusion boundaries, or other patch-level appearance statistics. The observed motion signal yt,p is then better understood as yt,p = Ψ(mt,p , ξt,p ),
(12)
for some image-formation-dependent map Ψ, rather than as a direct observation of mt,p alone. Thus, the codebook should not be interpreted as spanning the space of physical observed-transition primitives M in isolation. It is better viewed as learning from the space of observed motion effects Y = {Ψ(m, ξ) : m ∈ M, ξ ∈ Ξ}.
(13)
This distinction is important for interpreting the learned codes. For example, the same rightward displacement can produce different observed signals depending on whether it occurs on a vertical edge, a textured background, a uniform region, or an articulated limb boundary. Conversely, patches with different appearance may still activate similar codes when their local transition patterns are sufficiently similar. Therefore, our codes should not be interpreted as pure geometric motion variables. They are more accurately viewed as reusable observed-transition primitives: local transition patterns that combine motion with the visual carrier through which that motion becomes visible. 17
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Figure A5. Enlarged version of Fig. 8 for readability.
This interpretation also clarifies the role of the reference-frame conditioning branch. The codebook is encouraged to represent transition effects, while the reference frame provides complementary static support information such as object shape and local appearance. In this sense, the method does not attempt to remove the visual carrier entirely. Rather, it limits the carrier information available to each local patch and avoids forcing the codebook to encode full object identity. E.2. Patch Size and the Motion–Carrier Trade-off Patch size controls the granularity at which observed transition effects are quantized. It therefore determines an important trade-off between motion locality and carrier ambiguity. If patches are too small, each token contains only a limited visual neighborhood. This reduces leakage from object identity or large-scale appearance, but it can make local motion difficult to infer. Small patches are more sensitive to noise, texture aliasing, and the aperture problem (Horn & Schunck, 1981; Lucas & Kanade, 1981): a short edge fragment may not contain enough information to distinguish translation, rotation, or deformation. In such cases, the codebook may learn overly local residual patterns rather than stable motion-effect primitives. If patches are too large, each token contains richer spatial context and may provide more reliable evidence for local motion. However, larger patches are also more likely to contain multiple motion sources, object-part identity, texture statistics, or foreground-background boundaries. As a result, the learned codes may become more appearance- or part-specific, approaching object-conditioned transition templates rather than reusable observed-transition primitives. In this regime, the effective code space may expand from a compact motion-effect basis toward a larger product space of motion and local carrier factors. The desired patch scale is therefore task-dependent. A useful patch should be large enough to contain sufficient local structure for estimating a dominant transition effect, but small enough to avoid mixing multiple independent motion sources or encoding full object identity. In our setting, patchification is intended to preserve a direct local correspondence between the observed motion field and the latent code assignment. The patch size should therefore be interpreted as a design choice 18
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Figure A6. Example code assignments for controlled Moving MNIST transitions.
that controls the locality of the learned observed-transition primitives.
F. Training Details OTF and LAPO family OTF-LAM, LAPO, and LAPO variants are trained for 20k steps, while OTF-LAM-DINO is trained for 10k steps. FLAM (Wang et al., 2025) For FLAM, we use dmodel = 256, 2 transformer blocks, and z channels of size 128. We evaluate 4 and 8 slots. The action codebook uses 16 codes per factor, with the VAE-VQ KL weight annealed from 0 to 2 × 10−4 . We use sub-trajectory length 10, history length 1, and 5 prediction steps. The tokenizer uses an IMPALA encoder, a LAPO decoder, image size 64, an FSQ quantizer with codebook size 1024, levels (4, 4, 4, 4, 4), 128 latent channels, and codebook dimension 128. FLAM is trained for 10 epochs with batch size 32, learning rate 10−4 , weight decay 0, and gradient norm clipping at 1.0. We use xFormers attention, frame skip 1, HDF5 datasets, and seed 42. HiLAM (Kim et al., 2026)
For HiLAM, we train for 50k steps with learning rate 10−4 and latent action dimension 64.
Downstream For downstream policy learning, the OTF-LAM and baselines (FLAM, HiLAM, LAPO, and LAPO variants) behavior cloning policy is trained for 10 epochs, and action decoder is trained for 10k steps. For OTF-LAM-DINO, policy is trained for 5 epochs, and 5k steps for action decoder training
G. Full Results
19
Latent Actions from Factorized Transition Effects under Agent Ambiguity
Table A1. Mean and std of average return across 10 trajectories over 3 random seeds. Cheetah-Run
Model
Walker-Run
Mean
Std.
Mean
Std.
FLAM (4) (Wang et al., 2025)
41.58
28.48
27.61
7.22
Flam (8) (Wang et al., 2025)
31.95
21.11
34.97
9.15
HiLAM (Kim et al., 2026)
22.64
12.42
25.12
5.88
LAPO (Schmidt & Jiang, 2024)
11.48
7.19
28.71
7.59
Motion LAPO
9.68
4.18
23.93
4.40
F-LAPO
11.56
3.39
23.44
5.79
OTF-LAM
25.79
17.63
26.94
7.91
OTF-LAM-DINO
43.80
26.98
28.87
8.23
Table A2. Downstream evaluation of OTF-LAM and OTF-LAM-DINO with differet motion vocabulary sizes. Std. is computed as root mean square across three random seeds. OTF-LAM
K
Cheetah-Run
Walker-Run
OTF-LAM-DINO
Mean
Std.
Mean
Std.
16
10.142
6.574
40.85
23.69
32
13.764
12.272
43.80
26.98
64
19.393
10.503
33.09
20.46
128
25.788
17.664
42.92
26.01
16
26.941
7.919
28.87
8.23
32
24.231
7.856
27.64
6.57
64
24.812
5.734
28.70
7.74
128
25.019
6.712
27.65
8.09
20