arXiv CSopen access

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

Jusuk Lee 1 Seungjae Lee2 Jonghun Shin1 Hoseong Jung1 Sungha Kim1 3 Daesol Cho H. Jin Kim1 Jia-Bin Huang2,† Furong Huang2,†

arXiv:2605.30350v1 [cs.RO] 28 May 2026

1

Seoul National University 2 University of Maryland, College Park 3 Georgia Institute of Technology https://dynaflip-robotics.github.io

Dynamics-aware Visual Perception

Downstream Performance

…

DynaFLIP

MLP / DP / VLA

… Image R3M

Visual Encoder CLIP DINOv2

LIBERO Fine-tuned

Robot Action

Policy SigLIP

LIBERO Frozen

Simulation MetaWorld

DynaFLIP (Ours)

Control-Relevant Score

Grad-CAM heatmaps over action prediction

RLBench

Metric Real World In-Distribution R3M

Feature visualization with PCA

CLIP

DINOv2

Out-of-Distribution SigLIP

DynaFLIP (Ours)

Figure 1: DynaFLIP learns dynamics-aware visual representations that focus on control-relevant regions and capture spatially coherent structure, leading to strong downstream performance. DynaFLIP serves as a visual backbone for diverse downstream policies (MLP, diffusion policy, VLA). Grad-CAM shows DynaFLIP attending to manipulated objects and interaction regions, while PCA reveals coherent object-level structure. DynaFLIP outperforms baselines across simulation, real-world tasks, and the control-relevant metric.

Abstract Robot manipulation critically depends on perception that preserves the actionrelevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recognition or vision-language alignment, leaving motion understanding to downstream policies. We introduce DynaFLIP, a dynamics-aware multimodal pre-training framework that pushes motion understanding upstream into perception. We construct image–language–3D flow triplets from heterogeneous human and robot videos, and use these triplets as trainingtime supervision to shape an image-only encoder. Our key idea is to encourage the three modalities to span a small simplex volume in the shared hyperspherical space—a smaller simplex volume indicating stronger alignment. To avoid the geometric ambiguity and trivial collapse of naive volume minimization, we combine simplex-volume minimization with a cosine regularizer and a contrastive objective. Our analyses show that DynaFLIP focuses on control-relevant regions critical for manipulation. The resulting dynamics-aware representations serve as reusable visual backbones and consistently outperform baselines across diverse downstream † Equal Advising.

Preprint.

policies, including VLAs. We validate this across diverse simulation and real-world setups, with gains reaching +22.5% under out-of-distribution scenarios. Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.

1

Introduction

A central goal of robot learning is to build agents that generalize across diverse real-world environments and tasks—new objects, backgrounds, and distractors. Recent robot learning systems increasingly pursue this goal by reusing powerful vision encoders such as CLIP, SigLIP, and DINOv2 [42, 57, 39] inside diverse policies, ranging from imitation learning to Vision-Language-Action (VLA) models [31, 32, 5, 2, 22]. This practice inherits a key assumption: perception can be borrowed from encoders pre-trained for mainstream computer-vision objectives, while motion and dynamics are handled mainly by downstream planning or control. We argue that this assumption fundamentally limits robot generalization. In particular, manipulation is about how actions induce state transitions, yet existing visual encoders are not exposed to motion and dynamics during pre-training. As a result, they often attend to visually salient but control-irrelevant regions instead of the manipulated object or contact area. We therefore rethink the robotic pipeline by pushing dynamics awareness upstream into perception, so that visual encoders represent not only what is in the scene, but also how the scene changes under action. The challenge is then how to inject dynamics awareness into a visual encoder when the encoder ultimately operates on a single image at test time. Images alone do not always reveal which aspects of a scene are causally relevant for action, whereas other modalities can provide complementary evidence about intended and realized state changes. This suggests using such modalities not as additional inputs at test time, but as supervision to shape the visual encoder’s representation during training. In this work, we focus on three such modalities, each contributing information that the others cannot. Image transitions provide the most direct visual evidence of what changed between states, but cannot explain why a change occurred. Language fills this gap by describing the intended transition at a semantic level. 3D flow then adds what neither image transitions nor language can provide: an explicit, viewpoint-invariant account of how the scene moves in physical space, decoupled from 2D appearance. We deliberately select these three modalities because all of them can be extracted from action-free video data, allowing pre-training to leverage large-scale human and robot videos rather than the limited robot-collected datasets. With the three modalities identified, the remaining challenge is how to transfer their supervisory signal into the latent space of an image-only encoder. Standard anchor-based multimodal objectives [15, 62, 43]—even when the image serves as the anchor—do not ensure mutual alignment among the remaining modalities. An alternative strategy, inspired by prior work in multimodal retrieval [55, 11, 10], is to constrain all modality embeddings jointly through the simplex they span. However, naive simplex-volume minimization is itself prone to two pitfalls. First, geometric ambiguity: a low-volume simplex does not guarantee mutual alignment, since the simplex volume can shrink even when some modality pairs remain far apart. Second, trivial collapse: in the absence of negative tuples, the simplex volume is minimized when all modality embeddings collapse to a single point. A useful robotics representation must therefore exploit higher-order multimodal geometry to learn a coherent, control-relevant visual latent space, while avoiding these degeneracies. In this paper, we propose DynaFLIP, a Dynamics-aware 3D Flow-Language-Image Pre-training framework that uses image transitions, language, and 3D flow as training-time supervision to shape the latent space of an image-only encoder, yielding control-relevant visual representations for downstream manipulation. Building on simplex-based alignment [55, 11, 10], we minimize the volume of the simplex spanned by the three modalities in a shared embedding space (a triangle area in our threemodal setting). To address the two pitfalls of naive simplex-volume minimization, we resolve geometric ambiguity through a cosine regularizer between selected modality pairs, and prevent trivial collapse by embedding the cosine-augmented energy in an InfoNCE-style contrastive framework [38]. We further introduce two auxiliary objectives—a temporal contrastive loss and an actor loss—to reinforce trajectory-level temporal structure and strengthen dynamics-aware visual representations. Extensive experiments in both simulation and real-world environments show that the resulting encoder outperforms strong baselines, transfers effectively as a visual backbone across diverse downstream policies, and is especially robust under out-of-distribution variations. 2

Pre-Training

Deployment Alignment Loss (Eq. (3))

“Place the air column film onto the items”

𝐹!:!#$

𝐼!

𝐼!#%

(Image at time 𝑡)

(Image at time 𝑡 + 𝐻)

3D Motion Encoder

Image Encoder

Image Encoder

(Pretrained DINOv2)

(Pretrained DINOv2)

[CLS] N patches

[CLS] N patches

𝑑! Stop-grad.

Temporal Motion Transformer

Mean pooling Concat

Mean pooling

𝑨 𝒛𝑰

(Instruction)

Language Encoder

Small area 𝑨 (Well aligned)

(Pretrained T5)

Image Image Encoder

Instruction Language Encoder (Pretrained T5)

Actor Loss (Eq. (6)) Adapter

Adapter

Predict

[EOS]

Concat

𝑑!

𝒛𝑭

𝐿

(3D flow from time 𝑡 to 𝑡 + 𝐾)

[CLS] K tokens

Optional “Put the fork on the towel”

𝒛𝑳

𝑑!#%

Image

3D Flow

Temporal Contrastive Loss (Eq. (5))

Downstream Policy (MLP, Diffusion policy, VLA)

MLP Fusion Layer Positive

3D flow feature (𝒛𝑭 )

Image transition feature (𝒛𝑰 )

Language feature (𝒛𝑳 )

Negative

Robot action

Figure 2: Overview of DynaFLIP. Three modalities are encoded into embeddings in a shared hyperspherical space. The image encoder (initialized from DINOv2 and fully fine-tuned) produces per-frame features from It , It+H via CLS and mean-pooled patch tokens, which are then fused into zI . A frozen T5 with a learnable adapter produces zL from the EOS token of L, and a 3D flow encoder produces zF from Ft:t+K . The alignment loss minimizes the area A spanned by these embeddings, with auxiliary actor and temporal contrastive losses reinforcing dynamics-aware representations. Our pre-trained image encoder serves as a visual backbone for diverse downstream policies, with the language encoder optionally included for instruction-conditioned policies.

In summary, our contributions are threefold: (i) We recast robot generalization partly as a perception problem: robust manipulation requires visual representations that encode dynamics- and controlrelevant structure, rather than merely what is most visually salient. (ii) We introduce DynaFLIP that distills supervision from image transitions, language, and 3D flow into an image-only encoder through higher-order multimodal alignment while preventing geometric ambiguity and trivial collapse. (iii) We construct image–language–3D flow triplets from human and robot videos and show that DynaFLIP transfers strongly as a reusable backbone across simulation and real-world manipulation, achieving up to 22.5% improvement over the strongest baseline under real-world OOD perturbations.

2

Method

DynaFLIP shifts visual pre-training from static scene understanding to motion-induced state transitions. Section 2.1 introduces a simplex-guided multimodal alignment objective that aligns image transitions, language, and 3D flow into a shared embedding space while resolving two optimization pitfalls: geometric ambiguity and trivial collapse. Section 2.2 then presents the auxiliary objectives— temporal contrastive and actor losses—that further strengthen dynamics-aware visual representations. Finally, Section 2.3 describes how we construct large-scale image–language–3D flow triplets from human and robot videos. 2.1

Simplex-Guided Multimodal Alignment for Dynamics-Aware Representation

We aim to learn dynamics-aware visual representations by aligning three transition-based modalities— image transitions, language, and 3D flow. Image transitions capture visual state changes, language specifies the intended transition at a semantic level, and 3D flow encodes physical motion in the scene. We map each modality to an ℓ2 -normalized embedding on the unit sphere: zI for the image transition, zL for the language, and zF for the 3D flow. A common strategy for aligning multiple modalities is anchor-based contrastive learning, where one modality serves as a reference and each auxiliary modality is independently aligned to it [15, 62, 43]. However, this design enforces pairwise alignment only with the anchor and does not constrain the non-anchor modalities relative to each other. To capture mutual alignment among all three modalities, we adopt a simplex-volume-based formulation [55, 11, 10]. For an m-modal tuple of ℓ2 -normalized embeddings, the generalized simplex volume Vm measures the volume of the simplex spanned by the embeddings in the shared latent space, with smaller Vm indicating stronger joint alignment. In our

3

Sample 𝑖 Sample 𝑗

Pull

Image Language 3D Flow

𝒛𝒊𝑭

𝒛𝑳𝒊

𝐴≈0

𝒛𝒊𝑰

𝒋

𝒛𝑭

𝒛𝒊𝑳

𝒛𝒊𝑭

𝐴≈0 𝒛𝒊𝑰

𝒋

𝒛𝑳

𝒛𝒊𝑳

𝒛𝒊𝑰

𝒋

𝒛𝒊𝑰

𝒋

𝒛𝑭 𝒛𝒊𝑭

𝒛𝑰

𝒋

𝒛𝑳

𝒋

𝒛𝑰

Push

𝒛𝒊𝑳

𝒛𝒊𝑭

Flat triangle (Poorly aligned)

Compact triangle (Well aligned)

Representation Collapse

Representation Dispersion

w/o. cosine regularizer

w. cosine regularizer

w/o. negative tuples

w. negative tuples

(a) Cosine regularizer prevents geometric ambiguity

(b) Contrastive loss prevents trivial collapse

Figure 3: Two optimization pitfalls of naïve simplex-volume minimization. (a) Geometric ambiguity. A flat triangle has near-zero area even when one modality remains far from the other two. The cosine regularizer pulls selected modality pairs together, yielding a desired alignment (see Eq. (2)). (b) Trivial collapse. Without negative tuples, all modality embeddings collapse to a single point. Negative tuples in our contrastive framework push apart mismatched configurations, preventing collapse (see Eq. (3)).

three-modal setting, Vm reduces to the triangle area 1p V3 (zL , zI , zF ) = A(zL , zI , zF ) = ⟨u, u⟩⟨v, v⟩ − ⟨u, v⟩2 , u = zI − zL , v = zF − zL , (1) 2 spanned by the three embeddings. A small triangle area thus indicates joint alignment among all three modalities, capturing higher-order multimodal geometry beyond anchor-based pairwise alignment. The general m-modal formulation is provided in Appendix B.1. Cosine regularization. However, naive triangle-area minimization suffers from geometric ambiguity: the triangle area can shrink to zero even when one modality remains far from the other two—for example, when all three embeddings lie nearly on a single line, the triangle collapses to a flat shape with near-zero area despite poor mutual alignment (Figure 3 left). To prevent such configurations, we augment the triangle area with a cosine regularizer between language and 3D flow embeddings, defining the joint alignment energy as E(zL , zI , zF ) = A(zL , zI , zF ) − α⟨zL , zF ⟩,

(2)

where α ≥ 0 balances triangle-area minimization and pairwise cosine alignment. The cosine term explicitly pulls zL and zF together, penalizing flat configurations where these modalities remain far apart even though the triangle area is small. Combined with the triangle area’s joint constraint, the resulting energy encourages that low values reflect genuine alignment among all three modalities. Appendix B.3 provides a formal analysis of the issues underlying triangle-area minimization alone, and Appendix B.4 shows how the cosine regularizer mitigates them. Contrastive framework. Yet directly minimizing E admits another degeneracy: trivial collapse, where all three embeddings reduce to a single point and E vanishes (Figure 3 right). To prevent this, we embed the joint alignment energy into an InfoNCE-style contrastive objective [38]. For each sample i in a batch B, we construct a set of negative tuples N (i) by mismatching one or more modality embeddings across the batch, and define the alignment loss as i X exp(−E(zL , zIi , zFi )/τ ) P Lalign = − log , (3) i i i exp(−E(zL , zI , zF )/τ ) + z̃∈N (i) exp(−E(z̃)/τ ) i∈B where τ > 0 is the temperature parameter. By forcing matched tuples to achieve lower energy than mismatched ones, the contrastive loss prevents the collapse mode in which all samples share the same embedding and attain low energy simultaneously. Encoder architecture. We instantiate the three encoders as follows. Given an image observation It , a future observation It+H separated by temporal offset H, a language instruction L, and a 3D flow trajectory Ft:t+K over a temporal window of length K, we encode the three modalities as


(t) (t) zI = Π fϕ (It+H ) − fϕ (It ) , zL = Π hθ (L) , zF = Π gψ (Ft:t+K ; sg(fϕ (It ))) , (4) where Π(v) = v/∥v∥2 projects features onto the unit sphere, and fϕ , hθ , and gψ denote the image, (t) language, and 3D flow encoders, respectively. The image transition embedding zI is defined as the normalized feature difference between It and It+H , forcing the embedding to capture visual (t) state change rather than static appearance. The 3D flow embedding zF conditions on the current image feature with stop-gradient (sg) to preserve semantic grounding while blocking trivial shortcut solutions through the image branch. 4

2.2

Auxiliary Objectives for Dynamics-aware Representation

The alignment objective Lalign captures dynamics within each transition window, but it does not provide a signal about how representations should relate across longer temporal horizons. To encode trajectory-level temporal structure, we adopt a temporal contrastive loss [37, 24], which pulls embeddings of nearby frames closer than distant frames within the same trajectory. Given a triplet (It1 , It2 , It3 ) from the same video with t1 < t2 < t3 , let zti1 , zti2 , zti3 denote their embeddings, and let zt̸=1i denote a negative embedding from a different video in the batch. We define Ltcn = −

X

log

i∈B

exp(S(zti1 , zti2 )) exp(S(zti1 , zti2 )) + exp(S(zti1 , zti3 )) + exp(S(zti1 , zt̸=1i ))

,

(5)

where S(·, ·) is the negative ℓ2 distance, so that closer embeddings receive higher similarity scores. To further reinforce the dynamics-aware representations, we introduce an auxiliary actor loss via a single-step 3D flow prediction objective in the spirit of behavior cloning [41]. This objective requires the image encoder to predict motion explicitly from a single frame, thereby encouraging the representation to encode manipulation dynamics more directly. Given the image feature fϕ (It ), a 3D flow prediction head outputs F̂t , and we minimize the mean squared error to the ground-truth flow: X (i) (i) Lact = ∥F̂t − Ft ∥22 . (6) i∈B

Combining the three objectives yields the full pre-training objective LDynaFLIP = Lalign + λtcn Ltcn + λact Lact ,

(7)

where λtcn and λact control the relative importance of the two auxiliary objectives. 2.3

Dataset Construction

Our pre-training framework relies only on RGB videos. Although the training objective uses image– language–3D flow triplets, all three signals can be derived from video alone: image transitions are obtained by sampling frames, 3D flow trajectories are estimated through point tracking and depth estimation while compensating for camera motion, and language instructions are generated by a vision-language model. This video-only requirement enables pre-training to scale across both human and robot videos. Building on the unified data generation pipeline of [32] with several modifications tailored to our setting, we construct a large-scale dataset comprising 260K trajectories, each paired with image–language–3D flow triplets. The dataset is built from heterogeneous human and robot video sources [4, 16, 17, 29, 40, 49, 3, 27], providing broad diversity in objects, environments, and interaction patterns. Additional details on data sources, statistics, and generation procedures are provided in Appendix C.

3

Experiments

In this section, we evaluate DynaFLIP through extensive experiments in both simulation and the real world. Through these experiments, we aim to answer the following questions: Q1: Does DynaFLIP learn dynamics-aware representations that preserve control-relevant information for manipulation? Q2: Do dynamics-aware representations improve downstream policy learning compared to strong baselines? Q3: Can DynaFLIP improve real-world manipulation under both in-distribution and out-ofdistribution settings? Q4: Which design choices in DynaFLIP are most critical to its performance? 3.1

Benchmarks and Baselines

Benchmarks. We evaluate DynaFLIP on three simulation benchmarks and three real-world manipulation tasks. MetaWorld [56] uses a Sawyer arm with a two-finger gripper. We evaluate 15 tasks spanning varying difficulty levels [45] with 25 demonstrations per task. RLBench [23] employs a 5

DINOv2

LIV

DynaFLIP (Ours)

DynaFLIP (Ours) LIV VC-1

R3M R3M

SigLIP

DINOv2

SigLIP

CLIP VC-1

CLIP

Figure 4: Control-relevant score versus downstream success rate (MLP policy). The control-relevant score Sm [13] (x-axis) measures how well a frozen image encoder preserves state information relevant to control, and the y-axis reports policy success rate on MetaWorld [56] (left) and RLBench [23] (right). DynaFLIP appears in the top-right region of both plots, indicating its dynamics-aware representations preserve control-relevant information and improve manipulation performance. Image

R3M

CLIP

DINOv2

SigLIP

DynaFLIP

Image

(a) Grad-CAM heatmaps over action prediction.

R3M

CLIP

DINOv2

SigLIP

DynaFLIP

(b) Feature visualization with PCA.

Figure 5: Grad-CAM and PCA visualizations (MLP policy). (a) Grad-CAM heatmaps show that DynaFLIP attends to manipulated objects and interaction regions, whereas baselines often focus on task-irrelevant areas. (b) PCA visualizations show that DynaFLIP yields more spatially coherent, object-level feature structures than the baselines. Additional visualizations are provided in Appendix E.2 and Appendix E.3.

Franka Panda arm. We evaluate 6 tasks from front-view observations with 100 demonstrations per task collected via the Open Motion Planning Library [48]. LIBERO [33] is a multi-task, languageconditioned manipulation benchmark. We evaluate on LIBERO-90, LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long, where LIBERO-90 contains 90 tasks and each remaining suite contains 10 tasks with 50 demonstrations per task. Real-World Manipulation experiments use a UR3 robot arm equipped with a two-finger gripper. We consider two multi-instruction tasks, Pick <object> into Sink and Pour almonds into <object>, together with an Unfold Towel task. Baselines. We compare DynaFLIP with strong pre-trained representation baselines from three categories: robotic visual representations, self-supervised visual encoders, and vision-language pre-training models. Among robotic visual representations, R3M [37] trains a ResNet [19] on human videos via time-contrastive learning and video-language alignment. VC-1 [36] pre-trains a ViT [14] with Masked Auto-Encoding [20] on navigation and ImageNet [12] data. LIV [34] trains a ResNet on human videos by aligning goal images with language and modeling rewards relative to goal states. As a self-supervised visual encoder, DINOv2 [39] combines self-distillation with masked image modeling on large-scale curated image data. Among vision-language models, CLIP [42] and SigLIP [57] learn image-text alignment on large-scale paired data, with SigLIP replacing CLIP’s multinomial cross-entropy objective with a pairwise sigmoid loss. 3.2

Q1: Does DynaFLIP learn dynamics-aware and control-relevant representations?

Experiment setup. We first verify our central claim that DynaFLIP’s pre-training yields dynamicsaware representations that preserve control-relevant information. We analyze pre-trained image encoders on MetaWorld and RLBench: each encoder remains frozen, and only a lightweight threelayer MLP policy is trained on top, ensuring that downstream performance reflects representation quality rather than policy capacity. Appendix D.2 describes the training and evaluation protocols for MetaWorld and RLBench. Quantitative analysis. We measure how well each encoder preserves control-relevant information using the control-relevant score (Sm ) proposed in [13], which quantifies how well a visual representation captures information needed for control. This score is computed by training a lightweight

6

Table 1: LIBERO benchmark results (Diffusion policy). We evaluate various pre-trained encoders under two settings: Frozen keeps both image and language encoders frozen, while LoRA Fine-tuned adapts both encoders jointly with the diffusion policy as an additional comparison. The evaluation metric is success rate (%). Bold and underline numbers indicate the best and second-best results in each column, respectively. Image Encoder

Language Encoder

R3M [37] VC-1 [36] LIV [34] CLIP [42] DINOv2 [39] SigLIP [57] DynaFLIP (Ours)

CLIP [42] CLIP [42] LIV [34] CLIP [42] CLIP [42] SigLIP [57] DynaFLIP (Ours)

Frozen

LoRA Fine-tuned

90

Goal Object Spatial Long Mean

90

Goal Object Spatial Long Mean

24.4 12.8 22.3 13.8 14.4 24.3 31.7

45.0 52.5 64.0 38.5 75.0 54.5 70.5

53.0 52.0 51.0 50.0 42.5 52.0 51.5

38.5 72.4 72.7 78.1 83.6 82.6 78.1

67.0 83.0 78.5 79.5 77.5 80.5 84.5

0.5 11.5 6.5 1.5 33.5 13.0 37.5

13.5 12.5 9.0 9.5 20.5 8.5 16.5

27.3 28.3 30.6 22.7 37.2 30.5 41.5

2.5 83.5 49.0 79.0 82.0 82.0 83.5

56.5 71.0 75.5 75.5 81.0 74.0 78.5

37.5 62.0 62.0 68.5 67.5 76.5 80.5

40.4 74.4 67.5 76.1 78.3 79.1 81.0

probe on top of the frozen image encoder to predict robot joint angles, end-effector pose, and the 6D pose and shape of task-relevant objects; Appendix D.5 provides the formal definition and evaluation protocol. Figure 4 plots the control-relevant score (Sm ) against downstream success rate on MetaWorld and RLBench. DynaFLIP lies in the top-right region of both plots, achieving the highest downstream success rate with high control-relevant scores. This result indicates that DynaFLIP preserves control-relevant information more faithfully, leading to higher downstream success rates. Qualitative analysis. We further inspect the learned representations through two visualizations. (1) Grad-CAM [44], applied to the trained MLP policy with negative action-prediction error as the target, highlights the visual regions most influential for action prediction. (2) PCA on patch features examines the overall structure of the learned feature space. Figure 5 shows that DynaFLIP concentrates attention on task-relevant objects and interaction regions, whereas baselines distribute attention over less relevant areas such as the background or irrelevant objects. PCA visualizations further show that DynaFLIP produces a more spatially coherent and object-aware feature structures than the baselines. Together, the quantitative and qualitative results show that DynaFLIP learns dynamics-aware representations that preserve control-relevant information and focus on regions critical for manipulation. 3.3

Q2: Do DynaFLIP’s representations improve downstream policy learning?

Experiment setup. We next ask whether dynamics-aware representations improve downstream policy learning. We evaluate on the LIBERO benchmark (LIBERO-90, Goal, Object, Spatial, and Long) using Diffusion Policy [9] as the imitation-learning backbone. Each setup pairs a pre-trained image encoder with a language encoder; for baselines without their own text encoder, we substitute CLIP’s text encoder. Our primary setting is frozen: both encoders remain fixed, so downstream performance directly reflects the quality and reusability of the pre-trained representations. We additionally report a fine-tuned setting, in which LoRA [21] adapters on both encoders are trained jointly with the diffusion policy. Appendix D.3 provides detailed training settings and evaluation protocols. Results. Table 1 reports the LIBERO results. DynaFLIP achieves the highest mean success rate in both the frozen and fine-tuned settings, outperforming all baselines. (1) The frozen-setting results show that DynaFLIP’s pre-trained features can be reused effectively without encoder adaptation. (2) The fine-tuned setting further confirms that this advantage persists after task-specific adaptation. We attribute this consistent advantage to differences in pre-training paradigms. Most baselines are trained primarily on static visual data and therefore receive limited signal about how scenes evolve under interaction. In contrast, DynaFLIP explicitly aligns three transition-centric modalities—image transitions, language, and 3D flow trajectories—encouraging the encoder to focus on control-relevant regions rather than background appearance. 3.4

Q3: Does DynaFLIP improve real-world manipulation under distribution shift?

Experiment setup. We evaluate DynaFLIP in real-world manipulation by integrating a frozen pretrained image encoder into π0.5 [22], a vision-language-action (VLA) model. We adopt a lightweight visual-injection design similar to plug-in visual injection (PVI) [60]: an additional visual branch encodes features from the pre-trained image encoder, and an injection module projects them into the hidden feature space of the diffusion transformer of π0.5 . The additional visual branch remains frozen

7

Out-of-distribution tasks

In-distribution tasks

R3M

CLIP

DINOv2

Pick <object> into Sink

SigLIP

DynaFLIP (Ours)

Pour almonds into <object>

Unseen object position, distractor

Unseen object, instruction

Visual, Spatial Perturbations

Semantic Perturbations

Unfold Towel

Figure 6: Real-world manipulation results (VLA policy). DynaFLIP performs well not only on the three indistribution tasks, but also under both out-of-distribution perturbation types. The top row contrasts in-distribution (seen) tasks with out-of-distribution (unseen) evaluation settings, and the bottom row reports success rates (%) on the three in-distribution tasks together with two out-of-distribution settings.

and only the lightweight injection module is trained, testing whether DynaFLIP can be reused inside a VLA without end-to-end visual fine-tuning. We evaluate on a UR3 robot arm with a two-finger gripper across three in-distribution tasks: Pick <object> into Sink, Pour almonds into <object>, and Unfold Towel. For out-of-distribution (OOD) evaluation, we introduce two types of perturbations: visual and spatial perturbations (unseen object positions and distractors) and semantic perturbations (unseen objects and instructions). Appendix D.4 provides additional details on the hardware setup, data collection, model architecture, and training and evaluation protocol. Results. (1) In-Distribution. Figure 6 shows that DynaFLIP achieves the highest success rates across all three in-distribution tasks. Together with the frozen results on MetaWorld, RLBench, and LIBERO, this demonstrates that DynaFLIP transfers robustly across diverse downstream policies— MLP, diffusion policy, and VLA—without task-specific visual fine-tuning. (2) Out-of-Distribution. The advantage of DynaFLIP becomes even more pronounced under OOD settings. Under visual and spatial perturbations, CLIP and SigLIP often fail at precise grasping, while DynaFLIP’s focus on control-relevant regions enables it to remain robust to changes in object layouts and the presence of distractors. Under semantic perturbations, DINOv2 frequently interacts with objects irrelevant to the instruction, reflecting its lack of direct language grounding. By contrast, DynaFLIP incorporates language as one of its pre-training modalities and learns to align visual changes with task-relevant instructions, yielding representations that remain robust under unseen objects and instructions. 3.5

Q4: Which design choices of DynaFLIP matter most?

Table 2 presents ablations of our system with respect to four aspects: multimodal input, alignment design, optimization-pitfall mitigation, and auxiliary objectives. All three modalities are necessary for dynamics-aware representation learning. Removing either 3D flow or language causes a clear drop in performance. 3D flow provides explicit motion cues, while language supplies task-level semantics. Both contribute complementary signals beyond image transition alone. Alignment design matters more than simply adding modalities. Replacing the simplex-guided alignment with an anchor-based pairwise loss causes a substantial degradation. This result shows that DynaFLIP’s gains stem not merely from using multiple modalities, but from how those modalities are aligned through higher-order multimodal geometry rather than pairwise similarity.

8

Table 2: Ablation studies (Diffusion policy). We report mean success rate (%) averaged over LIBERO-Goal, Object, Spatial, and Long, with both image and language encoders frozen. (a) Multimodal input.

(b) Alignment design.

(c) Optimization pitfalls.

(d) Auxiliary objectives.

Variant

Mean

Variant

Mean

Variant

Mean

Variant

Mean

w/o. 3D flow w/o. Language

37.1 35.4

Anchor-based alignment

31.8

w/o. Negative tuples 18.1 w/o. Cosine reg. 39.8

w/o. Lact w/o. Ltcn

43.4 39.6

DynaFLIP (full)

44.0

DynaFLIP (full)

44.0

DynaFLIP (full)

DynaFLIP (full)

44.0

44.0

Mitigating optimization pitfalls is crucial for stable learning. Removing the contrastive framework—i.e., directly minimizing the joint alignment energy (Eq. (2)) without negative tuples— causes the most severe drop, confirming that the contrastive framework is necessary to prevent trivial collapse. Removing the cosine regularizer also reduces performance, supporting its role in mitigating geometric ambiguity. Geometric degeneracy is a theoretical possibility rather than a guaranteed failure mode; even when it does not occur, the cosine regularizer still improves performance by stabilizing positive alignment gradients. A detailed analysis is provided in Appendix B.4. Auxiliary objectives provide additional gains. Removing either auxiliary loss degrades performance. The larger drop from removing Ltcn confirms its complementary role: it captures trajectory-level temporal structure beyond the transition window covered by Lalign .

4

Related work

Visual Representations for Robotic Manipulation. Visual foundation models have driven progress in robot policy learning, mainly through two paradigms: self-supervised visual pre-training [18, 6, 7, 39] and contrastive vision-language pre-training [26, 42, 8, 57]. Self-supervised models such as DINOv2 [39] learn spatially precise features that capture both global context and local detail, but lack a direct interface to language, limiting their use in open-vocabulary settings and instruction-following robots. Contrastive vision-language models such as CLIP [42] and SigLIP [57] learn semantically aligned representations from large-scale paired data, supporting strong zero-shot generalization, but lack the fine-grained spatial reasoning needed for manipulation [25]. Both paradigms, however, learn primarily from static data and therefore lack dynamics awareness. This limitation matters for manipulation, where success depends on how scenes change under interaction, not only on object and instruction recognition. DynaFLIP addresses this gap by aligning three transition-centric modalities—image transitions, language, and 3D flow. These signals allow the encoder to focus on control-relevant regions rather than visually salient but task-irrelevant areas. A separate line of work develops pre-training objectives specifically for robotic representations, ranging from single-modality self-supervised objectives [53, 36, 35, 47] to multimodal alignment with language, action, or robot proprioception [37, 34, 24, 52]. However, none of these approaches jointly align all three modalities. Our method instead aligns image transitions, language, and 3D flow through a simplex-based formulation, enabling mutual alignment among all three modalities. Detailed comparison with these prior works is provided in Appendix A.1.

5

Conclusion

We present DynaFLIP, a dynamics-aware 3D flow-language-image pre-training framework that pushes motion understanding upstream into perception. By jointly aligning image transitions, language, and 3D flow through a simplex-based formulation—augmented with a cosine regularizer and a contrastive framework to resolve optimization pitfalls—DynaFLIP learns visual representations that focus on control-relevant regions. Across simulated and real-world manipulation, DynaFLIP transfers strongly as a reusable visual backbone and consistently outperforms baselines, with especially large gains under visual, spatial, and semantic distribution shifts. Our results indicate that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action. Limitations and future work. First, DynaFLIP is pre-trained on 260K trajectories, which is smaller than the data scales used by several large-scale visual and vision-language baselines [39, 57, 37].

9

Scaling DynaFLIP to larger human and robot video corpora is a promising direction for future work. Second, our 3D flow is extracted from a uniform 20 × 20 grid of keypoints, which captures all motion in the scene after compensating for camera motion—including task-irrelevant motion. As a result, pre-training videos containing task-irrelevant motion may inject noisy supervision into the representation; future work could explore keypoint sampling focused on the agent and task-relevant objects to mitigate this issue.

Acknowledgments This work was supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT2402-17. Lee and Huang are supported by DARPA HR001124S0029AIQ-FP-019, National Science Foundation TRAILS Institute (2229885). Private support was provided by Open Philanthropy and Apple.

10

References [1] Roy Betser, Eyal Gofer, Meir Yossef Levi, and Guy Gilboa. Infonce induces gaussian distribution. In International Conference on Learning Representations (ICLR), 2026. [2] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. π0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024. [3] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. Rt-1: Robotics transformer for real-world control at scale. In Robotics: Science and Systems (RSS), 2023. [4] Qingwen Bu, Jisong Cai, Li Chen, Xiuqi Cui, Yan Ding, Siyuan Feng, Shenyuan Gao, Xindong He, Xuan Hu, Xu Huang, et al. Agibot world colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2025. [5] Qingwen Bu, Yanting Yang, Jisong Cai, Shenyuan Gao, Guanghui Ren, Maoqing Yao, Ping Luo, and Hongyang Li. Univla: Learning to act anywhere with task-centric latent actions. In Robotics: Science and Systems (RSS), 2025. [6] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Neural Information Processing Systems (NeurIPS), 2020. [7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In International Conference on Computer Vision (ICCV), 2021. [8] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. [9] Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025. [10] Giordano Cicchetti, Eleonora Grassucci, and Danilo Comminiello. A triangle enables multimodal alignment beyond cosine similarity. In Neural Information Processing Systems (NeurIPS), 2025. [11] Giordano Cicchetti, Eleonora Grassucci, Luigi Sigillo, and Danilo Comminiello. Gramian multimodal representation learning and alignment. In International Conference on Learning Representations (ICLR), 2025. [12] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition (CVPR), 2009. [13] Jiahua Dong, Yunze Man, Pavel Tokmakov, and Yu-Xiong Wang. Capturing visual environment structure correlates with control performance. In International Conference on Learning Representations (ICLR), 2026. [14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. [15] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

11

[16] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In International Conference on Computer Vision (ICCV), 2017. [17] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. [18] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In Neural Information Processing Systems (NeurIPS), 2020. [19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016. [20] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. [21] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. [22] Physical Intelligence, Kevin Black, Noah Brown, James Darpinian, Karan Dhabalia, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, et al. π0.5 : a visionlanguage-action model with open-world generalization. In Conference on Robot Learning (CoRL), 2025. [23] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2): 3019–3026, 2020. [24] Guangqi Jiang, Yifei Sun, Tao Huang, Huanyu Li, Yongyuan Liang, and Huazhe Xu. Robots pre-train robots: Manipulation-centric robotic representation from large-scale robot datasets. In International Conference on Learning Representations (ICLR), 2025. [25] Cijo Jose, Théo Moutakanni, Dahyun Kang, Federico Baldassarre, Timothée Darcet, Hu Xu, Daniel Li, Marc Szafraniec, Michaël Ramamonjisoa, Maxime Oquab, et al. Dinov2 meets text: A unified framework for image-and pixel-level vision-language alignment. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [26] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data. In European Conference on Computer Vision (ECCV), 2016. [27] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on robot learning (CoRL), 2018. [28] Nikita Karaev, Yuri Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. In International Conference on Computer Vision (ICCV), 2025. [29] Alexander Khazatsky, Karl Pertsch, Suraj Nair, Ashwin Balakrishna, Sudeep Dasari, Siddharth Karamcheti, Soroush Nasiriany, Mohan Kumar Srirama, Lawrence Yunliang Chen, Kirsty Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024. [30] Moo Jin Kim, Chelsea Finn, and Percy Liang. Fine-tuning vision-language-action models: Optimizing speed and success. In Robotics: Science and Systems (RSS), 2025. 12

[31] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan P Foster, Pannag R Sanketi, Quan Vuong, et al. Openvla: An open-source vision-language-action model. In Conference on Robot Learning (CoRL), 2025. [32] Seungjae Lee, Yoonkyo Jung, Inkook Chun, Yao-Chih Lee, Zikui Cai, Hongjia Huang, Aayush Talreja, Tan Dat Dao, Yongyuan Liang, Jia-Bin Huang, et al. Tracegen: World modeling in 3d trace space enables learning from cross-embodiment videos. In Conference on Computer Vision and Pattern Recognition (CVPR), 2026. [33] Bo Liu, Yifeng Zhu, Chongkai Gao, Yihao Feng, Qiang Liu, Yuke Zhu, and Peter Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. In Neural Information Processing Systems (NeurIPS), 2023. [34] Yecheng Jason Ma, Vikash Kumar, Amy Zhang, Osbert Bastani, and Dinesh Jayaraman. Liv: Language-image representations and rewards for robotic control. In International Conference on Machine Learning (ICML), 2023. [35] Yecheng Jason Ma, Shagun Sodhani, Dinesh Jayaraman, Osbert Bastani, Vikash Kumar, and Amy Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. In International Conference on Learning Representations (ICLR), 2023. [36] Arjun Majumdar, Karmesh Yadav, Sergio Arnaud, Jason Ma, Claire Chen, Sneha Silwal, Aryan Jain, Vincent-Pierre Berges, Tingfan Wu, Jay Vakil, et al. Where are we in the search for an artificial visual cortex for embodied intelligence? In Neural Information Processing Systems (NeurIPS), 2023. [37] Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. In Conference on Robot Learning (CoRL), 2023. [38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, 2024. [40] Abby O’Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open xembodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In IEEE International Conference on Robotics and Automation (ICRA), 2024. [41] Han Qi, Haocheng Yin, and Heng Yang. Control-oriented clustering of visual latent representation. In International Conference on Learning Representations (ICLR), 2025. [42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), 2021. [43] Ludan Ruan, Anwen Hu, Yuqing Song, Liang Zhang, Sipeng Zheng, and Qin Jin. Accommodating audio modality in clip for multimodal processing. In AAAI Conference on Artificial Intelligence (AAAI), 2023. [44] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In International Conference on Computer Vision (ICCV), 2017. [45] Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter Abbeel. Masked world models for visual control. In Conference on Robot Learning (CoRL), 2023.

13

[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR), 2021. [47] Mohan Kumar Srirama, Sudeep Dasari, Shikhar Bahl, and Abhinav Gupta. Hrp: Human affordances for robotic pre-training. In Robotics: Science and Systems (RSS), 2024. [48] Ioan A Sucan, Mark Moll, and Lydia E Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72–82, 2012. [49] Homer Rich Walke, Kevin Black, Tony Z Zhao, Quan Vuong, Chongyi Zheng, Philippe HansenEstruch, Andre Wang He, Vivek Myers, Moo Jin Kim, Max Du, et al. Bridgedata v2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023. [50] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny. Vggt: Visual geometry grounded transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), 2025. [51] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International conference on machine learning (ICML), 2020. [52] Wuding Weng, Tongshu Wu, Liucheng Chen, Siyu Xie, Zheng Wang, Xing Xu, Jingkuan Song, and Heng Tao Shen. Language-grounded decoupled action representation for robotic manipulation. arXiv preprint arXiv:2603.12967, 2026. [53] Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022. [54] Yuxi Xiao, Jianyuan Wang, Nan Xue, Nikita Karaev, Yuri Makarov, Bingyi Kang, Xing Zhu, Hujun Bao, Yujun Shen, and Xiaowei Zhou. Spatialtrackerv2: 3d point tracking made easy. In International Conference on Computer Vision (ICCV), 2025. [55] Wenzhe Yin, Pan Zhou, Zehao Xiao, Jie Liu, Shujian Yu, Jan-Jakob Sonke, and Efstratios Gavves. Towards uniformity and alignment for multimodal representation learning. arXiv preprint arXiv:2602.09507, 2026. [56] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning (CoRL), 2020. [57] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In International Conference on Computer Vision (ICCV), 2023. [58] Bowei Zhang, Lei Ke, Adam W Harley, and Katerina Fragkiadaki. Tapip3d: Tracking any point in persistent 3d geometry. In Neural Information Processing Systems (NeurIPS), 2025. [59] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In International Conference on Computer Vision (ICCV), 2023. [60] Zezhou Zhang, Songxin Zhang, Xiao Xiong, Junjie Zhang, Zejian Xie, Jingyi Xi, Zunyao Mao, Zan Mao, Zhixin Mai, Zhuoyang Song, et al. Pvi: Plug-in visual injection for vision-languageaction models. arXiv preprint arXiv:2603.12772, 2026. [61] Ruijie Zheng, Yongyuan Liang, Shuaiyi Huang, Jianfeng Gao, Hal Daumé III, Andrey Kolobov, Furong Huang, and Jianwei Yang. Tracevla: Visual trace prompting enhances spatial-temporal awareness for generalist robotic policies. In International Conference on Learning Representations (ICLR), 2025. [62] Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In International Conference on Learning Representations (ICLR), 2024.

14

Appendix A Additional Related Works

16

A.1 Pre-training Objectives for Robotic Representations . . . . . . . . . . . . . . . . . B Mathematical Proofs and Theoretical Details

16 16

B.1 Generalized Simplex Volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

16

B.2 Contrastive Learning with Simplex-Guided Energy . . . . . . . . . . . . . . . . .

16

B.3 Why Simplex-Volume Alone is Insufficient . . . . . . . . . . . . . . . . . . . . .

17

B.3.1 Ambiguity of Low-Volume Configurations . . . . . . . . . . . . . . . . .

17

B.3.2

Conflicting Alignment Directions in Volume-Induced Gradients . . . . . .

17

B.4 Mitigating Volume-Only Pitfalls with Cosine Regularization . . . . . . . . . . . .

18

C Dataset Construction

19

C.1 Dataset Composition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

C.2 Dataset Generation Pipeline . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

19

D Experiment Details

20

D.1 Pre-training DynaFLIP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

D.2 MetaWorld and RLBench . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

21

D.3 LIBERO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

22

D.4 Real-world Robot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

23

D.5 Control-Relevant Metric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

27

E Additional Experimental Results

27

E.1 LIBERO Results with Paired Image Encoders . . . . . . . . . . . . . . . . . . . .

27

E.2 Grad-CAM visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

28

E.3 PCA visualizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

30

15

A

Additional Related Works

A.1

Pre-training Objectives for Robotic Representations

A growing body of work has developed pre-training objectives specifically tailored for robotic representations. Early efforts applied single-modality self-supervised objectives over static images: MVP [53] and VC-1 [36] utilize Masked Autoencoder (MAE) on large-scale human datasets to learn visual features, VIP [35] learns implicit value functions to encode distance-to-goal representations, and HRP [47] extracts human affordances from videos. Another line of work introduces multimodal supervision: R3M [37] and LIV [34] align images with language descriptions, while MCR [24] aligns images with robot action and proprioceptive trajectories; these methods use only two modalities, and MCR additionally requires robot-specific signals that prevent direct use of human videos. LaDA [52] extends to three modalities by aligning concatenated image-language features with action embeddings via contrastive learning for VLA training, but this design treats language as an auxiliary input to the image branch rather than aligning all three modalities jointly. In contrast, our method aligns image transitions, language, and 3D flow jointly through a simplex-based formulation, enabling mutual alignment among all three modalities.

B

Mathematical Proofs and Theoretical Details

This section provides the theoretical details of the proposed simplex-guided contrastive objective in DynaFLIP. The main paper identifies two optimization pitfalls of naive simplex-volume minimization: geometric ambiguity and trivial collapse, addressed through a cosine regularizer and a contrastive framework, respectively. In this section, we focus on the analysis underlying the cosine regularizer: we show that, beyond geometric ambiguity, naive volume minimization suffers from an additional issue—conflicting alignment gradients—and that the cosine regularizer mitigates both. We organize the analysis into four parts. Section B.1 introduces the generalized simplex volume, which reduces to the triangle area in the three-modal setting used by DynaFLIP. Section B.2 shows that our objective retains the standard energy-based contrastive learning structure and only modifies the geometry of the positive alignment energy. Section B.3 then analyzes the volume-only objective and identifies the two optimization pitfalls: geometric ambiguity and conflicting alignment gradients. Finally, Section B.4 explains how the cosine regularizer mitigates both pitfalls by introducing explicit pairwise attraction between selected modality embeddings. B.1

Generalized Simplex Volume

For an m-modal tuple, let z1 , . . . , zm ∈ Rd be ℓ2 -normalized modality embeddings, and define U = [z2 − z1 , . . . , zm − z1 ] ∈ Rd×(m−1) . Let G = U ⊤ U ∈ R(m−1)×(m−1) be the Gram matrix of the simplex edge vectors. The generalized simplex volume is defined as p 1 det(G). (8) Vm (z1 , . . . , zm ) = (m − 1)! A smaller value of Vm indicates that the modality embeddings form a lower-volume configuration in the shared latent space, reflecting stronger joint alignment across modalities. In the three-modal setting used by DynaFLIP, this quantity reduces to the triangle area defined in Eq. (1), and we focus on this case in the analyses that follow. B.2

Contrastive Learning with Simplex-Guided Energy

We recall the joint alignment energy function in Eq. (2): E(zL , zI , zF ) = A(zL , zI , zF ) − α⟨zL , zF ⟩, where A denotes the triangle area defined in Eq. (1) and α ≥ 0 balances triangle-area minimization and pairwise cosine alignment. 16

i For each matched tuple (zL , zIi , zFi ), let N (i) denote a set of mismatched negative tuples constructed i by mismatching one or more modality embeddings across the batch. We define Ei+ = E(zL , zIi , zFi ) − i i i for the matched tuple and Eiℓ = E(z̃L , z̃I , z̃F ) for each negative tuple ℓ, and incorporate the energy into an InfoNCE-style contrastive objective [38]:

exp(−Ei+ /τ ) . P − exp(−Ei+ /τ ) + ℓ exp(−Eiℓ /τ )

Li = − log

(9)

− Let p+ i and piℓ denote the corresponding softmax probabilities. Differentiating the loss yields:

∇Li =

X p− 1 − p+ − i iℓ ∇Ei+ − ∇Eiℓ . τ τ {z } | ℓ {z } | Alignment Term

(10)

Uniformity Term

The alignment term decreases the energy of the matched tuple, while the uniformity term increases the energy of mismatched tuples. In this work, we focus on the alignment term, which directly captures the effect of the proposed energy on matched multimodal tuples. The uniformity term follows the standard contrastive repulsion mechanism, and we refer interested readers to prior analyses [51, 55, 1]. Substituting E into the alignment term gives: i i ∇Ei+ = ∇A(zL , zIi , zFi ) − α∇⟨zL , zFi ⟩.

(11)

Therefore, the alignment gradient is a linear combination of a triangle-area term and a cosine-based pairwise attraction term, capturing higher-order geometry and directional consistency. While this decomposition reveals the structure of the alignment gradient, it also suggests that the volume term alone may not provide a reliable alignment signal. We analyze this issue in the following section. B.3

Why Simplex-Volume Alone is Insufficient

We analyze the simplex-volume alignment in the three-modal case, where the objective reduces to the triangle area defined in Eq. (1). This setting allows us to characterize the alignment gradient and reveal two key limitations: ambiguity of low-volume configurations and conflicting alignment gradients. B.3.1

Ambiguity of Low-Volume Configurations

Low simplex volume does not necessarily imply pairwise alignment among all modalities. The simplex volume can vanish even when some modality pairs remain far apart—for example, when a subset of embeddings collapses together, or when all embeddings become nearly collinear. For example, we illustrate this in the three-modal case. Consider unit vectors in R2 : x = e1 ,

y = −e1 ,

z = cos θ e1 + sin θ e2 ,

(12)

As θ → 0, we have z → x, so the pair (x, z) collapses and the triangle area satisfies A(x, y, z) = | sin θ| → 0. However, the pair (x, y) remains maximally misaligned, with ⟨x, y⟩ = −1, ∥x − y∥ = 2. This example shows that the simplex volume can be minimized by collapsing only a subset of modalities. Therefore, low volume does not guarantee pairwise alignment among all modalities. B.3.2

Conflicting Alignment Directions in Volume-Induced Gradients

The volume-induced alignment does not define a single direction. Instead, it decomposes into multiple edge-wise pulls that can conflict with each other. Let x, y, z ∈ Sd−1 denote unit-normalized embeddings from three modalities, and define: a = ⟨x, y⟩,

b = ⟨x, z⟩,

c = ⟨y, z⟩.

The triangle area A(x, y, z) can be expressed as a function of these pairwise inner products. For a non-degenerate triangle, the gradient with respect to modality x decomposes as: ∇x A =

∂A ∂A ∇x a + ∇x b, ∂a ∂b 17

(13)

Since the embeddings lie on the unit sphere, the gradients are taken in the tangent space: ∇x a = ∇x ⟨x, y⟩ = Px⊥ y,

∇x b = ∇x ⟨x, z⟩ = Px⊥ z,

(14)

where Px⊥ v = v − ⟨x, v⟩x. Thus, the gradient becomes: ∇x A = ωxy Px⊥ y + ωxz Px⊥ z,

ωxy =

∂A , ∂a

ωxz =

∂A . ∂b

(15)

Under gradient descent, the update direction is −∇x A, which decomposes into two edge-wise positive pulls: uxy = −ωxy Px⊥ y, uxz = −ωxz Px⊥ z. (16) The total volume-induced alignment pull on modality x then becomes: uvol x = uxy + uxz .

(17)

This shows that the volume-induced alignment is not governed by a single direction but by the sum of multiple edge-wise pulls. When these pulls are aligned, they reinforce each other. However, when they point in different directions, they partially cancel, leading to a weaker effective update. B.4

Mitigating Volume-Only Pitfalls with Cosine Regularization

The cosine regularizer complements the simplex-volume term by adding explicit pairwise alignment constraints. This additional pairwise signal mitigates the two pitfalls identified in Section B.3: geometrically ambiguous low-volume configurations and conflicting alignment gradients. Reducing low-volume ambiguity. The cosine regularizer also reduces low-volume ambiguity by introducing an explicit distance-based penalty between selected modality pairs. For unit-normalized embeddings x and y, we have: 1 (18) 1 − ⟨x, y⟩ = ∥x − y∥2 . 2 Thus, the cosine term directly penalizes large distances between modality embeddings. Under the combined objective
α A(x, y, z) + α 1 − ⟨x, y⟩ = A(x, y, z) + ∥x − y∥2 , (19) 2 the area term encourages a low-volume configuration, while the cosine term discourages distant modality pairs. As a result, configurations with low volume but large pairwise distances become less favorable. Reducing conflict in volume-based gradients. We consider the three-modal setting and recall the cosine-regularized energy: E(x, y, z) = A(x, y, z) − α⟨x, y⟩, (20) where A(x, y, z) denotes the triangle-area term. For the anchor modality x, the volume-only update direction is given by −∇x A = uxy + uxz , which decomposes into two edge-wise pulls that may partially cancel. With cosine regularization, the update becomes: −∇x E = −∇x A + αPx⊥ y = uxy + uxz + αPx⊥ y,

(21)

where Px⊥ y = y − ⟨x, y⟩x. The additional term αPx⊥ y introduces an explicit pairwise alignment direction. To see this, consider a small step δx = ηPx⊥ y with η > 0. Then d ⟨x + ηPx⊥ y, y⟩ = ⟨Px⊥ y, y⟩ = 1 − ⟨x, y⟩2 ≥ 0. dη

(22)

Thus, Px⊥ y is an ascent direction for ⟨x, y⟩, meaning that the cosine term directly increases the similarity between the selected pair. As a result, even when the volume-induced edge-wise pulls partially cancel, the cosine regularizer preserves a non-vanishing pairwise alignment signal for the selected modalities. In summary, the cosine regularizer complements the simplex-volume term by resolving its inherent ambiguities. While the volume term captures higher-order geometric structure, the cosine term introduces explicit pairwise constraints that prevent degenerate low-volume configurations and maintain a meaningful alignment signal. 18

C

Dataset Construction

In this section, we provide additional details on the construction of our image–language–3D flow dataset. We first describe the dataset composition across heterogeneous human and robot video sources. We then present the generation pipeline that converts raw videos into image–language–3D flow triplets. C.1

Dataset Composition

Our dataset is constructed from heterogeneous human and robot video sources in order to cover a broad range of objects, environments, camera viewpoints, and manipulation styles. As summarized in Figure 7, the final dataset contains 260K trajectories in total: 190K from robot demonstrations and 70K from human videos. The robot portion combines AgiBot [4] (135K), Droid [29] (20K), Open X-Embodiment [40] (17K), and BridgeData V2 [49] (18K). The human portion combines Ego4D [17] (35K) and Something-Something V2 [16] (35K).

Figure 7: Composition of pre-training dataset. The dataset contains 260K image–language–3D flow triplets in total, combining 190K trajectories from robot videos and 70K from human videos.

C.2

Dataset Generation Pipeline

We follow the unified data generation pipeline of TraceForge [32] with several modifications tailored to our setting. As illustrated in Figure 8, our pipeline converts raw videos into aligned image– language–3D flow triplets. Compared with the original pipeline, we omit event chunking and speed retargeting, and instead directly sample frames from each video so that the effective temporal resolution is approximately matched across datasets collected at different frame rates while preserving the original motion timing. Language instruction generation. For each sampled trajectory, we generate language instructions that describe the underlying manipulation intent using a vision-language model (VLM). The VLM takes as input a small set of representative frames sampled from the trajectory together with a prompt asking it to describe the task in three forms: a short imperative instruction, a detailed natural-language description, and a multi-step instruction that decomposes the task into sequential subgoals. 3D flow generation. For each sampled trajectory, we construct a 3D flow that remains consistent under moving camera viewpoints. We select the reference frame from the early part of the trajectory, since the first frame may not always contain the robot or human demonstrator. On this reference frame, we place a uniform 20 × 20 grid of keypoints and track them throughout the trajectory. Rather than representing motion in full camera coordinates, we represent each tracked point as (x, y, z), where (x, y) denotes the image-plane coordinates and z denotes the corresponding depth.

19

Video

Task instruction generation

Language instructions

Camera pose & Depth Estimation

Camera pose

Frame sampling

Depth

Images

2D Point Tracking

Unprojection & World-to-camera transformation

3D Flow

2D Flow

Figure 8: Dataset generation pipeline. Each raw video is first frame-sampled to obtain image observations. Three parallel branches then process these images: (i) a VLM generates language instructions describing the manipulation intent, (ii) per-frame camera pose and depth are estimated using SpatialTrackerV2 [54], and (iii) 2D points are tracked across frames using CoTracker3 [28]. Tracked 2D points are unprojected with the estimated depth and transformed into the reference camera coordinate frame to produce 3D flow trajectories that are invariant to camera motion.

This representation preserves spatial alignment with the original image while retaining physically meaningful motion in 3D space. To obtain the required 3D information, we estimate camera pose, depth, and point trajectories for every frame in the sampled trajectory. We use TAPIP3D [58] for 3D flow construction, CoTracker3 [28] for point tracking, and a fine-tuned VGGT [50] model from SpatialTrackerV2 [54] for efficient depth and camera-pose prediction. Given a trajectory, these models produce per-frame depth maps, camera poses, and tracked 2D keypoint trajectories. We then unproject the tracked points with the predicted depth to reconstruct their 3D trajectories over time. To compensate for camera motion, we express all reconstructed 3D flow in the coordinate system of the reference camera frame. Specifically, we first transform the 3D points from world coordinates into the reference camera coordinates using the estimated camera extrinsics. We then project them back to the image plane using the camera intrinsics. The final 3D flow is stored as a screen-aligned sequence Ft:t+L = [xi , yi , zi ]t+L i=t , where zi denotes the depth value in the reference camera frame. This formulation compensates for camera motion and isolates true scene motion, rather than mixing it with viewpoint-dependent image-plane displacement.

D

Experiment Details

D.1

Pre-training DynaFLIP

Model architecture. As shown in Figure 2, DynaFLIP consists of three modality encoders: image, language, and 3D flow. We describe each modality encoder below. • Image encoder. We initialize the image encoder with a pre-trained DINOv2-Base (ViT-B/14) backbone and keep the entire backbone trainable. Given an input image It , the backbone produces a [CLS] token and a sequence of patch tokens, each of dimension 768. We form the per-frame embedding by concatenating the [CLS] token with the average-pooled patch tokens:
dt = CLS(It ) ⊕ σ Patch(It ) ∈ R1536 , where σ(·) denotes average pooling over patch tokens. We apply the same procedure to every sampled frame in the clip. An MLP fusion block then combines the embeddings from each adjacent sampled frame pair to produce the image-transition embedding zI . • Language encoder. We use a frozen T5-Base encoder with a learnable adapter on top. Task instructions are tokenized with a maximum length of 77 tokens. The encoder produces a sequence of 768-dimensional token embeddings, from which we extract the sentence-level representation 20

via EOS-token pooling. The pooled representation is then projected through an adapter to obtain the language embedding zL . • 3D flow encoder. The 3D flow encoder receives a sequence of K timesteps of 20 × 20 × 3 flow data, representing 3D displacement vectors at 20 × 20 keypoints. The encoder consists of two stages: a 3D motion encoder and a temporal motion transformer. – 3D motion encoder. A 4-layer CNN encodes each timestep independently into a per-timestep feature. – Temporal motion transformer. A 4-layer transformer encoder aggregates information across the temporal window. To incorporate visual context, we prepend the current-frame image embedding dt as a conditioning token. A learnable temporal [CLS] token and positional embeddings are added to the sequence. The temporal [CLS] output is then projected through a linear layer to produce the 3D flow embedding zF . Training protocol. Following R3M [37], we sample five frames from each video clip during pretraining: an initial frame, a final frame, and three intermediate frames. The initial and final frames are sampled from the first 10% and the last 10% of the clip, respectively. The three intermediate frames are sampled from the remaining portion of the clip in temporal order. This sampling strategy yields an ordered frame sequence, from which we construct sequential transition pairs instead of a single pair such as (It , It+H ). The pre-training hyperparameters are summarized in Table 3, and pre-training takes approximately 4 days on 4 NVIDIA L40S. Table 3: Pre-training hyperparameters. Loss weights, optimization settings, and augmentation parameters used to pre-train DynaFLIP.

D.2

Category

Hyperparameter

Value

Loss

λtcn λact Contrastive temperature τ Cosine regularization α 3D flow temporal window of length K

1.0 1.0 0.07 1.0 7

Optimization

Optimizer Learning rate Weight decay Batch size

AdamW 10−4 10−2 32

Augmentation

Image resolution Brightness / contrast jitter Saturation / hue jitter

224 × 224 0.1 / 0.1 0.05 / 0.02

MetaWorld and RLBench

We provide additional details for the MetaWorld and RLBench experiments used to evaluate downstream performance and control-relevant representations. These experiments follow a frozenrepresentation protocol: the image encoder remains fixed throughout downstream training, and only a lightweight three-layer MLP policy is optimized on top. Each policy receives a visual feature extracted from a 224 × 224 third-person RGB observation, concatenated with the proprioceptive robot state. MetaWorld. MetaWorld evaluates single-task manipulation with a Sawyer arm and a two-finger gripper. We select 15 tasks that span multiple difficulty levels, following the task grouping used in prior work [45]. The easy tasks are button-press, drawer-open, reach, handle-pull, peg-unplug-side, lever-pull, and dial-turn. The medium tasks are hammer, sweep-into, bin-picking, push-wall, and box-close. The hard and very hard tasks are assembly, hand-insert, and shelf-place. For each task, we collect 25 demonstrations using the official scripted policy and use only the corner-view camera as visual input. Table 4 reports the detailed MetaWorld success rates grouped by task difficulty. RLBench. RLBench evaluates visuomotor manipulation with a Franka Panda arm. We evaluate six tasks: close box, put rubbish in bin, close laptop lid, water plants, unplug charger, and toilet seat down. For each task, we collect 100 demonstration trajectories using the Open Motion Planning 21

Table 4: MetaWorld success rates. Detailed success rates (%) grouped by task difficulty. Bold and underlined numbers indicate the best and second-best results in each column, respectively. Algorithm R3M [37] VC-1 [36] LIV [34] CLIP [42] DINOv2 [39] SigLIP [57] DynaFLIP (Ours)

Easy (7)

Medium (5)

Hard & Very Hard (3)

Mean

78.3 62.6 79.4 72.9 77.7 74.3 81.1

68.0 71.6 76.8 68.8 77.6 72.8 81.6

68.0 38.7 66.7 42.0 64.0 56.7 69.3

72.8 60.8 76.0 65.3 74.9 70.4 78.9

Library (OMPL) [48] and use only the front-view camera as visual input. Table 5 reports the task-wise RLBench success rates for each encoder. Table 5: RLBench success rates. Detailed task-wise success rates (%). Bold and underlined numbers indicate the best and second-best results in each column, respectively. Algorithm

close box

R3M [37] VC-1 [36] LIV [34] CLIP [42] DINOv2 [39] SigLIP [57] DynaFLIP (Ours)

96 84 92 60 84 80 88

put rubbish close laptop water in bin lid plants 4 12 8 0 12 4 8

56 72 76 56 76 52 76

8 8 4 4 4 0 20

unplug charger 20 16 20 12 24 12 36

toilet seat Mean down 92 96 92 80 84 76 96

46.0 48.0 48.6 35.3 47.3 37.3 54.0

Training and evaluation protocol. For both benchmarks, we train each method for 100 epochs with the visual encoder kept frozen throughout downstream training. Every 10 epochs, we evaluate the policy using 25 rollouts. We then select the best-performing checkpoint across training and report its average rollout success rate. D.3

LIBERO

We evaluate DynaFLIP on five LIBERO suites: LIBERO-90, LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long. LIBERO-90 contains 90 tasks, while each of the other four suites contains 10 tasks with 50 demonstrations per task. Model architecture. We adopt Diffusion Policy [9] as the downstream imitation-learning policy, using a U-Net backbone with channel dimensions [256, 512, 1024]. We use DDIM [46] for diffusionbased action generation, with 100 forward diffusion steps and 10 denoising steps during inference. We set the prediction horizon to 32, the execution horizon to 16, and the observation history to 1. For visual input, we use only third-person RGB observations and exclude gripper-view images. The image encoder output serves as the visual conditioning vector for the diffusion policy. For CNN-based encoders, we obtain the global image feature by applying global average pooling to the final feature map of the ResNet backbone. For ViT-based encoders, we concatenate the [CLS] token with the average-pooled patch tokens to form the image feature. Language instructions are encoded using the corresponding text encoder when available. For R3M, VC-1, and DINOv2, which do not provide native text encoders, we use the CLIP text encoder. CLIP, LIV, and DynaFLIP use the [EOS] token representation as the sentence-level language feature, whereas SigLIP uses mean pooling over all token embeddings. Training and evaluation protocol. Our primary LIBERO setting follows a reusable-encoder protocol: both the image and language encoders remain frozen, and only the diffusion policy is trained. This setting directly evaluates whether each pre-trained representation can transfer to downstream policy learning without task-specific encoder adaptation. As an additional comparison, we also report a LoRA setting that adapts both encoders jointly with the diffusion policy.

22

For each LIBERO suite, we train a separate diffusion policy using demonstrations from that suite and evaluate it on the corresponding suite. Each method is trained for 200 epochs. Every 20 epochs, we evaluate the policy using 20 rollouts per task. We then select the best-performing checkpoint across training and report its average rollout success rate. D.4

Real-world Robot

Hardware setup. Figure 9 shows the real-robot setup used for demonstration collection and policy evaluation. A fixed-base UR3 manipulator equipped with a two-finger gripper performs all manipulation tasks. Two RGB cameras, one third-person camera and one wrist-mounted camera, provide 224 × 224 visual observations. The policy also receives a 7D proprioceptive state consisting of the 6D end-effector pose and the gripper state. During demonstration collection, a human teleoperator controls the end-effector pose and gripper command through a custom teleoperation interface.

UR3

Wristed Camera Third-view Camera

Human Teleoperation

Figure 9: UR3 hardware setup. Real-robot setup used for demonstration collection and policy evaluation.

Task and data collection. We evaluate DynaFLIP on three representative real-world manipulation tasks: Pick <object> into Sink, Pour almonds into <object>, and Unfold Towel (see Figure 10 for in-distribution examples). These tasks cover both rigid-object manipulation and deformable-object interaction. For Pick <object> into Sink, the robot picks up the instructed object and places it in the sink. The object set contains nine objects: apple, block, bread, kettle, lemon, orange, pear, plate, and plum. We collect 10 demonstrations per object, yielding 90 trajectories in total. To control scene variation during training, we divide the nine objects into three groups of three and collect demonstrations for each group under a fixed scene layout. The task prompt is “pick up <object> and place it in sink.” For Pour almonds into <object>, the robot grasps a flat plate containing almonds and pours them into the instructed target object. The target set contains four objects: brown box, gray pan, white plate, and yellow plate. We collect 20 demonstrations per target object, yielding 80 trajectories in total. Across demonstrations, the source object remains fixed and only the target object changes. The task prompt is “pour almonds into <object>.” For Unfold Towel, the robot unfolds a towel initially folded in half. We collect 50 trajectories for this task. The task requires multi-stage deformable-object manipulation: the robot first opens the folded towel by grasping its middle region and then unfolds the two side edges. The task prompt is “unfold towel.” Model architecture. We integrate the frozen pre-trained image encoder into the pre-trained π0.5 through a lightweight visual-injection design inspired by plug-in visual injection (PVI) [60]. Our design enables parameter-efficient fine-tuning by freezing the π0.5 backbone and injecting auxiliary visual features into the action expert through a ControlNet-style [59] side branch.

23

Table 6: Real-robot fine-tuning hyperparameters. Task-specific training hyperparameter settings for fine-tuned π0.5 policies. Each subtable corresponds to one real-world task. Pick <object> into Sink

Pour almonds into <object>

Unfold Towel

Hyperparameter

Value

Hyperparameter

Value

Hyperparameter

Value

Action dimension Action horizon Batch size Optimizer Peak learning rate Final learning rate Warmup steps Decay steps Training steps

32 50 32 AdamW 1.5 × 10−5 1.5 × 10−6 500 5,000 5,000

Action dimension Action horizon Batch size Optimizer Peak learning rate Final learning rate Warmup steps Decay steps Training steps

32 50 32 AdamW 1.5 × 10−5 1.5 × 10−6 700 7,000 7,000

Action dimension Action horizon Batch size Optimizer Peak learning rate Final learning rate Warmup steps Decay steps Training steps

32 50 32 AdamW 1.5 × 10−5 1.5 × 10−6 1,000 10,000 10,000

We augment the original π0.5 pathway with three lightweight components: (i) an auxiliary visual encoder that processes each camera view into a sequence of patch tokens, (ii) a projection layer that maps these features to the VLA’s hidden dimension, and (iii) a trainable copy of the action expert that conditions on these auxiliary features. At each layer of the action expert, the trainable copy produces a residual signal that is added to the hidden state of the frozen main path, and the final action is predicted from the modified hidden state. During fine-tuning, we freeze both the π0.5 backbone and the auxiliary visual encoder, and optimize only the lightweight injection modules (projection layer, trainable copy of the action expert, and per-layer injectors). The trainable copy is initialized from the pre-trained action expert, and the projection and injection modules are initialized to zero. This makes the initial policy equivalent to the pre-trained VLA and allows the injected visual signal to become active gradually during training. For all real-robot comparisons, we keep the fine-tuning protocol fixed and change only the auxiliary visual encoder. This protocol isolates the effect of the visual representation from the effect of the VLA fine-tuning strategy. Feature extraction details. We extract patch-level features differently for CNN-based and ViT-based visual encoders because the two architectures produce spatial features in different forms. The resulting features are fed into the projection layer described above. • CNN-based encoders. We use the output of the final convolution block before spatial pooling, producing a feature map with shape B × C × H × W . We flatten the spatial dimensions to obtain a sequence of H · W patch tokens with shape B × (H · W ) × C. • ViT-based encoders. We use√ the patch √ tokens (excluding the [CLS] token) directly, producing N patch tokens arranged in a N × N spatial grid, with sequence shape B × N × C. Training and evaluation protocol. For each real-world task, we fine-tune a separate π0.5 policy from the same pre-trained base checkpoint, following the visual-injection setup described above. Table 6 summarizes the task-specific hyperparameters. After training, we evaluate each policy through closed-loop real-robot rollouts (20 rollouts per setting). The evaluation includes in-distribution trials for all three tasks. For Pick <object> into Sink and Pour almonds into <object>, we additionally evaluate two out-of-distribution perturbation types: visual-spatial perturbations and semantic perturbations. For Unfold Towel, we evaluate only the in-distribution setting. Figure 10 summarizes the overall real-world evaluation settings and the exact task instructions. A rollout is successful when the robot completes the instructed task within the episode horizon. For Pick <object> into Sink, success requires placing the instructed object inside the sink. For Pour almonds into <object>, success requires pouring the almonds into the instructed target object. For Unfold Towel, success requires unfolding both folded edges by the end of the episode.

24

In-distribution Tasks

- Pick up apple and place it in sink - Pick up kettle and place it in sink - Pick up lemon and place it in sink

- Pick up orange and place it in sink - Pick up pear and place it in sink - Pick up bread and place it in sink

- Pick up plate and place it in sink - Pick up plum and place it in sink - Pick up block and place it in sink

- Pour almonds into gray pan - Pour almonds into yellow plate

- Pour almonds into brown box - Pour almonds into white plate

- Unfold towel

Visual-Spatial perturbation

Out-of-distribution Tasks

(Unseen object position position and distractors)

- Pick up apple and place it in sink - Pick up bread and place it in sink - Pick up orange and place it in sink - Pick up plum and place it in sink

- Pour almonds into gray pan

Semantic perturbation (Unseen objects and instructions)

- Pick up green object and place it in sink - Pick up doll that has the same color as lemon and place it in sink - Pick up orange doll and place it in sink

- Pour almonds into metal tray - Pour almonds into white and yellow plate

Figure 10: Real-world evaluation tasks. Illustration of the in-distribution and out-of-distribution evaluation settings for the three real-world tasks, including the exact task instructions.

25

DINOv2

Wrong object selection

SigLIP

Grasping failure

DynaFLIP (Ours)

Successful rollout

(a) Pick up red doll and place it in sink

DINOv2

Spilling almonds

SigLIP

Spilling almonds

DynaFLIP (Ours)

Successful rollout

(b) Pour almonds into white and yellow plate

DINOv2

Wrong direction

SigLIP

Grasping failure

DynaFLIP (Ours)

Successful rollout

(c) Unfold towel

Figure 11: Representative rollout examples on three real-world tasks. We compare DynaFLIP with DINOv2 and SigLIP on (a) Pick up red doll and place it in sink (OOD), (b) Pour almonds into white and yellow plate (OOD), and (c) Unfold towel (in-distribution). Baselines exhibit distinct failure modes (wrong object selection, grasping failure, spilling, wrong direction), while DynaFLIP completes all three tasks successfully.

26

D.5

Control-Relevant Metric

We adopt the simulator-grounded state prediction metric proposed in [13] as a quantitative proxy for representation quality. This metric measures how well a visual representation preserves state information relevant to downstream control. We train lightweight probes to predict simulator state from visual features, and computes a normalized score Sm from the prediction errors. Simulator state. For a scene with No objects, the simulator state combines object-level and scenelevel information: i ∈ R4 , and bounding• Object-level state for each object i: position pipose ∈ R3 , orientation qpose i 3 box shape sshape ∈ R .

• Scene-level state: robot joint configuration q J ∈ RNj and end-effector pose pee ∈ RNee . The full simulator state s concatenates all object-level states and the scene-level state. State prediction probe. Given an input image I, we extract a spatial feature map and a global feature (see Feature extraction details below). Two probes predict the simulator state: • Object-level probe uses the feature map: it predicts each object’s state from RoI-pooled features inside its bounding box. • Scene-level probe uses the global feature: it predicts the robot and end-effector states. Both probes are lightweight regressors trained on top of the frozen visual features. Control-relevant score. For each state dimension a and model m, we compute the raw prediction score rm,a as the negative mean squared error between predicted and ground-truth values across all examples. To compare models on a unified scale, we min-max normalize rm,a across models within each state dimension and average the normalized scores: rm,a − minm̃ rm̃,a 1 X Sm = , |A| maxm̃ rm̃,a − minm̃ rm̃,a a∈A

where A denotes the set of evaluated state dimensions. A larger Sm indicates that the representation preserves more control-relevant information. Feature extraction details. We extract feature maps and global features differently for CNN-based and ViT-based visual encoders because the two architectures produce spatial features in different forms. • CNN-based encoders. – Feature map: the output of the final convolution block before spatial pooling, with shape B × C × H × W. – Global feature: the feature vector obtained by global average pooling over spatial dimensions, with shape B × C. • ViT-based encoders. – Feature map: patch √ tokens √ (excluding the [CLS] token), reshaped into a 2D spatial grid with shape B × C × N × N . – Global feature: the concatenation of the [CLS] token and the average-pooled patch tokens, with shape B × 2C.

E

Additional Experimental Results

E.1

LIBERO Results with Paired Image Encoders

Recent vision-language-action models commonly pair multiple image encoders to combine complementary visual features [31, 5, 61, 32, 30]: DINOv2 provides fine-grained, low-level spatial features, while language-aligned encoders such as CLIP and SigLIP capture high-level semantics. To assess 27

Table 7: LIBERO benchmark results with paired image encoders. We combine DINOv2 with various language-aligned vision encoders. All encoders are kept frozen, and only the diffusion policy is trained. The evaluation metric is success rate (%). Bold and underline numbers indicate the best and second-best results in each column, respectively. Image Encoders

Language Encoder

Frozen

DINOv2 + CLIP CLIP DINOv2 + SigLIP SigLIP DINOv2 + DynaFLIP (Ours) DynaFLIP (Ours)

Goal

Object

Spatial

Long

Mean

68.0 75.5 72.5

52.0 60.5 73.5

53.5 48.0 48.5

24.0 25.0 27.0

49.4 52.3 55.4

whether DynaFLIP remains beneficial in this setting, we pair DINOv2 with each language-aligned vision encoder—CLIP, SigLIP, and DynaFLIP—and evaluate them on LIBERO under the same frozen configuration as in Section 3.3. The image encoders are concatenated at the feature level before being passed to the diffusion policy, and the corresponding language encoder is paired with each setup. Table 7 reports the results. DINOv2 + DynaFLIP achieves the highest mean success rate, outperforming both DINOv2 + CLIP and DINOv2 + SigLIP. We attribute this advantage to two complementary properties of DynaFLIP’s representations. First, like CLIP and SigLIP, DynaFLIP aligns visual features with language and therefore provides the semantic grounding required for instruction following. Second, through dynamics-aware pre-training, DynaFLIP focuses on control-relevant regions critical for manipulation—a signal that purely image-text contrastive encoders do not provide. Together, these properties make DynaFLIP a more effective language-aligned counterpart to DINOv2’s fine-grained spatial features. E.2

Grad-CAM visualizations

Visualization protocol. We use the PyTorch-Grad-CAM library to generate Grad-CAM visualizations and identify the image regions that contribute most to downstream action prediction. For each frozen image encoder, we compute Grad-CAM with respect to a scalar target defined as the negative mean squared error between the action predicted by the trained three-layer MLP policy head and the ground-truth action. With this choice, the resulting heatmap highlights the visual regions that most strongly support accurate action prediction. As the target layer, we use the final convolutional layer (model.layer4[-1]) for CNN-based encoders and the pre-attention normalization layer in the last Transformer block (model.blocks[-1].norm1) for ViT-based encoders. Additional visualizations. We provide additional Grad-CAM visualizations to complement the qualitative analysis in Section 3.2. These examples further show that DynaFLIP consistently focuses on task-relevant objects and interaction regions, while baseline encoders more often exhibit diffuse attention or place substantial emphasis on less control-relevant regions. Table 8: Grad-CAM visualizations R3M

VC-1

LIV

CLIP

DINOv2

SigLIP

DynaFLIP

Bin-picking

Assembly

Task

Continued on next page

28

Reach

Push-wall

Peg-unplug-side

Lever-pull

Handle-pull Hand-insert

Hammer

Drawer-open Dial-turn

Button-press Box-close

Table 8: Grad-CAM visualizations (Continued)

Task R3M VC-1 LIV CLIP

29 DINOv2 SigLIP DynaFLIP

Continued on next page

Table 8: Grad-CAM visualizations (Continued) R3M

VC-1

LIV

CLIP

DINOv2

SigLIP

DynaFLIP

Water-plants

Unplug-charger Close-laptop-lid

Close-box

Sweep-into

Shelf-place

Task

E.3

PCA visualizations

Visualization protocol. We apply PCA to the spatial features of each encoder. For ViT-based encoders (VC-1, CLIP, DINOv2, SigLIP, DynaFLIP), we use the patch tokens; for CNN-based encoders (R3M, LIV), we use the 7 × 7 output of the final convolution block. We project the resulting features to 3 principal components and map them to RGB. Additional visualizations. We provide additional PCA visualizations to complement the qualitative analysis in Section 3.2. Table 9: PCA visualizations of learned representations R3M

VC-1

LIV

CLIP

DINOv2

SigLIP

DynaFLIP

Bin-picking

Assembly

Task

Continued on next page

30

Table 9: PCA visualizations of learned representations (Continued) R3M

VC-1

LIV

CLIP

DINOv2

SigLIP

DynaFLIP

Reach

Push-wall

Peg-unplug-side

Lever-pull

Handle-pull

Hand-insert

Hammer

Drawer-open

Dial-turn

Button-press

Box-close

Task

Continued on next page

31

Table 9: PCA visualizations of learned representations (Continued) R3M

VC-1

LIV

CLIP

Water-plants

Unplug-charger Close-laptop-lid

Close-box

Sweep-into

Shelf-place

Task

32

DINOv2

SigLIP

DynaFLIP