1
SkyJEPA: Learning Long-Horizon World Models for Zero-Shot Sim-to-Real Control of Quadrotors
arXiv:2606.23444v1 [cs.RO] 22 Jun 2026
Pratyaksh Rao, Wancong Zhang, Randall Balestriero, Yann LeCun, Giuseppe Loianno
Abstract—Accurate dynamics models are critical for informed decision-making in robotic systems, particularly for agile aerial vehicles operating under uncertainty. Neural network dynamics models are attractive for capturing complex nonlinear effects, but existing predictive approaches struggle with long-horizon forecasting because their autoregressive rollout mechanism amplifies errors over time. Joint Embedding Predictive Architectures (JEPAs) offer a compelling alternative by modeling dynamics in latent space, yet prior JEPA-style methods for robot navigation have been studied primarily for kinematic-level planning, with limited investigation in high-frequency control. In this work, we introduce the JEPA-style model for real-time quadrotor control. The proposed approach combines a latent dynamics model with a novel physics-inspired prober that maps frozen latents to interpretable state, enabling physically grounded long-horizon prediction. Additionally, we combine the learned model with a sampling-based optimal control solution to take advantage of its predictive capabilities for real-time control on embedded hardware. Finally, to reduce the dependence on expensive and unsafe real-world data collection, we develop a structured pipeline for automated dataset generation. Extensive open-loop and outdoor closed-loop experiments demonstrate accurate prediction, robust zero-shot sim-to-real transfer, and strong generalization across diverse operating conditions. Index Terms—Model Learning for Control; Aerial Systems: Mechanics and Control; Learning and Adaptive Systems; Optimization and Optimal Control
I. I NTRODUCTION
U
Nmanned Aerial Vehicles (UAVs) have become increasingly important in applications such as package delivery, infrastructure inspection, search and rescue, and environmental monitoring [1], [2]. These tasks require aerial robots to operate reliably in complex, uncertain, and often rapidly changing environments while executing agile maneuvers with limited onboard sensing and computation. Achieving this level of autonomy fundamentally depends on the ability to make accurate decisions. For this reason, model-based control has emerged as a particularly attractive paradigm for aerial robotics, as it explicitly reasons about future system evolution and can naturally incorporate task objectives, dynamical constraints, and robustness considerations [3]–[5]. A dynamics model, often referred to as predictive or world model, for aerial control should satisfy four key properties (see Figure 1). First, it should provide accurate long-horizon predictions. Its rollouts must remain accurate, stable, and physically plausible over the prediction horizons used for control. This requires capturing both the dominant rigid-body dynamics and difficult-to-model effects such as aerodynamic
Figure 1: Desirable properties of a quadrotor world model: accurate long-horizon prediction, interpretable, real-time inference for closed-loop control, and zero-shot task generalization.
drag, actuator delay, propeller–airframe interactions, wind disturbances, and hardware variations. Second, it should be interpretable. The model should be able physically to capture and expose meaningful quantities such as position, velocity, attitude, and angular velocity. This is essential for enforcing constraints, actuator limits, safety bounds, and task costs inside a controller. Third, it should be real-time. Real-time aerial control requires repeated high-frequency model evaluations inside resource-constrained onboard optimization loops. Finally, the last property is zero-shot task generalization. The same dynamics representation should be reusable across trajectories, controllers, objectives, and platform configurations. Classical first-principles models satisfy some of these requirements [6], but fall short in practice. They encode the known structure of quadrotor dynamics and are computationally efficient, yet real-world flight behavior is shaped by complex and platform-specific effects that are difficult to model exactly. Small changes such as payload attachment, propeller replacement, motor degradation, or sensor bias can significantly alter the system response. As a result, analytical models often require extensive system identification [7] and manual tuning, while still struggling to maintain accuracy across operating regimes. This limitation has motivated a large body of work on neural dynamics models [8]–[14]. These methods aim to learn unmodeled nonlinear effects directly from data and have been explored in both offline and online settings. Offline methods train models from pre-collected trajectory datasets [8]– [10], [12]–[14], while online methods adapt the model during deployment to account for changing dynamics or distribution shift [15]–[20]. Despite these differences, most approaches
2
share a common formulation: they learn a predictive encoder– encourages the model to capture control-relevant dynamics decoder model that estimates the next state or observation and while avoiding unnecessary reconstruction of nuisance details. recursively feeds this prediction back as input during rollout. To make the learned representation usable for model-based While intuitive, this autoregressive predictive formulation control, we introduce a physics-inspired prober that maps violates a central requirement for model-based control: long- frozen latent rollouts to physically meaningful state variables horizon stability. Since the model is repeatedly conditioned on through a lightweight kinematic structure. By combining its own imperfect outputs, small one-step errors arising from latent predictive representation learning, physical structure, uncertainty, approximation, or inaccuracies can accumulate over and domain-randomized simulation, our approach enables time, leading to drift, instability, and physically implausible accurate long-horizon prediction and real-time control while trajectories. In practice, the error may easily compound and remaining reusable across tasks and platform variations. Our can severely degrade rollout fidelity as the prediction horizon main contributions are summarized as follows: increases [10]. Adding physics-inspired regularization [8], • We introduce a JEPA-styled latent dynamics model with a residual structures [9], or online adaptation [16] can improve physics-inspired prober that maps frozen representations to local prediction accuracy, but does not remove the underlying physically meaningful state variables, enabling physically autoregressive error accumulation mechanism. Moreover, recongrounded long-horizon prediction for quadrotor control. structing the full future state or observation at every step can • We demonstrate that by integrating the learned latent force the representation to preserve nuisance factors such as dynamics model within a sampling-based optimization sensor noise, bias, or task-irrelevant variation. This becomes esframework, we exploit the predictive capabilities of the pecially problematic for high-dimensional observations, where learned model for real-time control and robust zero-shot full observation prediction is computationally expensive. sim-to-real transfer on resource-constrained embedded Model-Based Reinforcement Learning (MBRL) provides an platforms. alternative route by jointly learning a world model and a task• We propose a domain-randomized simulation pipeline driven policy from interaction data [21]–[23]. Although such for automated dataset generation, reducing the need for approaches can achieve strong task performance, the learned extensive and potentially unsafe real-world data collection. model is often coupled to a particular reward, policy, or visi• We extensively evaluate the proposed framework against tation distribution. Consequently, the resulting representation current dynamics learning baselines and key design hardly generalizes across controllers, reference trajectories, or choices through open-loop prediction, and outdoor closeddownstream tasks. For aerial robotics, where the same dynamics loop control experiments, demonstrating improved longmodel should ideally support different tasks, such task-coupled horizon accuracy, robust control, and generalization across learning can be restrictive. trajectories and platform variations. A further obstacle is data collection. For any learned dynamics model, the quality of the dataset is as important II. R ELATED W ORKS as the choice of architecture. The data collection process must informative (e.g., efficiently covering the overall flight envelope) A. Predictive Modeling and sample efficient and therefore covering a diverse range A large body of work has studied learning robot dynamics of states, control inputs, speeds, accelerations, and operating from data in both offline [12], [13], [24] and online setconditions. However, for aerial robots, there is currently no tings. Prior offline approaches have explored residual learning systematic procedure for collecting such a dataset. In practice, over nominal models [9], [25], improved multi-step training data collection often relies on hand-designed trajectories, expert objectives [10], and physics-inspired regularization [8]. On pilots, or task-specific flight logs. This makes the resulting the other hand, online methods [15]–[20] adapt the model dataset incomplete and biased toward a limited set of behaviors. during deployment, typically by finetuning the last few layers Collecting broader real-world data is also expensive, risky, of the neural dynamics model. Although these directions and labor-intensive. It often requires controlled environments improve the ability to capture dynamics, each has an inherent and aggressive maneuvers, which increase the likelihood of limitation. Residual learning strongly depends on the fidelity crashes or hardware damage. Moreover, when the platform of the nominal model. Physics-inspired regularization can configuration changes, existing pipelines often require new data guide training toward more plausible solutions, but it does not collection and retraining, limiting scalability. guarantee physically meaningful rollouts. Online adaptation These limitations raise the central question of this work: adds deployment complexity because it requires continual onCan we learn an efficient dynamic model for aerial robots robot data collection and model updates, along with extra that provides accurate long-horizon prediction, physically compute and safeguards to prevent unstable intermediate interpretable rollouts, real-time control capability, and zero- models from degrading control. Moreover, all these approaches shot generalization across tasks, while reducing dependence on remain based on an autoregressive encoder–decoder paradigm, risky real-world data collection? To address this question, we in which predicted states or observations are recursively reused propose a domain-randomized simulation-to-real framework as inputs for future prediction [8], [26]–[28]. As a result, small for training a JEPA-style latent dynamics model for real- one-step errors are propagated forward and accumulate over time aerial control. Rather than reconstructing future states or time, leading to drift and degraded long-horizon fidelity. This observations autoregressively, our model learns to predict future compounding-error problem is intrinsic to predictive modeling representations in a structured latent space. This formulation and is not removed by better losses, residual corrections, or
3
online updates alone, motivating the need for alternatives. In contrast, the proposed JEPA-styled approach produces more accurate long-horizon model predictions compared to these classic predictive models by properly capturing the temporal and spatial context of the problem using a compact abstract representation, avoiding reconstruction altogether. B. Joint Embedding Predictive Architectures JEPA-style methods offer an appealing alternative to predictive encoder–decoder models by learning to predict future embeddings rather than reconstructing full future observations, yielding compact representations that are naturally scalable to high-dimensional inputs. This idea has shown promise in image representation learning [29], video prediction [30]– [32], and more recently in robotics [33]. However, most JEPA-style robotics works have focused on vision-centric settings such as manipulation [34] and navigation [35], with several demonstrated only on toy problems or in exocentric simulation without real-world validation [36]–[38]. Moreover, these methods are typically studied at the level of highlevel kinematic planning, where abstract latent predictions are often sufficient. Their use for feedback control remains largely unexplored, particularly for aerial robots, where highfrequency inference of the system dynamics and real-time compute constraints make control-oriented latent modeling significantly more challenging. This is what we address in this work by proposing by proposing the first JEPA framework trained end-to-end for real-time quadrotor control. C. Model-Based Reinforcement Learning With the rise of deep Reinforcement Learning (DRL), a new class of controllers has emerged that leverage learned models of system dynamics to plan and act [3], [4], [21], [39], [40]. These approaches aim to improve data efficiency and generalization by explicitly reasoning about future system evolution, rather than relying solely on reactive, model-free policies. Early modelbased RL methods primarily focused on learning dynamics in state space [41], and have since been extended to handle high-dimensional sensory inputs such as images [42]. However, predicting directly in observation space is often data-intensive and computationally expensive, particularly for vision-based inputs, and can lead to the learning of noisy features when the training data distribution is imperfect. Latent-space prediction offers a more compact alternative, but many existing approaches rely on reconstruction-based objectives [43], inheriting similar limitations associated with observation reconstruction. Moreover, a large class of model-based RL methods incorporate reward prediction, either explicitly or as an auxiliary objective, when learning latent representations [44]–[47], which inherently couples the learned world model to a specific task. In contrast, in this work we, decouple task-dependent information from latent dynamics prediction to enable general-purpose, real-time quadrotor control without any task-specific reward conditioning. III. BACKGROUND A. Learning System Dynamics We consider a discrete-time dynamical system with state xt ∈ RDs and control action at ∈ RDa . The dynamics are
described by a transition function f : RDs × RDa → RDs xt+1 = f (xt , at ). For a quadrotor, we define the state as ⊤ ⊤ vt⊤ r⊤ xt = p⊤ t x,t ry,t rz,t
(1)
ωt⊤
⊤
.
(2)
Here, pt ∈ R3 and vt ∈ R3 denote position and velocity in the inertial frame. The attitude is represented by the rotation matrix Rt = rx,t ry,t rz,t ∈ SO(3), whose columns are rx,t , ry,t , rz,t ∈ R3 . This rotation matrix represents the attitude of the body frame in the inertial frame. The angular velocity ωt ∈ R3 is expressed in the body frame. The action at = [f0,t , f1,t , f2,t , f3,t ]⊤ ∈ R4 denotes the four motor forces. B. Predictive Modeling A common approach to learning dynamics is to train a predictive model on state-action transitions. Let D = {(xi , ai , xi+1 )}N i=1 be a dataset of N transitions. The goal is to approximate the dynamics using a neural network h, parameterized by weights θ. Formally, the state at the next time index, i + 1 is given by x̃i+1 = hθ (xi , ai ).
(3)
The model is trained by minimizing the prediction loss N
min θ
1 X 2 ∥xi+1 − hθ (xi , ai )∥2 . N i=1
(4)
This objective encourages accurate local predictions. However, model-based control requires prediction over a sequence of future actions. Given an action sequence, the model is rolled out recursively x̂t+T = hθ (. . . hθ (hθ (xt , at ), at+1 ) . . . , at+T ).
(5)
This recursive formulation introduces compounding error. Let the vector prediction error be ϵt+k = xt+k − x̃t+k . During rollout, each prediction is used as the input to the next step, x̃t+1 = hθ (xt , at ), x̃t+2 = hθ (xt+1 − ϵt+1 , at+1 ), .. .
(6)
x̃t+T = hθ (xt+T −1 − ϵt+T −1 , at+T −1 ). This makes the source of compounding explicit. At each step, the model is evaluated on a state that is shifted from the true trajectory by the accumulated prediction error. As the horizon increases, these errors alter the future inputs to the model. Thus, small one-step errors can grow into large rollout errors. This can lead to drift and physically implausible trajectories. IV. M ETHODOLOGY In this section, we describe the methodology underlying our approach (see Figure 2). We first introduce our proposed JEPA-styled dynamics learning formulation. Next, we present the training objective for learning dynamics. We then describe a sampling-based optimization framework that leverages the learned model for optimal real-time quadrotor control. Lastly,
4
Sim Data Synthesis
Latent Dynamics Model
Physics-Inspired Prober
Zero-Shot Sim2Real
Principled Data Generation
JEPA-Style
DKI: Differentiable Kinematic Integration
Sampling-Based Control
Figure 2: Overview of the proposed framework. We learn a JEPA-style latent dynamics model with a physics-inspired prober that maps abstract embeddings to physically meaningful states, enabling stable long-horizon quadrotor prediction. Trained entirely on domain-randomized simulation data and deployed within a sampling-based controller, the model achieves real-time execution and robust zero-shot sim-to-real transfer validated in outdoor flight experiments. we describe automated data collection framework, including trajectory generation, closed-loop control, and domain randomization strategy used to generate diverse trajectory-level data. A. Problem Formulation We seek to learn a dynamics model that supports longhorizon prediction for model-based quadrotor control. As discussed in Section III-B, directly predicting future states in an autoregressive manner can lead to compounding errors. Therefore, we formulate dynamics learning in a compact latent space. This follows the JEPA principle of predicting representations rather than reconstructing inputs [48]. The goal is to learn a representation that captures the evolution of the system while avoiding unnecessary reconstruction of task-irrelevant details. At each time step, the UAV receives an estimated full-state observation xt . To provide temporal context and promote better sim-to-real transfer, we condition the model on histories of states and actions over a window of length H. The state history and action history are defined as ⊤ ⊤ · · · x⊤ · · · a⊤ Xt = x ⊤ , At = a⊤ . t t t−H t−H (7) This history-based formulation gives the model access to recent motion and actuation trends. It also helps account for effects that are not fully captured by a single state, such as actuator delay, drag, sensor noise, and platform-dependent dynamics. The state and action histories are encoded into latent representations st = Encθ (Xt ) and zt = Encϕ (At ). The latent dynamics predictor then estimates the next latent state as s̃t+1 = Predφ (st , zt ). For a prediction horizon of T , the predictor is recursively unrolled using the encoded actions s̃t+T = Predφ (. . . Predφ (Predφ (st , zt ), zt+1 ), . . . , zt+T −1 ). (8) Thus, the learning problem is to train the encoders and predictor so that the latent rollout {s̃t+1 , . . . , s̃t+T } remains consistent with the encoded future trajectory {st+1 , . . . , st+T }. This formulation avoids direct reconstruction of future states while preserving the information needed for long-horizon dynamics.
B. Training Objective The objective is to learn latent representations that support accurate multi-step dynamics prediction for long-horizon forecasting (see Figure 3). The proposed loss contains two terms. The first term enforces predictive consistency between rolledout latent predictions and encoded future states. The second term regularizes the latent space to prevent representation collapse. We define the multi-step latent prediction loss as T
Lpred =
1 X ∥s̃t+k − st+k ∥22 . T
(9)
k=1
This loss encourages the encoder and predictor to learn representations that remain predictive over the full rollout horizon. However, minimizing this term alone admits degenerate solutions. For example, the encoder could map all inputs to nearly constant embeddings, yielding low prediction error without preserving meaningful system dynamics. We therefore utilize an anti-collapse regularization term. We employ Sketched Isotropic Gaussian Regularization (SIGReg) [49], which encourages the latent embeddings to match an isotropic Gaussian distribution. This promotes diversity and isotropy in the representation space. To apply SIGReg over a rollout during training, we collect the predicted latent embeddings into a tensor S ∈ RT ×B×D , where B is the training batch size during training, and D is the embedding dimension. Instead of matching the full D-dimensional latent distribution directly, SIGReg compares random one-dimensional projections of the latent distribution to a standard Gaussian. We sample M random unit vectors, where the directions are sampled uniformly on the hypersphere, D−1 {ξm }M and project the latent tensor along each m=1 ⊂ S (m) direction h = Sξm ∈ RT ×B . For each projection, we evaluate the univariate Epps–Pulley test statistic, measuring the distribution mismatch. Let B
ϕN (t; h(m) ) =
1 X ith(m) e b , B
(10)
b=1
denote the empirical characteristic function of the projected
5
Dataset
Latent Dynamics Model
Physics-Inspired Latent Probing
Action History
Stop Grad
Loss (Eq. 17)
Loss (Eq. 13)
State History
(Eq. 15-16) Future States Latent Targets
Replay Buffer
Shared
Figure 3: Two-stage training pipeline of our proposed approach. In the first stage, an encoder maps a history of past states into a latent representation, and a predictor propagates this latent forward in time conditioned on a sequence of control actions; in the second stage, a physics-inspired prober is trained on frozen latent embeddings to map them to physically grounded states. samples, and let ϕ0 (t) denote the characteristic function of the standard Gaussian N (0, 1). The projected discrepancy is Z ∞ 2 (m) T = w(t) ϕB (t; h(m) ) − ϕ0 (t) dt, (11) −∞
where w(t) is a weighting function, typically chosen as a Gaussian kernel. The SIGReg penalty is then M
1 X (m) T . LSIGReg = M m=1
(12)
By the Cramér–Wold theorem, matching all one-dimensional marginals is equivalent to matching the full joint distribution. Therefore, minimizing these projected discrepancies encourages the latent distribution to approach an isotropic Gaussian. In practice, the integral in T (m) is evaluated numerically using quadrature, following [49]. We average this SIGReg penalty across the temporal dimension. The final objective combines multi-step prediction consistency with SIGReg regularization Ltotal = Lpred + λsig LSIGReg .
(13)
Here, λsig controls the strength of the anti-collapse regularization relative to the prediction objective. The SIGReg component introduces only two practical hyperparameters, M and λsig , with λsig being the main parameter to tune. Following [49], we also find that performance is not highly sensitive to the number of random projections M . This is in contrast to many self-supervised representation learning objectives that require balancing several regularization terms [38], [50], stopgradient [51] design choices, exponential moving averages, or reconstruction weights. C. Physics-Inspired Probing Mechanism For model-based control, latent predictions must be converted into physically meaningful quantities. This is necessary for evaluating tracking costs, enforcing state constraints, and respecting actuator limits. However, the JEPA-style dynamics model predicts future evolution in an abstract representation space. We therefore introduce a physics-inspired probing mechanism that maps latent rollouts to interpretable state trajectories through a differentiable kinematic model. After training the latent dynamics model with eq. (13), we perform
a second training stage for metric-state recovery. During this stage, the encoders and predictor (Encθ , Encϕ , Predφ ), are frozen and only the probing network is optimized. This separation prevents the supervised state-recovery objective from altering the learned latent dynamics, while allowing the prober to learn a physically grounded map from frozen latent rollouts to metric state trajectories. Given the predicted latent sequence {s̃t+1 , . . . , s̃t+T }, the current estimated state xt , and the control sequence {at , . . . , at+T −1 } used for latent rollout which in our specific case will be generated using the optimal control procedure presented in Section IV-D. The probing network ψ predicts residual correction terms {∆v̇t+k , Kt+k } = ψ(s̃t+k ).
(14)
3
Here, ∆v̇t+k ∈ R represents a residual translational acceleration, Kt+k ∈ R3×4 parameterizes residual angular acceleration, and k is the unroll index. The control input is the individual rotor forces as mentioned in Section III. The predicted state trajectory is obtained by integrating a residualcorrected kinematic model. At each step, the translational acceleration is given by the nominal thrust-induced acceleration plus the learned residual P3 fi,t v̇t = i=0 Rt e3 − g + ∆v̇t , (15) m ∆τt = Kt at . Here, e3 = [0 0 1]⊤ , and ∆τt+k denotes the latent-conditioned residual angular acceleration. The state is then propagated using pt+1 = pt + vt ∆t, vt+1 = vt + v̇t ∆t, Rt+1 = Rt exp([ωt ]× ∆t) ,
(16)
ωt+1 = ωt + ∆τt ∆t. This compact integrator preserves the geometric structure of the attitude dynamics through the SO(3) exponential map, while allowing the latent representation to correct for unmodeled translational and rotational dynamics effects. During prober training, a stop-gradient operation is applied to the predicted latent embeddings before they are passed to ψ. This prevents the supervised state-recovery loss from modifying
6
the learned latent dynamics. Let x̃t+k denote the state obtained by integrating eq. (16). The probing network is optimized using the supervised rollout loss. T
Lprober =
1 X ∥x̃t+k − xt+k ∥22 . T
(17)
k=1
By combining frozen latent predictions with a structured differentiable integrator, the probing mechanism converts abstract representations into physically meaningful rollouts suitable for control.
where Qx and Qa are constant positive diagonal weight matrices for state and control, respectively. Softmax Weighting and Control Update. Following the MPPI formulation [4], we compute importance weights exp − λ1 (J (s) − Jmin ) w(s) = PS (23) , 1 (r) − J min ) r=1 exp − λ (J where Jmin = mins J (s) and λ > 0 is the temperature parameter. The nominal sequence is then updated as anom ← anom + k k
D. Sampling-Based Control with Learned Dynamics
S X s=1
(s)
w(s) δak ,
(s)
(s)
δak = ak − anom . k
(24) Only the first action anom is executed, and the procedure is 0 repeated in a receding-horizon fashion.
We consider the problem of tracking a reference trajectory over a finite horizon T . At each control timestep t, given the current estimated state history Xt and action history At , the objective is to compute a sequence of future control inputs E. Automated Data Synthesis {at , . . . , at+T −1 } that minimizes a trajectory tracking cost The performance of a learned dynamics model is strongly ref with respect to desired reference states {xref t+1 , . . . , xt+T }. determined by the data distribution used for training. In our case, We integrate our learned dynamics model within a sampling- we show a princpled procedure to synthetize data in simuation based optimization framework, MPPI, where future action to effectively learn zero-shot sim2real transfer. A useful dataset sequences are optimized using Monte Carlo sampling. should satisfy three requirements. First, it should be diverse to Action Sampling. Let the current nominal action sequence properly represent the overall flight envelope: the trajectories be anom = {anom , . . . , anom must cover a broad range of positions, velocities, accelerations, 0 T −1 }. We generate S candidate sequences by perturbing it attitudes, angular velocities, and control inputs. Second, it should be dynamically feasible: the recorded transitions should (s) (s) (s) ak = anom + ϵk , ϵk ∼ N (0, Σ), (18) k arise from closed-loop execution of physically valid quadrotor for s = 1, . . . , S and k = 0, . . . , T − 1, with covariance matrix dynamics, rather than from arbitrary state sampling. Third, it should be robust: the data should include variations in physical Σ. The admissible action set is defined element-wise as parameters so that the learned model does not overfit to a A = a ∈ R4 amin ≤ a ≤ amax , (19) single nominal platform. Reference Trajectory Generation. The first step in our where amin and amax denote lower and upper limits. Each data collection pipeline is to generate reference trajectories that sampled action is projected onto A via element-wise clamping excite diverse regions of the quadrotor state space. Manually (s) (s) designed trajectories, such as circles, figure-eights, or straightak ← ΠA ak , (20) line paths, cover only a narrow set of flight behaviors. They where ΠA (·) denotes element-wise clamping. can also bias the learned model toward a small number of Latent Rollout and State Prediction. For each sampled hand-selected motion patterns. To avoid this, we use Gaussian action sequence, we append these actions to the action history processes to automatically generate randomized, smooth, and previously executed to construct the updated action sequence diverse reference trajectories. For each trajectory, we sample (s) (s) (s) {at−H , . . . , at , . . . , at+T −1 }. The context states and actions the desired position independently along each spatial axis from are encoded using Encθ and Encϕ , respectively. For each a Gaussian process prior. A Gaussian process, denoted by rollout sample s, the latent dynamics are recursively unrolled GP(0, kj ), defines a distribution over smooth functions with (s) (s) using Predφ . The predicted latent sequence {s̃t+1 , . . . , s̃t+T } zero mean and covariance kernel kj (t, t′ ). We write this as, is mapped to physically grounded states via the probing pj (t) ∼ GP(0, kj (t, t′ )), j ∈ {x, y, z}. (25) mechanism, eqs. (14)–(16), yielding predicted physical states (s) (s) {x̃t+1 , . . . , x̃t+T }. Each kernel is chosen as a sum of periodic kernels with different Trajectory Cost. Each sampled trajectory is evaluated using characteristic length scales and periods. This allows the sampled a running cost of the form trajectories to contain both slow global motion and faster local T
J (s) =
1 X (s) ref (s) ℓ x̃t+k , xt+k , at+k−1 , T
(21)
k=1
where the per-step cost consists of weighted quadratic tracking errors and control effort: ℓ(.) = ∥x̃(s) − xref ∥2Qx + ∥a(s) − aref ∥2Qa ,
(22)
variations. The generated references induce a broad range of velocities, accelerations, attitudes, and angular velocities during tracking. We compute the corresponding velocity and acceleration references by differentiation, and then use differential flatness [52] to obtain the full quadrotor reference. The resulting reference trajectories are randomized, smooth, and dynamically rich. They provide a systematic alternative to
7
manually specified trajectory families and form the basis for NVIDIA Orin NX for onboard computation and a Pixracer collecting diverse state–action rollouts in simulation. Pro flight controller running PX4 [55] for low-level control, Closed-Loop Trajectory Tracking. The generated refer- with the full software stack integrated through ROS2. A laptop ences are not used directly as training data. Instead, each ground station is used only to issue high-level commands reference is tracked in simulation to produce physically feasible such as takeoff, landing, and reference trajectory upload. State state–action trajectories. At each time step, a tracking controller estimation is obtained from onboard GPS-based localization, computes the action from the current state and a local refer- and the we send collective thrust and body-rate commands ref ence horizon xref to the low-level flight controller. We conducted closed-loop t:t+T , at = πtrack xt , xt:t+T , where πtrack is implemented using a combination of nominal Nonlinear real-world evaluation (see Figure 4) under 3 trajectory tracking Model Predictive Control (NMPC) and Model Predictive Path settings: (a) nominal trajectory tracking, (b) payload variation, Integral (MPPI) [4]. The two controllers expose the dataset to and (c) propeller switching. All tests are conducted in a larger complementary action distributions. NMPC generates smooth, outdoor flying space 60 × 70 m2 . optimized tracking commands around the nominal dynamics, while MPPI produces sampling-based action sequences with B. Dataset Generation broader local variation. This diversity is important for learning a To promote sim-to-real transfer, we construct the training dynamics model that remains accurate under the control inputs dataset through automated trajectory generation and largeencountered during real-time sampling-based optimization. At scale domain randomization in simulation. For each rollout, the same time, collecting data through closed-loop controllers the quadrotor parameters are sampled independently from ensures that the recorded trajectories respect actuator limits, uniform distributions around the nominal platform values, dynamic feasibility, and realistic state–action correlations, as summarized in Table I. We sample 500 distinct domains, unlike independent random sampling of states and controls. producing a diverse ensemble of dynamic models that span Domain-Randomized Quadrotor Simulation. Data colvariations in inertial properties, actuation, motor response, lection is performed in simulation using standard quadrotor and aerodynamic drag according to the procedure presented rigid-body dynamics [53], with aerodynamic drag and firstin Section IV-E. For each domain, we generate smooth order motor delay included. To improve robustness and sim-torandomized reference trajectories using Gaussian processes [56]. real transfer, we do not generate data from a single nominal The desired position is sampled independently along the x, y, simulator. Instead, each rollout is collected from a randomized and z axes using sums of exponential sine-squared periodic quadrotor model using a similar ancestral sampling approach kernels (see eq. (25)). Each axis uses three periodic components: described in [54]. This sampling approach ensures that the the first has length scale 1.3, while the remaining two have parameter distribution is physical plausible. This allows the length scales 3.0 and 4.0; the periodicities are chosen differently dataset to reflect not only the nominal platform, but also across axes to avoid repetitive motion patterns. Specifically, nearby systems that may arise from modeling errors, hardware the x axis uses periodicities 37, 61, and 13, the y axis uses variation, or changes in operating conditions. We define the 17, 23, and 52, and the z axis uses 19, 29, and 53. This simulator parameter set as multi-frequency GP sampling produces trajectories with both η = {m, D, J, α, kf , kτ , l} , (26) slow global motion and faster local variations, which helps where m is the mass, D is the drag matrix, J is the inertia excite a broad range of translational and rotational behaviors matrix, α is the motor time constant, kf and kτ are the and properly explore the flight envelope. We generate 20,000 thrust and torque coefficients, and l is the arm length. These reference trajectories of 10 s duration, track them in simulation, parameters capture the main sources of mismatch between and record the complete state–action time series. Finally, all simulation and the real platform, including payload changes, trajectories are resampled using cubic splines at a discrete time actuator variation, and inertial differences. For each rollout r, step of 0.05 s, corresponding to 20 Hz, to match the temporal we sample a randomized parameter set ηr from a bounded resolution used for multi-step dynamics learning and control. The resulting dataset is split into 80% for training, 10% for uniform distribution around the nominal parameter set η̄ validation, and 10% for testing. ηr ∼ U (η̄ − ∆η, η̄ + ∆η) , (27) where ∆η defines the randomization range for each parameter. Each generated reference trajectory is therefore tracked under a different plausible realization of the quadrotor dynamics. This exposes the learned model to a family of systems rather than a single nominal model. Consequently, the learned representation is encouraged to capture dynamics that remain consistent across variations in mass, drag, actuation, and inertial properties. V. I MPLEMENTATION D ETAILS A. Platform Setup Our experiments are conducted on a 1.3-kg quadrotor with a thrust-to-weight ratio of 4:1. The platform uses an
C. Network Architecture and Training Both Encθ and Encϕ are implemented as Temporal Convolutional Networks (TCNs) [57] with channel sizes [8, 8, 16] and [4, 4, 8], respectively. Latent dynamics are modeled using a single-layer GRU predictor [58] with hidden dimension 24 and trained via recursive unrolling over T = 20 (1.0 s at 20 Hz) steps. The history length is set to H = 10 timesteps (0.5 s at 20 Hz). Training is performed for 50 epochs with batch size 2048. We employ a SigReg objective with 17 spline knots and regularization coefficient λsigreg = 0.02. Optimization uses Adam with weight decay 10−5 and gradient clipping at 0.5. The learning rate follows a linear warmup schedule from 0
8
Figure 4: Closed-loop real-world evaluation settings under trajectory tracking. We test the proposed framework under three scenarios: (a) nominal trajectory tracking, (b) payload variation, and (c) propeller switching. to 5 × 10−3 over 4,000 steps, followed by cosine decay to 1 × 10−4 over 20,000 steps. D. MPPI Controller Parameters The controller parameters of the proposed framework are summarized in Table II. To ensure real-time performance, the MPPI controller is implemented entirely in C++. The learned PyTorch latent dynamics model is exported and optimized using NVIDIA TensorRT for accelerated inference on the NVIDIA Jetson Orin NX. This enables high-frequency rollout evaluation and closed-loop control on embedded hardware. We analyze the runtime performance of the proposed approach on an NVIDIA Jetson Orin, with all computation executed fully on-board. Figure 5 reports the end-to-end control time as a function of rollout horizon U and sample count S, where each timing includes repeated queries to the learned latent dynamics model within MPPI. Since 100 Hz is the minimum rate required for real-time onboard control, the 10 ms line defines the feasible operating boundary. As expected, runtime increases with both U and S, reflecting the higher cost of evaluating more candidate trajectories over longer horizons. The results show that U = 20 and S = 512 lies near this boundary, making it an effective operating point that maximizes controller lookahead and sample diversity while remaining close to the real-time budget. This trade-off is enabled by the lightweight latent dynamics model described in Section V-C, which contains only about 9K parameters and allows the entire framework to run fully on-board on embedded hardware. VI. E XPERIMENTS We evaluate the proposed world model along two complementary axes: (i) offline open-loop prediction, to measure predictive accuracy and long-horizon consistency, and (ii) real time onboard closed-loop control on embedded hardware, to assess deployment performance under real-world conditions.
Table I: Domain randomization parameters used for simulation data collection. Parameter
Randomization
Mass m (kg) Inertia J (kg m2 ) Motor time constant α (s) Drag coefficients D Thrust coefficient kf Torque coefficient km
±50% nominal ±30% nominal [0.01, 0.1] [0.1, 0.5] ±50% nominal ±50% nominal
No. of domains Total trajectories Trajectory duration (s)
500 20000 10
Table II: MPPI Parameters Parameter Prediction timestep ∆t Horizon T Number of samples S Temperature λ Action noise Σ State cost Q Control cost R
Value 0.05 s 15 512 10−4 diag(0.60, 0.15, 0.15, 0.05) diag(400, 40, 20, 20) diag(0.01, 0.05, 0.05, 0.10)
Our experimental study is designed to answer the following questions: 1) Latent Dynamics Modeling: Does modeling dynamics in a latent space yield more informative and control-relevant representations than direct autoregressive prediction in state space? 2) Physics Interpretability: Does embedding kinematic structure enable latent rollouts to retain enough information for accurate recovery of metric-state trajectories in open loop? 3) Zero-Shot Sim-to-Real Transfer: Can a model trained entirely on domain-randomized simulation data transfer to real-world quadrotor navigation without any task-specific
9
S=128
Inference time [ms]
10
Real-time
S=256
S=320
S=512
The second metric is defined as error growth, the expected increase in error between consecutive recursive predictions ERk = E exk − exk−1 . (30)
This metric measures the amount of new error introduced at each rollout step. Lower values indicate slower error 8 accumulation and better long-horizon stability. As shown in Fig. 6 on the left, our method has a higher CR than the 7 predictive baseline for the first few rollout steps, particularly for k < 10. This does not imply that the predictive baseline 6 is better in this regime. At short horizons, both teacher-forced 5 10 15 20 25 30 and recursive errors are small, so the ratio can be sensitive to Unroll length small differences in the denominator of eq. (29). Moreover, Figure 5: Inference speed on NVIDIA Orin NX. Inference values below 1 indicate that recursive rollout error is still lower time increases with unroll length and number of MPPI rollouts than teacher-forced error at that horizon. Therefore, the earlyS; the selected horizon is chosen to remain below the 10 ms horizon behavior mainly reflects local differences between real-time control budget. teacher-forced and rollout trajectories, rather than meaningful long-horizon compounding. The long-horizon trend is more informative. The predictive baseline crosses CRk = 1 around k ≈ 12 and then grows rapidly, reaching approximately 2.4 by fine-tuning? 4) Robustness: Does the proposed framework remain effec- k = 60. This indicates that its recursive rollout error becomes tive under changes in platform geometry, such as payload more than twice as large as its teacher-forced error. In contrast, our method remains close to 1 for a much longer portion of attachment or propeller replacement? 5) Data Quality: How does the quality of the training the horizon and increases more gradually, reaching only about data distribution affect the predictive accuracy of learned 1.4 at k = 60. This shows that the latent-space model remains closer to its teacher-forced behavior under open-loop rollout. quadrotor dynamics? The error growth plot in Fig. 6 on the right confirms this interpretation. The predictive baseline introduces larger new A. Latent Dynamics Modeling error at nearly every rollout step. Around k ≈ 30, its error Compounding Error Analysis. We first evaluate whether growth is approximately 0.14, while ours is around 0.06. By the latent-space dynamics modeling is less susceptible to recursive end of the horizon, the predictive baseline reaches about 0.23, error accumulation than direct autoregressive prediction in state whereas our method remains near 0.11. Therefore, our method space. To this end, we use two complementary metrics: the not only has a lower long-horizon compounding ratio, but also compounding ratio (CR) and error rate (ER). injects less new error at each recursive step. These results show The CR compares recursive open-loop rollout against teacher- that latent-space dynamics modeling substantially mitigates forced prediction considering the same horizon. Let exk,TF recursive error accumulation compared to direct autoregressive denote the teacher-forced error, where the model is conditioned prediction in state space. on the true past states from the dataset, and let exk,rollout denote Takeaway: Latent-space dynamics modeling reduces compounding the open-loop rollout error, where the model is conditioned on error and has a much smaller gap between teacher-forced and its own past predictions. Let as previously defined the state recursive rollout predictions. prediction error at time step k as the root-mean-square error Temporal Straightening. We further analyze the geombetween the predicted state x̃k and the ground-truth state xk r etry of the learned rollouts through the lens of temporal 1 2 (28) straightening, motivated by the hypothesis in [59], [60] that exk = ∥x̃k − xk ∥2 , Ds complex temporal dynamics can be represented as smoother, where Ds denotes the dimensionality of the state vector. We straighter trajectories in representation space. The intuition is that a NN model becomes easier to roll out when its internal then define the compounding ratio as representation space evolves in a consistent temporal direction: x ek,rollout CRk = x . (29) if consecutive latent displacements are well aligned, the trajecek,TF tory unfolds smoothly over time and future states are easier This ratio isolates the excess error caused by recursive predic- to extrapolate recursively. In contrast, if the representation tion. Values near 1 indicate that open-loop rollout remains close repeatedly bends, oscillates, or backtracks, then small local to teacher-forced performance, while larger values indicate prediction errors can more easily deflect the rollout away stronger compounding effects. Values below 1 indicate that from the true temporal progression. Following this idea, we the rollout error is lower than the teacher-forced error at that evaluate temporal straightening over N trajectories. For the i-th horizon. This suggests that the model’s own predicted trajectory trajectory, let the sequence of predicted latent representations (i) T ×D̄ , and define the temporal velocity vectors as is locally easier to predict than the ground-truth trajectory, or be s̃1:T ∈ R (i) (i) (i) that teacher-forced inputs introduce larger local inconsistencies. s̃˙ t = s̃t+1 − s̃t . The smoothness score for a single trajectory 9
10
Pred.
Ours
0.20
2.0 Error Rate
Compounding Ratio
2.5
1.5 1.0 0.5
0.15 0.10 0.05
0.0
0.00 10
20
30 Unroll Length
40
50
60
10
20
30 Unroll Length
40
50
60
Figure 6: Recursive rollout error analysis. The compounding ratio (left) compares open-loop prediction to teacher-forced prediction; values above 1 indicate error accumulation caused by recursion. The error growth rate (right) measures the additional pose error introduced at each rollout step. Our method stays closer to teacher-forced behavior and has lower error growth than the predictive baseline, showing that latent-space dynamics modeling mitigates long-horizon compounding error. 5
20
Pred.
Pred.
PC2
y [m]
Recon.
0
Recon.
0 Ours
−20
Ours
−20
0 20 x [m] (a) Trajectory
−5
−10
−5
0 PC1 (b) Latent trajectory
5
−1
0 1 Smoothness (c) Temporal straightening
Figure 7: Temporal straightening analysis of latent rollouts. (a) Example Cartesian trajectory, with shading indicating progression from earlier to later states. (b) PCA projection of the corresponding latent trajectories, showing how each model evolves in representation space over time. (c) Temporal straightening score; higher values indicate smoother and more directionally consistent latent evolution. Temporally straighter latent trajectories suggest that complex system dynamics are represented in a simpler geometry, making recursive prediction easier and reducing directional drift during rollout. is computed as the mean pairwise cosine similarity between consecutive latent velocities D E (i) ˙ (i) ˙ T −2 s̃ , s̃ X t t+1 1 (i) . Sstraight = (i) T − 2 t=1 ∥s̃˙ (i) ˙ ∥ t ∥ ∥s̃ t+1
We then report the mean and variance across the N evaluated trajectories. A value close to 1 is ideal, since it indicates that successive latent displacements are highly aligned and the rollout follows a smooth, nearly straight path. Values near 0 indicate weak alignment and a more curved or wandering trajectory, while negative values indicate strong directional inconsistency, such as oscillation or backtracking. Figure 7 shows a clear separation between direct predictive modeling and the two latent-space approaches. Averaged over the N evaluated trajectories, the predictive baseline attains a negative mean straightening score of roughly −0.4, with a large spread extending from about −1.0 to 0.2, indicating that its rollouts often change direction and evolve along a temporally inconsistent path. In contrast, both latent models remain strongly in the positive regime: the reconstruction-based
latent model achieves the highest mean score, around 0.95, while our JEPA-based model attains a mean of about 0.75. Thus, despite their different training objectives, both latent models produce substantially smoother and straighter temporal evolution than the predictive baseline. This suggests that an important emergent property of latent dynamics learning is temporal smoothness in representation space, which makes recursive rollouts easier to maintain over long horizons. In our setting, this result helps explain why both latent approaches outperform direct predictive modeling: by organizing temporal evolution into smoother latent trajectories, they reduce directional drift under recursive rollout and are therefore less susceptible to compounding error. While the reconstruction model achieves the highest straightening score, our model remains strongly positive and, when interpreted together with the rollout-error results, indicates that latent-space modeling itself is a key ingredient for learning temporal representations.
11
Ours + PI Prober
Ours + Prober
Recon + PI Prober
Recon + Prober
20 15 10 5 0
20
40
Unroll Length
60
15
Attitude RMSE [deg]
Velocity RMSE [m/s]
Position RMSE [m]
25
10 5 0
20
40
Unroll Length
60
Predictive
120 100 80 60 40 20 0
20
40
Unroll Length
60
Figure 8: Open-loop rollout fidelity across dynamics models. Position, velocity, and attitude RMSE are shown versus unroll length, where lower error indicates more accurate long-horizon prediction. Direct predictive models accumulate error rapidly, especially in attitude, while latent dynamics models grow more slowly. The proposed physics-inspired prober substantially improves metric-state recovery, showing that structured physical decoding is critical for accurate state prediction. 16
Takeaway: Both latent models learn temporally smoother latent trajectories than direct predictive modeling, suggesting that temporal smoothness is an important emergent property.
Pred.
Ours
12
ex
Robustness to Noise. We next evaluate whether the learned 8 dynamics remain reliable under corrupted sensing. To simulate observation noise, we perturb the input observation history with i.i.d. Gaussian noise before passing it to the proposed 4 framework. The corruption is applied only to the model input. The reference ground-truth trajectory remains unchanged for 0 evaluation. We then recursively unroll each model over the 0.00 0.05 0.10 0.15 0.20 0.25 0.30 Noise level prediction horizon and compute the pose RMSE between the predicted trajectory, obtained through the latent dynamics model Figure 9: Robustness to observation corruption. Pose RMSE and physics-inspired prober, and the ground-truth trajectory. under increasing i.i.d. Gaussian observation noise, showing Figure 9 shows that our method remains consistently more that the JEPA model consistently outperforms the predictive robust than the predictive baseline as the observation noise baseline and remains more robust as corruption increases. increases. At zero noise, our method reduces the median state RMSE by approximately 55% compared to the predictive baseline. As the noise level increases, this advantage remains clear. At moderate noise levels, our method achieves roughly a to metric state space, we compare six baselines. Predictive 25–30% reduction in median state RMSE. Even at the highest is a direct autoregressive state-space predictor. Predictive + noise level, where both methods degrade, our method still Physics Reg. augments this baseline with kinematic physics maintains a lower median error, with an improvement of regularization loss during training. Recon + Prober is a latent about 10% over the predictive baseline. The distributional dynamics model in which the embeddings are trained through trend is also important. The predictive baseline exhibits wider a reconstruction: collapse is avoided by forcing the predicted error distributions and heavier upper tails as noise increases, embeddings to decode back the full recorded metric state and indicating more frequent high-error rollouts. In contrast, our the prober is a 3-layed MLP. Recon + PI Prober replaces the method produces more concentrated distributions across noise unconstrained prober in this model with the proposed physicslevels. This suggests that the latent dynamics model, together inspired probing mechanism. Ours + Prober is our JEPA-based with the physics-inspired prober, is less sensitive to input- latent dynamics model with a 3-layered MLP prober. Finally, level perturbations and yields more consistent long-horizon Ours + PI Prober is the full method, combining JEPA latent predictions under sensing corruption. dynamics with the proposed physics-inspired prober. Together, these baselines isolate three design choices: direct predictive Takeaway: Our JEPA-style approach remains more accurate under versus latent-space dynamics modeling, reconstruction-based input corruption, indicating stronger robustness to corrupted inputs. versus JEPA-based embedding learning, and unconstrained versus physics-grounded state. B. Physics Interpretability To evaluate the prediction accuracy of each baseline, we Baselines. To evaluate whether injecting physical structure decompose the state error into position and attitude components. improves the open-loop fidelity of mapping latent rollouts Let p̃k , pk ∈ R3 denote the predicted and ground-truth
12
1 y [m]
y [m]
y [m]
0
60
1
2
0
40 0 20
−1 3
4
5 x [m]
6
10
7
12
14 x [m]
16
18
3
4
5 x [m]
6
Tracking error [cm]
Reference
7
Figure 10: Real-world zero-shot trajectory tracking using the proposed controller. The plots show executed flight trajectories overlaid on the desired references. Color indicates tracking error in centimeters, with darker regions corresponding to lower error. The controller closely follows diverse real-world trajectories using a dynamics model trained only in simulation. Table III: Open loop prediction analysis of different neural dynamics frameworks. Approach Predictive Predictive + Physics Reg. [8] Reconstruction + Prober Reconstruction + PI Prober Ours + Prober Ours + PI Prober
Pos. RMSE [m] ↓
Att. Err. [◦ ] ↓
Mean
Var.
Mean
Var.
8.80 7.12 6.82 1.53 5.56 1.43
2.3 2.1 1.6 0.13 1.31 0.10
53.4 49.1 45.2 5.28 40.20 4.71
14.9 13.2 9.7 0.70 9.30 0.50
positions at time step k, respectively, and let R̃k , Rk ∈ SO(3) denote the corresponding predicted and ground-truth rotation matrices. The position error is defined as
underlying instability of autoregressive prediction. In contrast, the latent baselines without PI probing already perform better: Recon + Prober achieves 6.82 m position RMSE and 45.2◦ attitude error, while Ours + Prober improves further to 5.56 m and 40.2◦ . Relative to the predictive baseline, this corresponds to reductions of 22.5% and 33.8% in position error, and 15.4% and 24.7% in attitude error, for Recon + Prober and Ours + Prober, respectively. Qualitatively, the rollout curves in Fig. 8 show the same trend: the predictive variants exhibit rapidly growing position, velocity, and especially attitude error, whereas the latent methods remain consistently lower across the horizon. These results suggest that latent dynamics modeling is already a better design choice than direct autoregressive prediction, and that simply adding physics regularization to a predictive model is not enough.
Training Objective. We next compare the two latent learning paradigms and the effect of injecting kinematic structure into ep (31) state recovery. Without PI probing, Ours + Prober outperforms k = ∥p̃k − pk ∥2 . The attitude error is computed using the relative rotation Recon + Prober, reducing mean position RMSE from 6.82 to 5.56 m and mean attitude error from 45.2◦ to 40.2◦ . This between the ground-truth and predicted orientations: shows that JEPA-style latent training produces a stronger Rerr,k = R⊤ (32) predictive representation than reconstruction-based training, k R̃k . We map this relative rotation to the Lie algebra so(3) using where embeddings are optimized to reconstruct the recorded state and avoid collapse through decoder supervision. However, the logarithmic map: ∨ the largest gains appear only after introducing the PI Prober. 3 ϕk = LogSO(3) R⊤ R̃ ∈ R , (33) For the reconstruction model, replacing the generic prober k k with the PI Prober reduces position RMSE from 6.82 to 1.53 where (·)∨ converts a skew-symmetric matrix in so(3) to its m and attitude error from 45.2◦ to 5.28◦ , corresponding to corresponding vector representation. The scalar attitude error roughly 4.5× and 8.6× improvements. For our JEPA model, is then given in degrees as the same replacement reduces position RMSE from 5.56 to 180 1.43 m and attitude error from 40.2◦ to 4.71◦ , i.e., about R ek = ∥ϕk ∥2 . (34) π 3.9× and 8.5× improvements. Variance is reduced just as Physics Structure in NN Design. Figure 8 and Table III sharply, dropping from 1.31 to 0.10 in position and from first reveal a clear advantage of latent dynamics models over 9.30◦ to 0.50◦ in attitude for the JEPA model. The qualitative direct autoregressive prediction. The weakest performance is rollout curves (see Figure 8) reinforce the same trend. This obtained by the predictive baseline, which reaches 8.80 m suggests that the proposed kinematic structure makes the statemean position RMSE and 53.4◦ mean attitude error, with recovery problem substantially easier: instead of learning metric large rollout variance (2.3 m and 14.9◦ respectively). Adding dynamics entirely from scratch through a flexible decoder, the kinematic physics regularization improves this baseline only model only needs to predict structured corrections on top of modestly, reducing the mean position RMSE to 7.12 m and a parameter-free prior. The PI Prober is not learning the full the mean attitude error to 49.1◦ , corresponding to relative dynamics end-to-end. It rather learns the missing dynamics improvements of only 19% and 8%. This indicates that physics- needed to complement the imposed kinematic structure and inspired penalties help, but are insufficient to overcome the recover the full system evolution. Importantly, in our setup this
13
Table IV: Quantitative results of real-world trajectory tracking across reference trajectories. Each trajectory is executed 5 times. The mean and variance are reported. Trajectory
Circle Oval Figure 8 Fish Lemniscate
∥v∥max ∥v̇∥max [m s−1 ] [m s−2 ]
2.45 4.50 5.20 5.70 7.20
1.40 3.78 5.44 7.68 12.5
Position RMSE [m] MPPI (Pred.+Phy.) MPPI (Pred.)
Ours
Attitude Error [◦ ] MPPI (Pred.+Phy.) MPPI (Pred.)
Ours
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
0.24 0.33 0.35 0.40 0.45
0.010 0.018 0.022 0.031 0.047
0.36 0.44 0.47 0.54 0.56
0.022 0.083 0.041 0.056 0.076
0.39 0.48 0.51 0.59 0.61
0.024 0.096 0.044 0.061 0.083
7.87 9.11 9.25 10.78 19.43
0.82 1.06 1.18 1.42 2.65
10.99 15.20 17.75 20.95 26.83
3.66 5.46 5.88 6.85 7.51
11.95 16.53 20.20 22.78 29.16
3.81 5.68 6.10 7.19 7.90
Table V: Quantitative results of closed-loop tracking under propeller switching and payload scenarios. All trajectories were executed at an average velocity of 2 ms−1 and executed 5 times. The table reports the mean and variance. Scenario
Trajectory Ours
Position RMSE [m] MPPI (Pred. + Phy.)
MPPI (Pred.)
Ours
Attitude Error [◦ ] MPPI (Pred. + Phy.)
MPPI (Pred.)
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
Mean
Var.
Propeller Switching
Circle Figure 8 Fish
0.33 0.35 0.39
0.02 0.01 0.02
0.43 0.46 0.51
0.03 0.01 0.03
0.45 0.47 0.53
0.03 0.01 0.03
9.54 10.21 10.89
1.04 0.98 0.88
12.40 13.27 14.16
1.35 1.27 1.14
12.88 13.78 14.70
1.40 1.32 1.19
Payload Transportation
Circle Figure 8 Fish
0.46 0.49 0.53
0.08 0.07 0.08
0.60 0.64 0.69
0.10 0.09 0.10
0.62 0.66 0.72
0.11 0.09 0.11
10.11 9.44 11.87
1.43 1.37 1.92
13.14 12.27 15.43
1.86 1.78 2.50
13.65 12.74 16.02
1.93 1.85 2.59
2
60
1
1
40
0
0 5
6
7
8
9
10
x [m] (a) Payload
20 6
7
8
9
10
Tracking error [cm]
2
y [m]
y [m]
Reference
11
x [m] (b) Propeller Switching
Figure 11: Real-world tracking under platform changes without retraining. (a) Payload transportation modifies the mass and inertial properties of the quadrotor. (b) Propeller switching changes the actuation characteristics. The executed trajectories remain close to the reference paths, with color indicating tracking error in centimeters. These rollouts show that the proposed framework remains effective under non-nominal deployment conditions. structure can be imposed without requiring nominal parameter estimation. Overall, these results show that JEPA provides the stronger dynamics model, while the PI Prober is the key mechanism that converts embeddings into accurate and stable metric-state trajectories for control. Takeaway: Embedding kinematic structure enables high-fidelity metric-state recovery, showing that the latent rollouts retain sufficient information to reconstruct the full dynamics.
C. Zero-Shot Sim-to-Real Transfer We now evaluate how far the proposed framework performs when trained entirely on domain-randomized simulation data, with no task-specific fine-tuning on real-world trajectories. We compare three controllers in closed loop: Ours, which uses the proposed framework; MPPI (Predictive + Physics Regularization), which uses a direct predictive state-space model trained with physics regularization; and MPPI (Predictive), which uses a standard predictive state-space model without additional physical grounding. These experiments therefore test whether
14
the benefits observed in open-loop prediction translate into real control performance under sim-to-real transfer. Each trajectory is executed 5 times and we report the mean and varience of the tracking performance. Table IV shows a consistent advantage of the proposed method across all reference trajectories. In terms of position tracking, Ours achieves the lowest mean RMSE on every trajectory, reducing the error from 0.39 to 0.24 m on the circle (≈ 38% improvement over MPPI (Predictive)), from 0.48 to 0.33 m on the oval (≈ 31%), from 0.51 to 0.35 m on the figure-8 (≈ 31%), from 0.59 to 0.40 m on the fish trajectory (≈ 32%), and from 0.61 to 0.45 m on the lemniscate (≈ 26%). Similar gains are obtained relative to MPPI (Predictive + Physics Regularization), with position-error reductions ranging from about 20% to 33%. The same trend appears in attitude tracking. Relative to MPPI (Predictive), our method reduces mean attitude error from 11.95◦ to 7.87◦ on the circle (≈ 34%), from 16.53◦ to 9.11◦ on the oval (≈ 45%), from 20.20◦ to 9.25◦ on the figure-8 (≈ 54%), from 22.78◦ to 10.78◦ on the fish trajectory (≈ 53%), and from 29.16◦ to 19.43◦ on the lemniscate (≈ 33%). Variance is also consistently lower for our method in both position and attitude, indicating not only better average tracking but also more repeatable closed-loop behavior across trials. The qualitative tracking examples in Fig. 10 illustrate representative real-world tracking of the proposed method across multiple reference trajectories. These examples show that the controller is able to track diverse closed-loop paths in the real world using a dynamics model trained entirely in simulation, with the color-coded error remaining generally concentrated along the reference trajectory. The quantitative results in Table IV provide the stronger evidence: across all trajectories, our method consistently achieves lower position and attitude errors than both predictive baselines, and this advantage persists as the trajectories become faster and more aggressive. Overall, these results suggest that simulation alone can be sufficient for real-world control when the learned dynamics model is structured appropriately. A likely reason is that the proposed framework combines three complementary ingredients: latent dynamics modeling reduces the compounding effects of direct autoregressive state prediction, domain randomization broadens the training distribution to better capture real-world variability, and the physics-inspired prober grounds latent rollouts in metric state space so that the model remains useful for control. Together, these components enable more reliable zero-shot transfer from simulation to the real platform than purely predictive alternatives. D. Robustness We finally evaluate whether the proposed framework remains effective when the real platform deviates from its nominal training configuration. In particular, we consider two deployment scenarios that alter the vehicle dynamics without any retraining or task-specific adaptation: propeller switching, which changes the actuation characteristics, and payload transportation, which modifies the vehicle mass and inertial properties. These scenarios were not covered during data collection. These experiments are significant because such
changes are common in real deployments and can induce substantial model mismatch. All trajectories were executed at an average velocity of 2 ms−1 and executed 5 times. We compare the same three closed-loop controllers as before: Ours, MPPI (Pred.+Phy.), and MPPI (Pred.). The goal is to assess whether latent-space dynamics modeling yields a controller that is more robust to platform variations than direct predictive baselines. Table V shows that our method achieves the best performance across both deployment scenarios. Under propeller switching, our method consistently attains about 1.3× lower position RMSE than MPPI (Pred.) and about 1.2× lower error than MPPI (Pred.+Phy.) across the evaluated trajectories. A similar trend appears in attitude tracking, where our method delivers roughly 1.3× lower error than the predictive baseline and about 1.25× lower error than the physics-regularized variant. Under payload transportation, with a payload of 300g the same pattern persists. Our method again achieves approximately 1.35× lower position RMSE than MPPI (Pred.) and about 1.27× lower error than MPPI (Pred.+Phy.). Attitude error follows a nearly identical trend, with our method consistently providing about 1.4× lower error than the predictive baseline. Variance is also lower across both position and attitude metrics, indicating not only improved tracking accuracy but also more repeatable behavior under model mismatch. The qualitative examples in Fig. 11 illustrate representative tracking rollouts of our method under both propeller switching and payload attachment. Even under these non-nominal conditions, the controller remains close to the reference path, with tracking error remaining concentrated along the trajectory rather than diverging in curved segments. When interpreted together with the quantitative results, these experiments suggest that the proposed latent dynamics model is more robust than predictive models because it captures a more compact and transferable representation of system evolution, rather than depending directly on precise next-state prediction under a fixed nominal configuration. As a result, when the platform dynamics shift due to changes in actuation or mass properties, the learned latent dynamics appear to degrade more gracefully than direct predictive state-space models. Takeaway: A well covered diverse domain-randomized simulation data is sufficient for zero-shot sim2real deployment.
E. Data Quality Trajectory Distribution Quality. The performance of a learned dynamics model depends strongly on the distribution of transitions used for training (see Section IV-E). We therefore introduce a Trajectory Distribution Quality (TDQ) score to quantify whether D provides broad, dynamically informative, and robust coverage for learning the transition map. Since the relevant variables are continuous and high-dimensional, measuring coverage of the state–action space is difficult. Therefore, we discretize each normalized feature space using clustering, which provides an empirical partition of the data distribution into local regions. The occupancy of these regions can then be used to compute an entropy-based measure of distributional spread [61], [62], while the number of occupied
15
5.0
ex
4.0
# of samples 1k 10k 100k 1.0M 1.5M
gives the state–action coverage score Scov , transition richness score Sdyn , and parameter robustness score Sη , respectively. The final TDQ score is defined as the harmonic mean TDQ(D) =
3 1 1 . + Scov Sdyn + Sη 1
(38)
3.0
The harmonic mean penalizes datasets that score poorly in any one component, ensuring that a high TDQ value requires broad state–action coverage, diverse local transition behavior, and 2.0 sufficient coverage of the randomized simulator parameters. To evaluate whether the proposed TDQ score reflects 0.00 0.20 0.40 0.60 0.80 the usefulness of a dataset for dynamics learning, we train TDQ the proposed latent dynamics framework using datasets of Figure 12: As dataset size increases, TDQ rises from 0.01 to increasing size and compute the resulting state prediction 0.94, while state RMSE ex decreases from 5.4 to 1.4, showing error ex . For each dataset size, we compute TDQ from the that higher trajectory distribution quality correlates with better corresponding training distribution and evaluate the trained model on the same held-out test set. Figure 12 shows a clear learned dynamics prediction. inverse relationship between TDQ and predictive error: as TDQ increases, the state RMSE ex decreases. With only 1k samples, regions measures coverage. This clustering-based discretization the dataset has a very low TDQ of approximately 0.01, and the is similar in spirit to vector quantization and K-means- model produces a large state error of about 5.4. Increasing the based density summarization [63], [64]. TDQ measures three dataset to 10k samples improves TDQ to roughly 0.08, reducing properties that directly affect neural dynamics learning: state– ex to about 2.5. With 100k samples, TDQ further increases action coverage, transition richness, and parameter robustness. to approximately 0.25, and the error decreases to about 1.9, Each trajectory in the dataset is obtained using a simulator indicating that additional data substantially improves state– parameter vector η (see eq. (26)), which specifies the physical action coverage, transition richness, and parameter robustness. parameters of the randomized quadrotor model. For each Beyond this point, the curve begins to flatten: increasing the transition i, we define the augmented state–action feature vector dataset size to 1.0M and 1.5M raises TDQ to approximately 0.91 and 0.94, respectively, but only reduces ex from about ⊤ ⊤ ⊤ ⊤ ⊤ r r ω a yi = vi⊤ r⊤ , (35) 1.5 to 1.4. This saturation suggests that the dataset increasingly x,i y,i z,i i i covers the relevant dynamics manifold, and that further gains which includes the velocity, attitude, angular velocity, and are likely limited by model capacity, optimization, or residual motor forces. We omit absolute position from yi because the dynamics complexity rather than data coverage alone. local quadrotor dynamics are primarily governed by velocity, Takeaway: Higher TDQ leads to lower state prediction error, attitude, angular velocity, and control input. To measure the indicating that broader state–action coverage, richer transitions, diversity of local transitions, we define and better parameter diversity improve learned dynamics prediction. ⊤ ⊤ gi = yi ∆yi⊤ , ∆yi = yi+1 − yi . (36) The feature yi measures which regions of the state–action space are visited, while gi measures how the system locally evolves from those regions. After normalizing each feature space, we cluster the state– action features yi , transition features gi , and simulator parameter vectors η using Ky , Kg , and Kη clusters, respectively. For a generic clustered feature space with K clusters, let qk denote the empirical occupancy probability of cluster k, and let nk denote the number of samples assigned to that cluster. We define the entropy–coverage score ! ! K K 1 X 1 X S= − qk log(qk ) I[nk ≥ nmin ] , log K K k=1 k=1 (37) where the first term is a normalized Shannon entropy that measures how uniformly the data occupies the clustered feature space, and the second term measures the fraction of clusters that are sufficiently represented. The threshold nmin prevents isolated or rarely visited regions from being counted as meaningfully covered. Applying eq. (37) to yi , gi , and η
VII. D ISCUSSION
The experimental results demonstrate that the proposed framework satisfies the four key properties of a useful world model for quadrotor control. First, the model improves longhorizon prediction by reducing recursive error accumulation, as shown by the compounding ratio and error growth analysis in Fig. 6. Second, the physics-inspired prober makes the latent rollout interpretable by mapping learned representations back to physically meaningful state variables, which is essential for model-based control. This is supported by the open-loop prediction results in Fig. 8 and Table III, where the proposed PI Prober substantially improves metric-state recovery. Third, the resulting model is real-time for real-time control, since it is directly used inside an MPPI controller onboard an embedded device. Finally, the model can zero-shot generalize across tasks: the same learned model is used across multiple trajectories and deployment conditions without task-specific retraining or fine-tuning, as shown in Tables IV and V. A central result of this work is that a well-designed simulation data pipeline, combined with an appropriate learning
16
framework, can be sufficient for zero-shot sim-to-real transfer. automated dataset generation. Across open-loop prediction, Rather than relying on accurate system identification or noise robustness, zero-shot sim-to-real transfer, and platformtask-specific fine-tuning, our approach systematically collects variation experiments, the proposed framework consistently diverse trajectories and exposes the model to a broad family of outperformed direct predictive baselines and demonstrated plausible quadrotor dynamics through domain randomization, robust real-world deployment without task-specific fine-tuning. as summarized in Fig. 12. The TDQ analysis in Fig. 12 further Future works will extend this framework from lowsupports this design choice by showing that higher-quality dimensional state inputs to high-dimensional observations such training distributions consistently lead to lower state prediction as RGB and RGB-D images. This setting is a natural fit error. This indicates that broader state–action coverage, richer for JEPA-style learning, since direct predictive reconstruction transition behavior, and greater parameter diversity improve the becomes increasingly expensive and often forces the model usefulness of the dataset for learning dynamics. The improve- to preserve task-irrelevant visual details. Such an extension ment is most pronounced when moving from small, poorly would open the door to a broad class of visual navigation covered datasets to larger datasets that better span the relevant tasks, including goal-directed flight from onboard camera dynamics distribution; as coverage increases, the gains begin observations, obstacle avoidance in cluttered environments, and to saturate, suggesting that further improvements are likely navigation toward semantic targets. This direction is especially limited by model capacity, optimization, or residual dynamics important for quadrotors, where observations compared to complexity. The real-world tracking results in Table IV show other domains like in robot manipulation, are egocentric and that this data-generation strategy is sufficient to outperform tightly coupled to the vehicle’s underactuated dynamics. A predictive baselines across diverse reference trajectories. More second direction is to incorporate safety more explicitly into importantly, the robustness experiments in Table V show that the learned representation and planning objective. We would the same model remains effective under deployment changes like to investigate how latent dynamics models can encode such as propeller switching and payload transportation. This safety-relevant structures, such as obstacle proximity, visibility, suggests that the learned representation captures dynamics that uncertainty, and recoverability, to support reliable planning in are not tied to a single nominal platform. While this work cluttered and uncertain environments. focuses on simulation-only training, the same framework could also be applied directly to real-world data to capture difficultR EFERENCES to-model nonlinear effects, especially at high speeds where [1] V. Kumar and N. Michael, “Opportunities and challenges with auaerodynamic and actuator effects become more pronounced. tonomous micro aerial vehicles,” The International Journal of Robotics Another practical advantage of the proposed framework Research, vol. 31, no. 11, 2012, pp. 1279–1291. is the simplicity of the SIGReg training objective. Many [2] Y. Wang, Q. Lu, and B. Ren, “Wind turbine crack inspection using a quadrotor with image motion blur avoided,” IEEE Robotics and representation learning methods require multiple carefully Automation Letters, vol. 8, no. 2, 2023, pp. 1069–1076. tuned loss terms [38], [50], target networks, stop-gradient [3] K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep reinforcement choices [51], or reconstruction weights. In contrast, SIGReg learning in a handful of trials using probabilistic dynamics models,” Advances in Neural Information Processing Systems, vol. 31, 2018. introduces a compact anti-collapse mechanism with only one [4] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg, B. Boots, main regularization weight, λsig , to tune. This makes the and E. A. Theodorou, “Information theoretic mpc for model-based objective easier to use in practice and more appealing for reinforcement learning,” in IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 1714–1721. general-purpose dynamics representation learning. Combined [5] T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker, “Model-based with the physics-inspired prober, this leads to a framework reinforcement learning: A survey,” Foundations and Trends in Machine that is both expressive and structured: the latent model learns Learning, vol. 16, no. 1, 2023, pp. 1–118. [6] G. Loianno, C. Brunner, G. McGrath, and V. Kumar, “Estimation, control, temporally coherent predictive representations, while the prober and planning for aggressive flight with a small quadrotor with a single converts them into interpretable rollouts suitable for control. camera and imu,” IEEE Robotics and Automation Letters, vol. 2, no. 2, VIII. C ONCLUSION AND F UTURE W ORKS In this work, we introduced the first JEPA-style latent dynamics framework for real-time quadrotor control. The proposed approach learns predictive representations of system evolution without directly reconstructing future states, and uses a physics-inspired prober to map frozen latent rollouts into physically meaningful metric states. This design addresses the four key requirements of a useful world model for aerial control. It improves long-horizon prediction by reducing recursive error accumulation, provides interpretable state-space rollouts for evaluating costs and constraints, remains computationally practical for real-time MPPI control, and is task-agnostic across trajectories and deployment conditions. To reduce reliance on costly and risky real-world data collection, we also developed a domain-randomized simulation pipeline for
pp. 404–411, 2017. [7] J. Eschmann, D. Albani, and G. Loianno, “Data-driven system identification of quadrotors subject to motor delays,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 8095– 8102. [8] A. Saviolo, G. Li, and G. Loianno, “Physics-inspired temporal learning of quadrotor dynamics for accurate model predictive trajectory tracking,” IEEE Robotics and Automation Letters, vol. 7, no. 4, 2022, pp. 10256– 10263. [9] L. Bauersfeld, E. Kaufmann, P. Foehn, S. Sun, and D. Scaramuzza, “Neurobem: Hybrid aerodynamic quadrotor model,” in Robotics: Science and Systems (RSS), 2021. [10] P. P. Rao, A. Saviolo, T. C. Ferrari, and G. Loianno, “Learning longhorizon predictions for quadrotor dynamics,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024, pp. 12 758– 12 765. [11] A. Saviolo and G. Loianno, “Learning quadrotor dynamics for precise, safe, and agile flight control,” Annual Reviews in Control, vol. 55, 2023, pp. 45–60. [12] T. Duong and N. Atanasov, “Hamiltonian-based neural ode networks on the se(3) manifold for dynamics learning and control,” in Robotics: Science and Systems (RSS), 2021.
17
[13] A. Punjani and P. Abbeel, “Deep learning helicopter dynamics models,” in IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 3223–3230. [14] L. Hewing, J. Kabzan, and M. N. Zeilinger, “Cautious model predictive control using gaussian process regression,” IEEE Transactions on Control Systems Technology, vol. 28, no. 6, 2019, pp. 2736–2743. [15] J. Fu, S. Levine, and P. Abbeel, “One-shot learning of manipulation skills with online dynamics adaptation and neural network priors,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2016, pp. 4019–4026. [16] A. Saviolo, J. Frey, A. Rathod, M. Diehl, and G. Loianno, “Active learning of discrete-time dynamics for uncertainty-aware model predictive control,” IEEE Transactions on Robotics, vol. 40, 2023, pp. 1273–1291. [17] L. Wang, E. A. Theodorou, and M. Egerstedt, “Safe learning of quadrotor dynamics using barrier certificates,” in IEEE International Conference on Robotics and Automation (ICRA), 2018, pp. 2460–2465. [18] T. Lew, A. Sharma, J. Harrison, A. Bylard, and M. Pavone, “Safe active dynamics learning and control: A sequential exploration–exploitation framework,” IEEE Transactions on Robotics, vol. 38, no. 5, 2022, pp. 2888–2907. [19] M. O’Connell, G. Shi, X. Shi, K. Azizzadenesheli, A. Anandkumar, Y. Yue, and S.-J. Chung, “Neural-fly enables rapid learning for agile flight in strong winds,” Science Robotics, vol. 7, no. 66, 2022, pp. eabm6597. [20] T. Z. Jiahao, K. Y. Chee, and M. A. Hsieh, “Online dynamics learning for predictive control with an application to aerial robots,” in Conference on Robot Learning, 2023, pp. 2251–2261. [21] J. Robine, M. Höftmann, T. Uelwer, and S. Harmeling, “Transformerbased world models are happy with 100k interactions,” in International Conference on Learning Representations (ICLR), 2023. [22] C. Finn and S. Levine, “Deep visual foresight for planning robot motion,” in IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 2786–2793. [23] B. Wu, S. Nair, L. Fei-Fei, and C. Finn, “Example-driven model-based reinforcement learning for solving long-horizon visuomotor tasks,” in Conference on Robot Learning (CoRL), 2021. [24] S. L. Brunton, J. L. Proctor, and J. N. Kutz, “Discovering governing equations from data by sparse identification of nonlinear dynamical systems,” Proceedings of the National Academy of Sciences, vol. 113, no. 15, 2016, pp. 3932–3937. [25] G. Kulathunga, H. Hamed, and A. Klimchik, “Residual dynamics learning for trajectory tracking for multi-rotor aerial vehicles,” Scientific Reports, vol. 14, no. 1, 2024, pp. 1858. [26] S. Bansal, A. K. Akametalu, F. J. Jiang, F. Laine, and C. J. Tomlin, “Learning quadrotor dynamics using neural network for flight control,” in IEEE 55th Conference on Decision and Control (CDC), 2016, pp. 4653–4660. [27] N. Mohajerin and S. L. Waslander, “Multistep prediction of dynamic systems with recurrent neural networks,” IEEE Transactions on Neural Networks and Learning Systems, vol. 30, no. 11, 2019, pp. 3370–3383. [28] S. Looper and S. L. Waslander, “Temporal convolutions for multi-step quadrotor motion prediction,” in 2022 19th Conference on Robots and Vision (CRV), 2022, pp. 32–39. [29] M. Assran, Q. Duval, I. Misra, P. Bojanowski, P. Vincent, M. Rabbat, Y. LeCun, and N. Ballas, “Self-supervised learning from images with a joint-embedding predictive architecture,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 15 619–15 629. [30] A. Bardes, Q. Garrido, J. Ponce, X. Chen, M. Rabbat, Y. LeCun, M. Assran, and N. Ballas, “V-jepa: Latent video prediction for visual representation learning,” arXiv preprint arXiv:2312.14119, 2023. [31] M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, M. Muckley, A. Rizvi, C. Roberts, K. Sinha, A. Zholus, et al., “V-jepa 2: Selfsupervised video models enable understanding, prediction and planning,” arXiv preprint arXiv:2506.09985, 2025. [32] A. Bardes, J. Ponce, and Y. LeCun, “Mc-jepa: A joint-embedding predictive architecture for self-supervised learning of motion and content features,” arXiv preprint arXiv:2307.12698, 2023. [33] G. Zhou, H. Pan, Y. LeCun, and L. Pinto, “Dino-wm: World models on pre-trained visual features enable zero-shot planning,” in International Conference on Machine Learning (ICML), 2025. [34] R. G. Goswami, A. Bar, D. Fan, T.-Y. Yang, G. Zhou, P. Krishnamurthy, M. Rabbat, F. Khorrami, and Y. LeCun, “World models for learning dexterous hand-object interactions from human videos,” arXiv preprint arXiv:2512.13644, 2026.
[35] A. Bar, G. Zhou, D. Tran, T. Darrell, and Y. LeCun, “Navigation world models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 15 791–15 801. [36] S. Yin, K. Yin, W. Chen, Y. Liu, G. Li, and L. Lin, “Ddp-wm: Disentangled dynamics prediction for efficient world models,” arXiv preprint arXiv:2602.01780, 2026. [37] L. Maes, Q. L. Lidec, D. Scieur, Y. LeCun, and R. Balestriero, “Leworldmodel: Stable end-to-end joint-embedding predictive architecture from pixels,” arXiv preprint arXiv:2603.19312, 2026. [38] V. Sobal, W. Zhang, K. Cho, R. Balestriero, T. G. J. Rudner, and Y. LeCun, “Learning from reward-free offline data: A case for planning with latent dynamics models,” arXiv preprint arXiv:2502.14819, 2025. [39] N. O. Lambert, D. S. Drew, J. Yaconelli, S. Levine, R. Calandra, and K. S. J. Pister, “Low-level control of a quadrotor with deep model-based reinforcement learning,” IEEE Robotics and Automation Letters, vol. 4, no. 4, 2019, pp. 4224–4230. [40] C. Li, A. Krause, and M. Hutter, “Robotic world model: A neural network simulator for robust policy optimization in robotics,” in NeurIPS 2025 Workshop on Embodied World Models for Decision Making, 2025. [41] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson, “Learning latent dynamics for planning from pixels,” in International Conference on Machine Learning, 2019, pp. 2555–2565. [42] V. Micheli, E. Alonso, and F. Fleuret, “Transformers are sample-efficient world models,” in International Conference on Learning Representations (ICLR), 2023. [43] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse control tasks through world models,” Nature, vol. 640, 2025, pp. 647–653. [44] A. Verraest, S. Bahnam, R. Ferede, G. de Croon, and C. De Wagter, “Skydreamer: Interpretable end-to-end vision-based drone racing with model-based reinforcement learning,” arXiv preprint arXiv:2510.14783, 2025. [45] M. Krinner, E. Aljalbout, A. Romero, and D. Scaramuzza, “Accelerating model-based reinforcement learning with state-space world models,” arXiv preprint arXiv:2502.20168, 2025. [46] A. Romero, A. Shenai, I. Geles, E. Aljalbout, and D. Scaramuzza, “Dream to fly: Model-based reinforcement learning for vision-based drone flight,” in IEEE International Conference on Robotics and Automation (ICRA), 2026. [47] F. Nan, H. Ma, Q. Guan, J. Hughes, M. Muehlebach, and M. Hutter, “Efficient model-based reinforcement learning for robot control via online learning,” arXiv preprint arXiv:2510.18518, 2025. [48] Y. LeCun, “A path towards autonomous machine intelligence version 0.9.2,” Open Review, vol. 62, no. 1, 2022, pp. 1–62. [49] R. Balestriero and Y. LeCun, “Lejepa: Provable and scalable self-supervised learning without the heuristics,” arXiv preprint arXiv:2511.08544, 2025. [50] A. Bardes, J. Ponce, and Y. LeCun, “Vicreg: Variance-invariancecovariance regularization for self-supervised learning,” in International Conference on Learning Representations (ICLR), 2022. [51] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. G. Azar, et al., “Bootstrap your own latent: A new approach to self-supervised learning,” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 21271–21284. [52] D. Mellinger and V. Kumar, “Minimum snap trajectory generation and control for quadrotors,” in IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 2520–2525. [53] Y. Song, S. Naji, E. Kaufmann, A. Loquercio, and D. Scaramuzza, “Flightmare: A flexible quadrotor simulator,” in Conference on Robot Learning, 2021, pp. 1147–1157. [54] J. Eschmann, D. Albani, and G. Loianno, “Raptor: A foundation policy for quadrotor control,” Science Robotics, vol. 11, no. 114, p. eaec1481, 2026. [55] L. Meier, D. Honegger, and M. Pollefeys, “Px4: A node-based multithreaded open source robotics framework for deeply embedded platforms,” in 2015 IEEE International Conference on Robotics and Automation (ICRA), 2015, pp. 6235–6240. [56] N. Wiedemann, V. Wüest, A. Loquercio, M. Müller, D. Floreano, and D. Scaramuzza, “Training efficient controllers via analytic policy gradient,” in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 1349–1356. [57] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager, “Temporal convolutional networks for action segmentation and detection,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 156–165. [58] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, “Gated feedback recurrent neural networks,” in Proceedings of the 32nd International Conference on Machine Learning, vol. 37, 2015, pp. 2067–2075.
18
[59] O. J. Hénaff, R. L. T. Goris, and E. P. Simoncelli, “Perceptual straightening of natural videos,” Nature Neuroscience, vol. 22, no. 6, 2019, pp. 984–991. [60] Y. Wang, O. Bounou, G. Zhou, R. Balestriero, T. G. Rudner, Y. LeCun, and M. Ren, “Temporal straightening for latent planning,” arXiv preprint arXiv:2603.12231, 2026. [61] C. E. Shannon, “A mathematical theory of communication,” The Bell System Technical Journal, vol. 27, no. 3, pp. 379–423, 1948. [62] T. M. Cover and J. A. Thomas, Elements of Information Theory. Wiley, 1999. [63] S. P. Lloyd, “Least squares quantization in pcm,” IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 129–137, 1982. [64] J. MacQueen, “Some methods for classification and analysis of multivariate observations,” in Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1967, pp. 281–297.