Mind Dreamer: Untethering Imagination via Active Latent Intervention on Latent Manifolds Shaojun Xu 1 Xiaoling Zhou 2 Yihan Lin 3 Yapeng Meng 1 4 Xinglong Ji 1 Luping Shi 1 Rong Zhao 1
Abstract
arXiv:2605.16030v1 [cs.LG] 15 May 2026
Model-Based Reinforcement Learning (MBRL) leverages latent imagination for sample efficiency, yet remains constrained by Historical Tethering: imagination is typically initialized from observed states. This creates a learning asymmetry, where the world model’s manifold discovery outpaces the policy’s sparse-reward optimization. We propose Mind Dreamer (MD), a framework that operationalizes Active Latent Intervention (ALI) to transcend Markovian continuity. MD reformulates discovery as the minimization of a global Relay Manifold Expected Free Energy (R-EFE); by sampling initial states from a learned generator s0 ∼ pgen (·) rather than the historical buffer, MD utilizes an adversarial generator to synthesize non-continuous latent jumps to epistemic blind spots that are physically plausible yet cognitively challenging. To resolve the credit assignment paradox across these spatial ruptures, we derive the Relay Value Function (RVF) and Relay Uncertainty Function (RUF). These potentials treat synthesized anchors as counterfactual intermediary states, propagating pragmatic and epistemic value through a principled Bellman-style formulation. Notably, we prove that uncertainty propagation across discontinuities necessitates a quadratic discount γ 2 , establishing a formal epistemic horizon. Theoretically, MD approximates a varianceminimizing importance sampler that expands the manifold’s spectral gap, reducing the hitting time to critical bottleneck states. Empirically, MD achieves a 1.67× average speedup over DreamerV3 on DeepMind Control Suite, reaching 8.8× in sparse-reward tasks.
Figure 1. Untethering Imagination via Active Latent Intervention. Unlike standard MBRL (blue) which is tethered to historical observations (s ∼ D), Mind Dreamer enables proactive latent intervention beyond historical support. An adversarial generator G performs non-continuous latent jumps via generated intervential states (orange) to synthesize counterfactual anchors (red). These anchors bridge OOD regions, enabling goal-directed imagination on the learned manifold.
1. Introduction Model-Based Reinforcement Learning (MBRL) has emerged as a cornerstone of sample-efficient AI, predicated on the power of latent imagination—the ability to simulate future outcomes within a learned world model (Ha & Schmidhuber, 2018; Sutton, 1991; Hafner et al., 2025). This paradigm is conceptually anchored by the Manifold Hypothesis: while sensory observations are high-dimensional, they reside on a low-dimensional manifold M governed by the environment’s underlying physics (Bengio et al., 2013). Under this view, MBRL is a dual-process of geometric discovery: learning the tangent space of M (dynamics) and regressing a value functional over its topology.
1
Center for Brain-Inspired Computing Research, Department of Precision Instrument, Tsinghua University, Beijing, China 2 College of Computer Science and Technology,Zhejiang University, Hangzhou, China 3 Pen-Tung Sah Institute of Micro-Nano Science and Technology, Xiamen University, Xiamen, China 4 Primevision Technology, Shanghai, China. Correspondence to: Rong Zhao <r [email protected]>.
However, a fundamental bottleneck remains: Historical Tethering. While modern world models rapidly recover the global structure of M via dense self-supervised objectives (LeCun, 2022; Bardes et al., 2021), imagination remains a “prisoner of history”—typically initialized only
Preprint. May 18, 2026.
1
Submission and Formatting Instructions for ICML 2026
from observed states in a replay buffer (Ecoffet et al., 2019; Dabney et al., 2021). This creates a critical Learning Asymmetry: the agent’s internal atlas of the environment’s physics (the manifold) expands far more rapidly than its policy can optimize for sparse rewards (Baker et al., 2022; Sekar et al., 2020). Consequently, imagination is confined to previously traversed regions, forcing the agent into a costly random walk when encountering out-of-distribution (OOD) regions, even if its world model already possesses the structural knowledge to bridge them. Prior attempts to mitigate exploration bottlenecks have largely relied on intrinsic motivation (Sekar et al., 2020; Houthooft et al., 2016) or goal-relabeling (Andrychowicz et al., 2017). While these methods incentivize the discovery of novel regions, they remain trajectory-bound. In this work, we operationalize this asymmetry via Active Latent Intervention (ALI). We re-envision the world model as an adversarial generator capable of targeted latent intervention rather than a passive simulator By sampling initial imagination states from a learned generator s0 ∼ pgen (·) rather than the historical buffer s0 ∼ D, ALI transcends the constraints of Markovian continuity, enabling the agent to perform noncontinuous “latent jumps” to synthesized anchors at the frontiers of its knowledge.
Figure 2. The Mind Dreamer Paradigm. Unlike standard MBRL tethered to historical observations, MD enables proactive discovery via: (1) Adversarial Synthesis, where a generator Gθ uses generated initial states to create counterfactual anchors s′ at manifold frontiers. (2) Relay Guidance, which leverages the Relay Value (VRV F ) and Uncertainty (VRU F ) functions to assign potential across spatial ruptures. This untethers imagination from past trajectories, targeting physically plausible yet epistemically rich regions for global manifold repair.
We introduce Mind Dreamer (MD), a framework that transforms MBRL from a passive replay mechanism into an information-theoretic stress test. MD utilizes an adversarial state generator G that lifts the principle of Expected Free Energy (EFE) (Friston, 2009; Tschantz et al., 2020) from a local policy-selection metric to a global discovery functional. This allows the agent to identify counterfactual intermediary states—regions where pragmatic goal-alignment and epistemic uncertainty are optimally balanced (Friston et al., 2017b). However, evaluating a non-local jump presents a credit assignment paradox: temporal consistency is broken. To resolve this, we derive two recursive functionals: the Relay Value Function (RVF) and Relay Uncertainty Function (RUF). Unlike standard value functions that treat states as destinations, these Relay Potentials treat synthesized anchors as counterfactual intermediary states, providing a Bellman-style formulation to evaluate the multi-step utility of a jump across spatial ruptures.
MD increases the manifold’s conductance, effectively expanding the manifold’s spectral gap, reducing the hitting time to critical states. Our contributions are summarized as follows: A New Paradigm: Active Latent Intervention (ALI). We identify the “Historical Tethering” limitation and introduce the latent generator distribution pgen to enable goal-directed, non-continuous intervention. Bridging Discontinuity: Relay Potential Fields. We derive RVF and RUF as recursive functionals as credit assignment to bridge spatial ruptures, introducing the γ 2 discount for stable uncertainty propagation across “latent jumps”.
Our theoretical analysis demonstrates that Mind Dreamer approximates a variance-minimizing importance sampler for manifold refinement. We prove that minimizing the R-EFE objective is mathematically equivalent to minimizing the variance of the world model’s gradients, effectively performing a ”Manifold Repair” curriculum. Crucially, we show that integrating informational shocks requires a quadratic discount γ 2 , establishing a formal Epistemic Horizon that prevents the generator from chasing distal hallucinations. From a topological perspective, we prove that
Theoretical Optimality and Speedup. We prove that MD performs optimal importance sampling on manifold residuals, yielding a convergence speedup proportional to the expansion of the manifold’s spectral gap. Empirical Validation. On the DeepMind Control Suite, Mind Dreamer achieves a 1.67× average speedup in sample efficiency over DreamerV3, with acceleration reaching up to 8.8× in challenging bottleneck tasks. 2
Submission and Formatting Instructions for ICML 2026
2. Related Works
from rewards, they typically require continuous discovery to build the occupancy map. MD extends this logic via Relay Potential Fields. Unlike Goal-Conditioned RL (Schaul et al., 2015) where targets are terminal sinks, our RVF and RUF reformulate synthesized states as counterfactual intermediary states. This allows the agent to propagate pragmatic and epistemic value across spatial ruptures, identifying high-leverage via-points that connect disjoint regions of the manifold.
The evolution of Model-Based RL has transitioned from simple Dyna-style planning (Sutton, 1991) to complex latent imagination (Ha & Schmidhuber, 2018; Hafner et al., 2023). However, the reliance on historical trajectory support remains a bottleneck. Mind Dreamer sits at the intersection of Active Inference, Counterfactual Reasoning, and Manifold Topology. 2.1. From Passive Curiosity to Active Manifold Repair
3. Methodology: Mind Dreamer
Traditional exploration, such as prediction error (Pathak et al., 2017) or information gain (Houthooft et al., 2016) often relies on intrinsic bonuses to steer the agent’s random walk. While recent active inference works (Tschantz et al., 2020; Mazzaglia et al., 2021; Millidge, 2020; Friston et al., 2017b; Çatal et al., 2020) treat control as inference, unified exploration and exploitation under Free Energy Principle (Friston, 2009), they remain trajectory-bound: Expected Free Energy (EFE) is typically evaluated along continuous Markovian paths. Plan2Explore (Sekar et al., 2020) optimizes for global curiosity but is still constrained by initializing rollouts from the replay buffer. MD transcends this by lifting EFE into a global curriculum for Manifold Repair, using an adversarial generator to synthesize counterfactual anchors in regions where the world model’s geometry is most “brittle”.
The core philosophy of MD is that efficient learning requires the agent to transcend its historical trajectories and proactively explore unmastered regions of the latent manifold M through structured “latent jumps.” 3.1. Problem Formulation: MBRL on Latent Manifolds We reformulate the standard POMDP through the lens of the Manifold Hypothesis (Cayton et al., 2005), by assuming that observations o ∈ X are realizations of an underlying lowdimensional latent manifold M ⊂ Rd , where d ≪ dim(X ), and an encoder eψ : X → M maps observations to latent states s ∈ M. This perspective allows us to decompose MBRL into two distinct learning processes: Learning the Transition Structure (World Model): The world model pψ (st+1 |st , at ) approximates the transition dynamics. By minimizing reconstruction and transition errors, the model learns the local transition structure of the manifold.
2.2. Beyond Trajectory Exploitation: Latent do-Intervention Standard techniques like Hindsight Experience Replay (HER) (Andrychowicz et al., 2017) and Hierarchical RL (Nachum et al., 2018; Li et al., 2020) excel at trajectory exploitation—maximizing the utility of collected data through relabeling or abstraction. However, they remain “prisoners of the buffer”, unable to reason beyond the convex hull of past experiences. Our approach shifts the paradigm toward Manifold Intervention (Pearl, 2009). By sampling from a learned intervention distribution pgen , MD decouples environmental physics from historical policy bias. Unlike Go-Explore (Ecoffet et al., 2021), which requires explicit environment resets to known states, MD performs mental teleportation to synthesized frontiers, performing intervention-based exploration in unvisited latent regions.
Learning the Value Field (Value Function): The optimal value function Vϕ∗ (s) represents a scalar field over M that quantifies task-optimal potential. 3.2. Active Latent Intervention To transcend the “Historical Tethering” of standard MBRL, we propose ALI. This mechanism shifts the world model’s role from a passive simulator of past experiences to an active adversarial generator for directed manifold discovery. The core of ALI is the operationalization of the Learning Asymmetry: the world model captures the transition structure of the latent manifold M via dense self-supervised signals far more rapidly than the policy resolves sparse rewards. We exploit this by introducing a learned intervention distribution pgen .
2.3. Credit Assignment Across Spatial Ruptures Breaking Markovian continuity presents a significant credit assignment paradox. Standard temporal difference methods like TD(λ) (Sutton, 1988) or Eligibility Traces (Sutton & Barto, 2018; Seijen & Sutton, 2014) fail when trajectories are non-continuous. While Successor Representations (SR) (Dayan, 1993; Machado et al., 2018) decouple dynamics
Definition 3.1. Latent Intervention Jump. Sampling s0 ∼ pgen (·) allows the agent to initialize imagination at any anchor s′ ∈ M, provided s′ satisfies the manifold consistency Lmf (see Sec. 3.5). This decouples discovery from historical buffer D while remaining anchored to the Markovian flow st ∼ pψ (·|st−1 , at−1 ) from learned physics of the 3
Submission and Formatting Instructions for ICML 2026
(st → st+1 ), they fail to propagate gradients across noncontinuous spatial ruptures (st → s′t+1 ). We decompose the Eq. 2 into two recursive Relay Potential Fields. These fields act as tractable proxies, allowing the generator G to perform functional gradient ascent on the manifold’s utility landscape.
transition structure. Unlike standard rollouts initialized from the replay buffer s0 ∼ D, ALI allows the agent to sample s0 ∼ pgen (·), where s′ ∈ M is a intervention anchor synthesized by an adversarial generator Gθ (s, ϵ). This allows the imagination to “teleport” across spatial ruptures, investigating regions that are physically consistent with the environment’s laws but have not been mastered by the current policy. The generator G acts as an adversarial stress-tester that identifies regions where the latent manifold’s transition structure T M is under-sampled or where the value function Vϕ exhibits high curvature. Gθ (s, ϵ) operates exclusively during the imagination phase, ensuring that no extra interaction costs are imposed on the environment.
Pragmatic Proxy (RVF): Unlike goal-conditioned RL typically treats s′ as a terminal destination (Andrychowicz et al., 2017; Schaul et al., 2015), Relay Value Function VRVF treats the anchor s′ as a counterfactual intermediary state rather than a terminal destination. It integrates the cost of reaching s′ with the value function Vϕ (s′ ) available thereafter (Formal definition in Appendix C.2): ′
VRV F (s, s ) ≜ max E π
3.3. From Local EFE to Global Relay Discovery To guide the state generator G, we leverage the framework of Active Inference, replacing simple reward maximization with the minimization of Expected Free Energy (EFE) (Friston et al., 2017b). While standard MBRL is reactive, EFE provides a proactive objective that unifies goal-directed behavior (pragmatic value) with curiosity-driven exploration (epistemic value). In App. A.2, we demonstrate that MaxEnt RL is a special case of EFE (Millidge et al., 2020).
I(sτ ; oτ |π) | {z }
Epistemic Value E(s′ )
t
k
′
γ rt + γ Vϕ (s )
(3)
t=0
This ensures G identifies “Pragmatic Gateways”—states that are not necessarily high-reward themselves but are strategically positioned to unlock high-density reward regions. Epistemic Proxy (RUF): The Relay Uncertainty Function VRUF quantifies the informational density of a jump. Crucially, it aggregates expected information gain I using a quadratic discount γ 2 (Derivation in Appendix B.3):
Definition 3.2. The Local Objective. For a synthesized anchor s′ , the local EFE G(s′ ) quantifies its immediate utility (inference details in Appendix A.1): G(π, τ ) = −β
π sk =s′
k−1 X
VRU F (s, s′ ) ≜ E
π sk =s′
k−1 X
γ 2t It+1 + γ 2k Uϕu (s′ )
(4)
t=0
Where Uϕu (s′ ) is the Bellman Uncertainty Function (O’Donoghue et al., 2018a). As detailed in 4.4, the γ 2 term is not a hyperparameter but a requisite for stable variance propagation, ensuring the generator targets regions of genuine epistemic volatility while remaining robust to distal hallucinations where model variance compounds uncontrollably.
−η Eq(oτ ) [ln p(oτ )] (1) {z } | Pragmatic ValueP(s′ )
where P(s′ ) aligns with the agent’s goal-prior p(o|C) (Levine, 2018) and E(s′ ) measures the Mutual Information I, identifying regions where the world model’s tangent space is ill-defined (Houthooft et al., 2016).
3.5. The Adversarial Curriculum: Structural Constraints and Co-evolution
However, Point-wise EFE fails for non-local jumps because it ignores the reachability and distal potential beyond the jump. We therefore lift G(s′ ) into the Relay EFE (R-EFE), denoted as Ψ(s, s′ ). "H # X ′ k ′ Ψ(s, s ) = Eq γ G(sk ) s0 = s, s ∈ ξ (2)
By redefining EFE minimization through RVF and RUF maximization, we transform the active inference problem into a standard RL optimization task for the generator Gθ (s, ϵ): h i max E s∼D ηVRV F (s, s′ )+βVRU F (s, s′ )−λLmf (s′ )
k=1
θ
R-EFE represents a path-integral of free energy, evaluating not just the immediate utility of an anchor s′ , but the cumulative potential of all trajectories branching from s′ .
s′ ∼Gθ (s,ϵ)
(5) This creates a adversarial minimax optimization: G proactively identifies the weakest links (high RUF) or shortest paths (high RVF) in the agent’s current atlas. To prevent G from generating latent hallucinations, we constrain it to the manifold learned by the world model pψ (st+1 |st , at ) through structural self-consistency. The manifold constraint Lmf includes:
3.4. Bridging the Gap: Relay Potential Fields Evaluating the R-EFE objective across spatial ruptures presents a significant credit assignment challenge: as standard value functions depend on local temporal consistency 4
Submission and Formatting Instructions for ICML 2026
Dynamics Coherence: Transition entropy H(pψ (·|s′ , a)) is penalized to prevent G from exploiting ill-defined “cracks” in the world model’s dynamics. This effectively defines a “Pessimistic Trust Region” where the model’s predictions remain valid (Kidambi et al., 2020).
Algorithm 1 Mind Dreamer: Active Latent Intervention 1: Initialize: World Model WM (Encoder eψ , RSSM
pψ ), Value Vϕ , Uncertainty Uϕu , Policy πω , Generator Gθ , Relay Function VRV F and VRU F , Buffer D 2: for each training step do 3: 1. World Model Learning: Update {e, p} using D (Standard RSSM loss) 4: 2. Mind Dreamer 5: Sample s0 ∼ D 6: s′ ← Gθ (s0 , ϵ) where ϵ ∼ N (0, I) 7: Sample negative pool sneg ∼ Dbuf f er ∪ Delite 8: Compute EFE Scores: Ψ(s′ ) ← ηVRV F (s′ ) + βVRU F (s′ ) and Ψ(sneg ) 9: Lcontrast ← max(0, m − (Ψ(s′ ) − max Ψ(sneg ))) {InfoNCE Surrogate} 10: Lmf ← H(pψ (·|s′ , a)) + DKL [Enc(Dec(s′ ))∥s′ ] Update θ ← ∇θ (Lcontrast + λLmf ) {Stable Adver11: sarial Optimization} 12: 3. Policy Optimization: Start imagination rollouts from generated zs′ (pos13: terior features of s′ ) 14: Update π, Vϕ and Uϕu via TD-learning 15: Update VRV F and VRU F via k-horizon HER with non-recursive bootstrap target (Eq. 3 and Eq. 4) 16: 4. Environment Interaction 17: Execute πω in real environment, collect data to D 18: end for
Cycle-Consistency: Enforces DKL [Enc(Dec(s′ ))∥s′ ] to anchor jumps within the verifiable reconstruction manifold (Zhu et al., 2017), ensuring that synthesized states remain representable by the World Model. This constraint ensures that Lmf acts as a trust-region boundary for the adversarial generator (Schulman et al., 2015). Generator-Policy Co-evolution. As G shifts the imagination distribution, the policy π must remain stable. We employ quasi-static Target Potential Networks for VRV F and VRU F to ensure non-recursive bootstrap stability (Mnih et al., 2015), preventing the divergence typically seen in off-policy estimation on non-linear manifolds. Furthermore, the World Model pψ is updated at a lower frequency relative to the generator G, providing a more stationary transition landscape for state synthesis (Heusel et al., 2017). While optimizing Eq. 5 implies direct gradient ascent on the potential functions, doing so with neural approximators can lead to value overestimation. In practice, we stabilize the adversarial training by casting Ψ as an energy function within an InfoNCE contrastive objective, forcing the generator to produce states that exhibit strictly higher potential than a dynamic baseline of historical experiences.
TU as:
4. Theoretical Analysis: From Local Actions to Global Manifold Interventions
τs′ −1
(TV V )(s, s′ ) = Eπ [
X
γ t rt + γ τs′ Vϕ (s′ )]
(6)
γ 2t It+1 + γ 2τs′ Uϕu (s′ )]
(7)
t=0
We formally ground Mind Dreamer (MD) by demonstrating that RVF and RUF are not merely heuristics but represent the decomposition of Expected Free Energy (EFE) into recursive functionals, and that “jumping” to high-EFE states is a kind of optimal sampling on the global manifold that accelerates manifold refinement and optimal value function convergence.
τs′ −1
(TU U )(s, s′ ) = Eπ [
X t=0
Theorem 4.2. Path-Integral Equivalence. The fixed ∗ ∗ points VRV F and VRU F uniquely recover the k-step path-integrated constituents of the Expected Free Energy (EFE) (Kappen, 2005).
4.1. Operator Foundations: Mapping EFE to Relay Potentials
Proof Sketch: Under an instrumental prior p(o|C) ∝ exp(V ∗ (s)/λ), maximizing VRV F minimizes cumulative pragmatic risk (see Appendix C.1), while under Gaussian posterior approximations (Millidge et al., 2021) VRU F providing a recursive measure of epistemic leverage across manifold ruptures, maximizing VRU F is equivalent to maximizing the expected reduction in model surprisal (Friston et al., 2015; Houthooft et al., 2016), where informational shocks I propagate with a quadratic discount γ 2 . Lemma 4.3. Contraction and Uniqueness. On a connected latent manifold M, for any s′ ̸= s, the operator TV and TU is a γ and γ 2 contraction mapping under the L∞ norm.
We first establish that RVF and RUF are not merely heuristics but represent the decomposition of Expected Free Energy (EFE) into recursive functionals.
Definition 4.1. Relay Potential Operators. Let s be an anchor and s′ an intervention jump. Given τs′ = inf{t ≥ 0 : st = s′ } as the first hitting time of s′ , we define the Pragmatic Relay Operator TV and Epistemic Relay Operator 5
Submission and Formatting Instructions for ICML 2026
Proof Sketch: Since τs′ ≥ 1 for any non-trivial jump, the discount factor applied to the bootstrap term Vϕ (s′ ) satisfies γ τs′ ≤ γ < 1. This ensures that TV obeys the Banach ∗ Fixed-Point Theorem, admitting a unique fixed point VRV F that represents the maximum potential of all policy flows passing through the gateway s′ . The same applies to TU , guaranteeing a stable global landscape for the state generator. Detailed proof in Appendix C.4 Remark 4.4. The Epistemic Horizon: The γ 2 discount in RUF (Eq. 4) is not an empirical tuning parameter but a necessity of variance propagation. While pragmatic rewards propagate via γ, epistemic informational shocks decay quadratically (O’Donoghue et al., 2018b) (Derivation in Appendix B.3). This quadratic decay ensures that the generator ignores “distal hallucinations” where compounding model variance renders information unverifiable. Remark 4.5. Robustness to Aleatoric Noise. A higherorder discount (γ 3 ) is not required to handle aleatoric noise. The γ 2 discount strictly the variance operator propP Pfollows erty (Var( γ t ϵt ) = γ 2t Var(ϵt )); using γ 3 would shift tracking to the third central moment (skewness), breaking the theoretical equivalence to Expected Free Energy. Moreover, in RSSM-based world models, irreducible stochasticity manifests as high entropy, but local epistemic shock is measured via latent KL-divergence (DKL [q(z|s, o) ∥ p(z|s)]). Once the world model fits inherent stochastic dynamics, the prior p matches the posterior q, collapsing the KL to zero. Thus, VRU F inherently filters out aleatoric noise and triggers only on true epistemic novelty.
(via the Bernstein-von Mises theorem), the RUF potential VRU F is a first-order surrogate for the trace of the Fisher Information Matrix (FIM): E[I] ≈ 12 Tr(F(θ)Σθ ). (Derivation in Appendix B.5). Proof Sketch: By the Bernstein-von Mises Theorem, epistemic gain I aligns with the parameter-space curvature. Maximizing RUF forces G to synthesize anchors s′ at highcurvature regions where observations provide maximal parameter refinement, effectively performing active manifold repair (Amari, 1998). Accelerator II: RVF for Value Convergence: Simultaneously, MD accelerates the value field optimization by transforming value propagation from local diffusion into global targeted updates. Theorem 4.8. Pragmatic Gradient Maximization. Let ∆V (s) = sups′ VRV F (s, s′ ) − Vϕ (s) be the Relay Advantage. The convergence Vϕ → V ∗ is maximized when G identifies anchors s′ that maximize the Variational Bellman Residual ∆V (s) (See Appendix C.1). Proof Sketch: ∆V (s) represents the latent gap between the current belief Vϕ and the optimal k-step potential. Maximizing this gap is equivalent to Prioritized Sweeping (Moore & Atkeson, 1993) on a global scale. By “teleporting” to Residual Peaks—states where the value mismatch is greatest—MD collapses value error across the manifold at an accelerated rate, bypassing the limitations of trajectory-bound sampling. Remark 4.9. The Symbiotic Convergence. The generator G acts as a dual-objective optimizer: by maximizing RUF, it performs Manifold Repair (reducing RM ), and by maximizing RVF, it performs Policy Alignment (reducing ∆V ). This ensures that the imagination process is not just ’dreaming’ of novel states, but specifically targeting states that collapse the joint error of the world model and the value field.
4.2. Variance Reduction: The Dual Accelerators We formalize the Generator G as an active importance sampler that minimizes the variance of stochastic gradients for both the world model and the value function. Theorem 4.6. Variance-Reducing Proposal. Let J (θ, ϕ) be the joint manifold error. The variance of the gradient estimators ĝθ and ĝϕ is minimized if G samples from a distribution q ∗ such that q ∗ (s) ∝ ρ(s)∥∇G(s)∥2 . In the MD framework, the R-EFE potential Ψ serves as a tractable path-integrated proxy for this gradient magnitude (Alain et al., 2016). (Proof in Appendix B.8).
4.3. Topological Acceleration: Breaking the Conductance Bottleneck
Proof Sketch: Following (Kahn & Marshall, 1953), importance sampling variance is minimized when the proposal density is proportional to the integrand’s magnitude. By maximizing Ψ, G aligns qG with q ∗ , concentrating imagination on “high-leverage” regions where gradients are steepest.
To quantify the efficiency gain of MD, we compare its convergence rate against trajectory-constrained agents (e.g., DreamerV3). Trajectory-based exploration is fundamentally bounded by the conductance Φ of the latent manifold M. In environments with OOD regions or bottleneck states, Φ → 0, leading to an exponential hitting time τ ≈ Φ−2 for distal discovery (Jerrum & Sinclair, 1989).
Accelerator I: RUF for Manifold Repair: While standard MBRL passively updates the world model via observed transitions, MD actively repairs the latent manifold by identifying structural gaps. Proposition 4.7. RUF as a Fisher Information Catalyst. Under a Gaussian world model pψ , in the asymptotic limit
Proposition 4.10. Conductance Expansion and Speedup. Under a discrete abstraction of the latent manifold (detailed in Appendix D.4), the latent intervention s0 ∼ pgen in MD induces a synthetic expansion of the manifold’s spectral gap. The acceleration ratio ν relative to trajectory-sampling scales with the χ2 -divergence between the optimal EFE pro6
Submission and Formatting Instructions for ICML 2026
posal q ∗ and the transition-constrained distribution qtraj : τtraj ν= ≈ 1 + χ2 (q ∗ ∥qtraj ) ∝ Φ−2 τM D This provides theoretical intuition for how MD reduces the hitting time to the discovery frontier from O(poly(Φ−1 )) to O(log |M|) by establishing non-local counterfactual intermediary states. (Full derivation in Appendix D.4 and D.5). Remark 4.11. The OOD Region Paradox. Standard MBRL suffers from an “exponential wall” when qtraj → 0 in sparse-reward regimes (Osband et al., 2016). By “teleporting” the imagination directly to high-EFE regions, MD collapses the χ2 divergence, transforming exploration from a random walk into targeted manifold intervention (See Case Study in Appendix D.4).
Figure 3. Visualization of Sampling Dynamics on the Synthetic Three-Ring Manifold. We compare the distribution of imagined states between DreamerV3 and Mind Dreamer across training snapshots. DreamerV3 exhibits Historical Tethering, its sampling distribution is strictly coupled with historical occupancy, hard to escape the ring attractor. Mind Dreamer demonstrates Active Manifold Refinement, the sampling distribution rapidly aggregates at the topological junctions between rings. This illustrates MD’s ability to proactively bridge manifold bottlenecks, enabling the agent to transcend the limits of the replay buffer and linearize the discovery of global task potential.
4.4. Manifold Anchoring: Stability and Safety Jumps A primary challenge in adversarial MBRL is preventing the generator from exploiting “manifold cracks”—highfrequency artifacts where the world model is inaccurate. We establish the safety margin for such non-local jumps. Theorem 4.12. Hallucination Error Bound. Let δ = ∥s′ − ProjM (s′ )∥ represent the manifold deviation of a jump, regularized by Lmf . Under the assumption of LLipschitz continuity of the value field, the value estimation error ϵV is bounded by: ϵV = ∥VRV F (s′ ) − V ∗ (s′ )∥ ≤
5. Experiments Our experiments aim to answer four key questions: (1) Does MindDreamer (MD) improve sample efficiency and final performance by untethering imagination? (2) Can MD identify and fill “epistemic blind spots” on the latent manifold? (3) Does our manifold safeguarding effectively bound the risk of latent hallucinations?
L·δ 1 − γn
where the quadratic discount γ 2 in RUF (Eq. 4) ensures that distal epistemic errors decay faster than pragmatic rewards, shielding the policy from recursive hallucination (Proof in Appendix D.9).
5.1. Experimental Setup and Baselines We emphasize that Mind Dreamer (MD) is a modular framework compatible with various MBRL architectures. In this study, we implement MD upon the state-of-the-art DreamerV3 (Hafner et al., 2023) to demonstrate its potential, use identical RSSM backbones and interaction budgets to ensure fairness. We evaluate against: (1) DreamerV3 as the primary baseline for trajectory-tethered imagination; (2) DreamerV2 (Hafner et al., 2020b) to isolate the gains from backbone advancements versus our ALI mechanism; and (3) Plan2Explore (Sekar et al., 2020) for curiosity-driven exploration comparison.
Proof Sketch: The estimation error δ propagates through the Bellman recursion. By minimizing Lcycle and Ldyn , MD functions as an implicit spectral filter on the latent Jacobian ∇s pψ , effectively anchoring adversarial jumps to the physically verifiable support of M. The use of γ 2 for uncertainty propagation ensures a finite Epistemic Horizon, preventing the generator from chasing compounding model variance (Asadi et al., 2018). Remark 4.13. While strict L-Lipschitz continuity is a strong assumption for neural value functions globally, MD implicitly controls the local Lipschitz constant through target network soft-updates and the pessimistic trust region defined by Lmf . This practical regularization ensures the theoretical error bound remains meaningful during adversarial training. Corollary 4.14. Gradient Stability. Under the Manifold Anchoring bound, the variance of the policy gradient ∇ω J(πω ) remains L∇V · δ stable, precluding the catastrophic collapses typical of unconstrained adversarial dreaming (Details in Appendix D.11).
5.2. Visualizing ALI: The Three-Ring Manifold Case Study To demystify the “Latent Jump” mechanism, we design a synthetic Three-Ring Manifold environment where the underlying geometry M is known but topologically challenging. The agent resides on three coplanar coaxial rings, with the top view of the field as observation. Each ring acts as a Local Attractor (simulating a local optimum), to transi7
Submission and Formatting Instructions for ICML 2026
Figure 4. Evaluation on DeepMind Control (DMC) Benchmarks. (Top) Performance Comparison: Mind Dreamer (Gold) demonstrates superior sample efficiency and higher asymptotic performance against state-of-the-art baselines: DreamerV3 (Green Dashed), DreamerV2 (Blue Dashed), and Plan2Explore (Cyan Dashed). Our method significantly accelerates convergence in bottleneck tasks like Pendulum Swingup and achieves higher final returns in sparse environments like Hopper Hop, at the same sample budget and with marginal additional inference cost. (Bottom) Ablation Study: We analyze the role of specific components by comparing the full model (Gold) with variants removing the Pragmatic Relay VRV F (Solid Green), Epistemic Relay VRU F (Solid Blue), and Manifold Loss Lmf (Solid Red). The significant performance drops in the ablated variants validate the necessity of the proposed Active Latent Intervention framework. Shaded regions indicate the standard deviation over 5 random seeds.
tion to the higher reward ring, the agent must move against the gravitational pull toward the manifold boundary—a feat nearly impossible for random walk exploration.
ing “linearizing” imagination via latent jumps. On average, MD achieves 90% peak performance in 334.7k steps, delivering a 1.67× speedup over DreamerV3 (557.6k) (see Tab. 2 and Tab. 3 for details). In bottleneck tasks like Pendulum Swingup, MD accelerates convergence by an order of magnitude (>8.8×).
As shown in Fig. 3, standard DreamerV3 trajectories remain trapped within the first ring, as its imagination is initialized from the replay buffer which is dominated by high-density attractor data. In contrast, MD’s generator G proactively identifies transition gateways at the rings’ boundaries where VRU F (uncertainty) is high. By sampling s0 ∼ pgen , MD teleports imagination to the boundary ruptures, allowing the policy to discover the transition to the next ring without requiring thousands of random physical steps, thus achieving nearly a 4.2× acceleration for first hitting time over DreamerV3. This confirms that MD converts a Global Discovery problem into a Local Refinement task.
Breaking the Exploration Plateau (Asymptotic Performance). Tab. 1 confirms that MD overcomes “Historical Tethering,” achieving an average return of 831.1 against DreamerV3 (780.3). This performance gain is most prominent in sparse-reward tasks, specifically, MD improves Hopper Hop by +59.8% and Quadruped Run by +30.3%, where Relay Potential Fields successfully bridge disjoint manifold segments, guiding the policy to discover critical transitions that trajectory-bound baselines fail to encounter.
5.3. Performance on DMC Vision Benchmarks
5.4. Ablation: The Role of Relay Potentials and Hallucination Control
We evaluate MD on 20 tasks from the DeepMind Control Suite with raw pixel observations. MD demonstrates a significant lead in learning speed. Fig. 4 presents part of training curves.
We isolate the contributions of the Pragmatic (VRV F ) and Epistemic (VRU F ) relay functions, as shown in Fig 4. The Necessity of Manifold Grounding (Lmf ). Crucially, removing the manifold constraint Lmf leads to a catastrophic performance drop, with the agent failing to surpass
Accelerating Manifold Coverage (Sample Efficiency). MD fundamentally alters sample complexity by untether8
Submission and Formatting Instructions for ICML 2026
the DreamerV3 baseline. As shown in our Self-Consistency Error: E∥s′ − Dec(Enc(s′ ))∥, Lmf achieves a 43.5-fold reduction in jump consistency error compared to standard Markovian transitions.
ric discovery in physical agents. MD represents a critical step toward agents that don’t just dream of where they have been, but actively illuminate where they need to be—anchoring knowledge at the frontiers of the unknown, and synthesizing a world model shaped by proactive discovery rather than passive history.
Pragmatic vs. Epistemic Synergy. VRU F (Epistemic) drives exploration, while VRV F (Pragmatic) enforces taskorientation. Removing VRU F limits the agent to known regions despite efficient local propagation; removing VRV F sustains exploration entropy but sacrifices reward convergence due to a diminished Relay Advantage.
Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
6. Conclusion and Discussion In this work, we introduced Mind Dreamer (MD), a novel MBRL framework that untethers latent imagination from historical trajectories. By operationalizing Active Latent Intervention (ALI) through an adversarial state generator, MD synthesizes counterfactual anchors to proactively bridge topological gaps in the learned manifold. To resolve the credit assignment paradox across spatial ruptures, we derived the Relay Value Function (RVF) and Relay Uncertainty Function (RUF), formally establishing a quadratic discount (γ 2 ) for stable uncertainty propagation. Theoretically, MD acts as a variance-minimizing importance sampler that accelerates manifold repair. Empirically, it achieves state-ofthe-art sample efficiency on DMC benchmarks, particularly excelling in sparse-reward tasks.
References Alain, G., Lamb, A., Sankar, C., Courville, A., and Bengio, Y. Variance reduction in SGD by distributed importance sampling. In International Conference on Learning Representations (ICLR), 2016. Amari, S.-I. Natural gradient works efficiently in learning. Neural Computation, 10(2):251–276, 1998. Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel, O., and Zaremba, W. Hindsight experience replay. Advances in Neural Information Processing Systems, 30, 2017.
Limitations. While MD significantly accelerates exploration and breaks historical tethering, it introduces two primary limitations:
Asadi, K., Misra, D., and Littman, M. Lipschitz continuity in model-based reinforcement learning. In International Conference on Machine Learning, pp. 264–273. PMLR, 2018.
Computational Overhead: The adversarial optimization of the generator G and the computation of the InfoNCE surrogate introduce additional computational overhead during the imagination phase. In scenarios where environmental interaction is extremely cheap but compute is a strict bottleneck, this sample-to-compute trade-off may be less favorable.
Baker, B., Akkaya, I., Zhokov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J. Video pretraining (vpt): Learning to act by watching unlabeled online videos. Advances in Neural Information Processing Systems, 35:24639–24654, 2022.
Entanglement of Uncertainty and Evolving Mechanisms: Our current epistemic proxy effectively leverages the latent KL-divergence to detect structural gaps under a static transition assumption. However, in non-stationary environments where causal mechanisms evolve continuously (e.g., gradual friction changes or payload shifts), the current world model may entangle this continuous mechanism shift with aleatoric noise or pure epistemic ignorance (Fan et al., 2026), potentially causing the generator to target changing physics as spurious “blind spots”.
Bardes, A., Ponce, J., and LeCun, Y. Vicreg: Varianceinvariance-covariance regularization for self-supervised learning. arXiv preprint arXiv:2105.04906, 2021. Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pp. 41–48, 2009.
Future Work Future research will focus on decouple state uncertainty from dynamic shifts. Additionally, scaling the ALI framework to complex, partially observable 3D environments (e.g., embodied navigation and real-world robotics) presents an exciting avenue for advancing proactive geomet-
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35 (8):1798–1828, 2013. 9
Submission and Formatting Instructions for ICML 2026
Botvinick, M. and Toussaint, M. Planning as inference. Trends in Cognitive Sciences, 16(10):485–488, 2012. Foundational work on Control as Inference.
Friston, K. J., Lin, M., Frith, C. D., Pezzulo, G., Hobson, J. A., and Ondobaka, S. Active inference, curiosity and insight. Neural Computation, 29(10):2633–2683, 2017b.
Buckley, C. L., Kim, C. S., McGregor, S., and Seth, A. K. The free energy principle for action and perception: A mathematical review. Journal of Mathematical Psychology, 81:55–79, 2017.
Goyal, A., Islam, R., Strouse, D., Ahmed, Z., Botvinick, M., Larochelle, H., Levine, S., and Bengio, Y. Infobot: Transfer and exploration via the information bottleneck. In International Conference on Learning Representations, 2019.
Çatal, O., Wauthier, S., Verbelen, T., De Boom, C., and Dhoedt, B. Deep active inference for autonomous robot navigation. arXiv preprint arXiv:2003.03220, 2020.
Ha, D. and Schmidhuber, J. World models. arXiv preprint arXiv:1803.10122, 2(3), 2018. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actorcritic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861–1870. PMLR, 2018.
Cayton, L. et al. Algorithms for manifold learning. Univ. of California at San Diego Tech. Rep, 12(1-17):1, 2005. Dabney, W., Barreto, A., Rowland, M., Dadashi, R., Quan, J., Bellemare, M. G., and Silver, D. The valueimprovement path: Towards better representations for reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7160– 7168, 2021.
Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020a. Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mastering atari with discrete world models. arXiv preprint arXiv:2010.02193, 2020b.
Dayan, P. Improving generalization for temporal difference learning: The successor representation. Neural Computation, 5(4):613–624, 1993.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse domains through world models. In International Conference on Learning Representations, 2023.
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. Go-explore: a new approach for hardexploration problems. arXiv preprint arXiv:1901.10995, 2019.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering diverse control tasks through world models. Nature, 640 (8059):647–653, 2025.
Ecoffet, A., Huizinga, J., Lehman, J., Stanley, K. O., and Clune, J. First return, then explore. Nature, 590(7847): 580–586, 2021.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, volume 30, 2017.
Fan, S., Zhang, K., and Cheng, L. Trace: Trajectory recovery for continuous mechanism evolution in causal representation learning. arXiv preprint arXiv:2601.21135, 2026.
Houthooft, R., Chen, X., Duan, Y., Schulman, J., De Turck, F., and Abbeel, P. Vime: Variational information maximizing exploration. In Advances in neural information processing systems, volume 29, 2016.
Friston, K. J. The free-energy principle: a rough guide to the brain? Trends in Cognitive Sciences, 13(7):293–301, 2009. Friston, K. J. The free-energy principle: a unified brain theory? Nature Reviews Neuroscience, 11(2):127–138, 2010.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, volume 32, 2019.
Friston, K. J., Rigoli, F., Ognibene, D., Mathys, C., FitzGerald, T., and Pezzulo, G. Active inference and epistemic value. Cognitive Neuroscience, 6(4):187–214, 2015.
Jerrum, M. and Sinclair, A. Approximating the permanent. SIAM journal on computing, 18(6):1149–1178, 1989. Kahn, H. and Marshall, A. W. Methods of reducing sample size in monte carlo computations. Journal of the Operations Research Society of America, 1(5):263–278, 1953.
Friston, K. J., FitzGerald, T., Rigoli, F., Schwartenbeck, P., and Pezzulo, G. Active inference: A process theory. Neural Computation, 29(1):1–49, 2017a. 10
Submission and Formatting Instructions for ICML 2026
Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267–274, 2002.
McGovern, A. and Barto, A. G. Automatic discovery of subgoals in reinforcement learning using diverse density. In International Conference on Machine Learning, pp. 361–368, 2001.
Kappen, H. J. Path integrals and symmetry breaking for optimal control theory. Journal of statistical mechanics: theory and experiment, 2005(11):P11011, 2005.
Millidge, B. Deep active inference as variational policy gradients. Journal of Mathematical Psychology, 96:102348, 2020.
Kearns, M. and Singh, S. Near-optimal reinforcement learning in polynomial time. In Machine Learning, volume 49, pp. 209–232. Springer, 2002.
Millidge, B., Tschantz, A., and Buckley, C. L. On the relationship between active inference and control as inference. arXiv preprint arXiv:2006.12964, 2020.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. In Advances in Neural Information Processing Systems, volume 33, pp. 21810–21823, 2020.
Millidge, B., Tschantz, A., and Buckley, C. L. Whence the expected free energy? Neural Computation, 33(2): 447–482, 2021. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. Nature, 518(7540): 529–533, 2015.
Kingma, D. P. and Welling, M. Auto-encoding variational bayes. In International Conference on Learning Representations (ICLR), 2014. LeCun, Y. A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27. Open Review, 62(1):1–62, 2022.
Moore, A. W. and Atkeson, C. G. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13:103–130, 1993.
Levine, S. Reinforcement learning and control as probabilistic inference: tutorial and review. arXiv preprint arXiv:1805.00909, 2018.
Nachum, O., Gu, S. S., Lee, H., and Levine, S. Data-efficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
Li, A., Florensa, C., Clavera, I., and Abbeel, P. Sub-policy adaptation for hierarchical reinforcement learning. In International Conference on Learning Representations, 2020.
O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp. 3836–3845. PMLR, 2018a.
Luo, Y., Xu, H., Li, Y., Tian, Y., Darrell, T., and Ma, T. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In International Conference on Learning Representations, 2018.
O’Donoghue, B., Osband, I., Munos, R., and Mnih, V. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp. 3836–3845. PMLR, 2018b.
Machado, M. C., Rosenbaum, C., Guo, X., Liu, M., Tesauro, G., and Campbell, M. Eigenoption discovery through the deep successor representation. In International Conference on Learning Representations, 2018.
Osband, I., Blundell, C., Pritzel, A., and Van Roy, B. Deep exploration via randomized value functions. In Journal of Machine Learning Research, volume 17, pp. 1–31, 2016.
MacKay, D. J. Information-based objective functions for active data selection. Neural Computation, 4(4):590–604, 1992.
Parr, T., Pezzulo, G., and Friston, K. J. Active inference: The free energy principle in mind, brain, and behavior. MIT Press, 2022.
Mannor, S., Simester, D., Sun, P., and Tsitsiklis, J. N. Mean and variance of the certain-equivalence estimate. IEEE Transactions on Automatic Control, 56(2):468–473, 2011.
Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 16–17, 2017.
Mazzaglia, P., Verbelen, T., and Dhoedt, B. Contrastive active inference. Advances in Neural Information Processing Systems, 34:13870–13882, 2021.
Pearl, J. Causality. Cambridge University Press, 2009. 11
Submission and Formatting Instructions for ICML 2026
Puterman, M. L. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 8, pp. 1433–1438, 2008.
Schaul, T., Horgan, D., Gregor, K., and Silver, D. Universal value function approximators. In International conference on machine learning, pp. 1312–1320. PMLR, 2015. Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In International Conference on Learning Representations, 2016. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International Conference on Machine Learning, pp. 1889–1897, 2015. Seijen, H. and Sutton, R. True online td (lambda). In International Conference on Machine Learning, pp. 692– 700. PMLR, 2014. Sekar, R., Rybkin, O., Daniilidis, K., Abbeel, P., Hafner, D., and Pathak, D. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pp. 8583–8592. PMLR, 2020. Sobel, M. J. The variance of discounted markov decision processes. Journal of Applied Probability, 19(4):794–802, 1982. Sutton, R. S. Learning to predict by the methods of temporal differences. Machine learning, 3(1):9–44, 1988. Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. ACM SIGART Bulletin, 2(4): 160–163, 1991. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. MIT press, second edition, 2018. Talvitie, E. Model regularization for stable sample rollouts. In Proceedings of the 30th Conference on Uncertainty in Artificial Intelligence, pp. 780–789, 2014. Thrun, S. and Schwartz, A. Issues in using function approximation for reinforcement learning. In Proceedings of the 1993 Connectionist Models Summer School, pp. 255–263, 1993. Tschantz, A., Millidge, B., Seth, A. K., and Buckley, C. L. Reinforcement learning through active inference. In International Conference on Learning Representations, 2020. Van der Vaart, A. W. Asymptotic statistics. Cambridge Series in Statistical and Probabilistic Mathematics, 3, 2000. Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223–2232, 2017. 12
Submission and Formatting Instructions for ICML 2026
A. Appendix A: Detailed Derivation of EFE-RL Equivalence A.1. EFE Decomposition into Epistemic and Pragmatic Values In this section, we provide a step-by-step derivation of the Expected Free Energy (EFE), denoted as G(π, τ ), decomposing it into Epistemic Value (Information Gain) and Pragmatic Value (Extrinsic Value). Lemma A.1. The Expected Free Energy G(π) for a policy π at a future time τ is typically defined as follows, accroding to (Friston et al., 2015; 2017a): G(π, τ ) = Eq(oτ ,sτ |π) [ln q(sτ |π) − ln p(sτ , oτ )] we can decompose G into: G(π, τ ) = − I(sτ ; oτ |π) − Eq(oτ ) [ln p(oτ )] | {z } | {z } Epistemic Value
(8)
Pragmatic Value
Proof. Starting from the definition of EFE and applying the identity p(sτ , oτ ) = p(sτ |oτ )p(oτ ), we expand the denominator within the logarithm: G(π, τ ) = Eq(oτ ,sτ |π) [ln q(sτ |π) − ln p(sτ , oτ )] = Eq(oτ ,sτ |π) [ln q(sτ |π) − (ln p(sτ |oτ ) + ln p(oτ ))] = Eq(oτ ,sτ |π) [ln q(sτ |π) − ln p(sτ |oτ ) − ln p(oτ )] Utilizing the linearity of expectation, we partition the expression into two distinct components: q(sτ |π) G(π, τ ) = Eq(oτ ,sτ |π) ln − Eq(oτ |π) [ln p(oτ )] p(sτ |oτ ) {z } | | {z } Term 2 Term 1
Term 1: Epistemic Value (Mutual Information) In Active Inference, we assume the agent’s inference is amortized or ideal, such that the true posterior p(sτ |oτ ) is approximated by the variational posterior q(sτ |oτ , π).By definition, the Mutual Information between states and observations under a policy π is: q(sτ |oτ , π) I(sτ ; oτ |π) = Eq(oτ ,sτ |π) ln q(sτ |π) Observing Term 1: Term 1 = Eq(oτ ,sτ |π) [ln q(sτ |π) − ln q(sτ |oτ , π)] = −I(sτ ; oτ |π) Intuition: Mutual information quantifies the reduction in uncertainty regarding state s afforded by observation o. Minimizing G necessitates maximizing this information gain, representing the “Epistemic Value” or exploration drive. Term 2: Pragmatic Value (Extrinsic Value) Term 2 is defined as −Eq(oτ |π) [ln p(oτ )]. Here, p(oτ ) represents the agent’s prior preferences (often denoted as p(oτ |C)), which encodes desired outcomes or goal states.When an observation oτ aligns with these preferences, p(oτ ) is high, thereby minimizing − ln p(oτ ) and, consequently, G. This term constitutes the “Pragmatic Value,” driving the agent toward goal-directed behavior (exploitation). Combining the terms yields the final EFE objective: G(π, τ ) = − I(sτ ; oτ |π) − Eq(oτ ) [ln p(oτ )] | {z } | {z } Epistemic Value
Pragmatic Value
Remark A.2. This formulation illustrates the fundamental trade-off in Active Inference: the agent seeks to minimize EFE by simultaneously maximizing information gain (exploration) and satisfying prior preferences (exploitation) (Parr et al., 2022; Friston et al., 2017a). 13
Submission and Formatting Instructions for ICML 2026
A.2. EFE-MaxEnt RL Equivalence In this section, we provide the formal derivation showing that the Expected Free Energy (EFE) framework, under specific constraints on the prior and temperature, reduces to the standard objective of Maximum Entropy Reinforcement Learning (MaxEnt RL). This equivalence justifies the use of RL-based value functions to approximate the pragmatic components of Active Inference (Millidge et al., 2021). Proof. In Active Inference, the agent’s “goals” are encoded as a prior distribution over observations, p(o). To recover RL, we define this prior as a Boltzmann distribution over a reward function r(s) (Tschantz et al., 2020; Millidge et al., 2020): p(oτ ) =
exp(r(sτ )/α) Z
where α is a temperature parameter representing the “precision” of the agent’s preferences. Substituting this into the pragmatic term: r(sτ ) − ln p(oτ ) = − + ln Z α In the limit where the world model becomes deterministic and perfectly known (i.e., we ignore the epistemic/ambiguity term I(sτ ; oτ |π) → 0 ), the EFE reduces to (Millidge et al., 2021): r(sτ ) G(π, τ ) ≈ Eq(sτ |π) ln q(sτ |π) − α Summing over the infinite horizon with a discount factor γ: "∞ # ∞ X X r(sτ ) τ τ JEF E = γ G(π, τ ) = Eπ γ ln π(aτ |sτ ) − α τ =0 τ =0 (Note: Under the assumption of state-action consistency, ln q(sτ ) is effectively bounded by the entropy of the policy ln π(aτ |sτ ) (Levine, 2018)). By multiplying the entire objective by −α, we transform the minimization of R-EFE into the maximization of a new objective JRL : "∞ # X τ max Eπ γ (r(sτ ) − α ln π(aτ |sτ )) π
τ =0
This is exactly the Maximum Entropy RL objective originally formulated by Ziebart et al. (2008) and used in modern algorithms like Soft Actor-Critic (SAC) (Haarnoja et al., 2018).
B. Appendix B: Formal Derivation of the Relay Uncertainty Function (RUF) The Relay Uncertainty Function (RUF) serves as the path-integrated estimator of the epistemic component within the Relay Expected Free Energy (R-EFE) framework. Here, we provide the formal derivation transitioning from information-theoretic primitives to a computable Bellman recursion (Parr et al., 2022). B.1. From Expected Free Energy to Epistemic Variance In this section, we establish the formal link between the information-theoretic Epistemic Value (Mutual Information) and the Predictive Variance commonly used in stochastic filtering and uncertainty estimation. In the Active Inference framework, the agent minimizes the Expected Free Energy (EFE), denoted as G(π, τ ). For a future time step τ , the local EFE is defined as: G(π, τ ) = Eq(oτ ,sτ |π) [ln q(sτ |π) − ln p(sτ , oτ )] 14
Submission and Formatting Instructions for ICML 2026
Applying the identity p(sτ , oτ ) = p(sτ |oτ )p(oτ ), we decompose G into: G(π, τ ) = − I(sτ ; oτ |π) − Eq(oτ ) [ln p(oτ )] | {z } | {z } Epistemic Value
Pragmatic Value
Lemma B.1. Epistemic-Variance Equivalence under Gaussian Approximation Under a Gaussian approximation of the latent state posterior, the epistemic term (ambiguity) is locally proportional to the predictive variance σ 2 (Buckley et al., 2017). Proof. Following the definition of Mutual Information, the Epistemic Value can be decomposed into the difference between the marginal and conditional entropies: I(sτ ; oτ |π) = H(sτ |π) − Eq(oτ |π) [H(sτ |oτ , π)] where: H(sτ |π) denotes the Prior Entropy, representing the agent’s uncertainty regarding the hidden state s before receiving an observation. H(sτ |oτ , π) represents the Ambiguity (Conditional Entropy), characterizing the residual uncertainty in s after the observation o is integrated. In the context of Expected Free Energy (EFE) minimization, the agent typically seeks to minimize the expected ambiguity, which facilitates a more precise mapping between observations and states. We assume that the variational posterior over the latent states s follows a multivariate Gaussian distribution, q(s) ∼ N (µ, Σ). The differential entropy for a k-dimensional Gaussian is given by: H(s) =
k 1 (1 + ln(2π)) + ln |Σ| 2 2
In the univariate case (k = 1), this simplifies to: H(s) =
1 ln(2πeσ 2 ) 2
This formulation highlights that under a Laplace or Gaussian approximation, the entropy—and thus the epistemic uncertainty—is a monotonic logarithmic function of the variance σ 2 . To demonstrate that the epistemic term is locally proportional to the predictive variance, we perform a first-order Taylor expansion of the entropy function f (σ 2 ) = 21 ln(2πeσ 2 ) around a baseline variance σ02 .The derivative of the entropy with respect to the variance is: df 1 = dσ 2 2σ 2 Consequently, for small perturbations in the model’s uncertainty, the change in entropy ∆H relates to the change in variance ∆σ 2 as follows: 1 ∆H ≈ ∆σ 2 2σ02
Remark B.2. This confirms that the local epistemic cost is proportional to the model’s predictive variance: Gepistemic ∝ σ 2 . This equivalence justifies the use of variance-based exploration bonuses as a direct proxy for information-theoretic active inference. B.2. Convergence of Uncertainty Bellman Equation via γ 2 Unlike pragmatic rewards which represent scalar utilities, epistemic gains represent the resolution of stochastic shocks. We prove that the path-integration of these shocks requires a quadratic discount to ensure stability (O’Donoghue et al., 2018b). 15
Submission and Formatting Instructions for ICML 2026
Theorem B.3. Quadratic Epistemic Discounting. Let the cumulative epistemic risk U be the discounted sum of future informational shocks ϵt ∼ N (0, σt2 ). The variance of this total risk Var(U) is governed by a quadratic discount factor γ 2 . P∞ Proof. Consider the total discounted epistemic accumulation: U = t=0 γ t ϵt . Under the Markovian assumption, informational shocks P P at different time steps are independent. Utilizing the variance property for independent variables, Var( ai Xi ) = a2i Var(Xi ) (Sobel, 1982), we have: ! ∞ ∞ X X Var(U) = Var γ t ϵt = Var(γ t ϵt ) t=0
t=0
By the scaling property of the variance operator, Var(aX) = a2 Var(X), we derive: VRU F =
∞ X
(γ t )2 Var(ϵt ) =
t=0
∞ X (γ 2 )t σt2 t=0
This demonstrates that the integration of informational variance over time naturally induces a γ 2 contraction coefficient. The tighter horizon induced by γ 2 < γ prevents the generator G from over-optimizing toward distal, high-variance regions of the manifold (hallucinations), consistent with the stability conditions for variance in MDPs (Mannor et al., 2011). B.3. Convergence of Relay Uncertainty Bellman Equation We redefine the RUF as the potential of a state s′ to serve as an informational gateway. We prove that the integration of variance across trajectories traversing s′ necessitates a quadratic discount γ 2 . Theorem B.4. Relay Epistemic Recursion. Let ξ = {s0 , a0 , . . . , sk , . . . } be a trajectory where s0 = s and sk = s′ . The RUF, representing the expected cumulative uncertainty of all paths traversing through s′ , satisfies a Bellman recursion with a quadratic discount γ 2 . Proof. Define the cumulative epistemic risk U as the discounted sum of future informational shocks ϵt ∼ N (0, σt2 ). For a given counterfactual anchor s′ , we consider the expectation over trajectories conditioned on the event sk = s′ : " ! # ∞ X ′ t ′ VRU F (s, s ) ≜ Eπ Var γ ϵt s0 = s, sk = s t=0
Given the independence of informational shocks in a Markovian latent manifold, the variance of the sum is the sum of the variances. Decomposing the series at horizon k: "k−1 ! # ∞ X X ′ t t ′ VRU F (s, s ) = Eπ Var(γ ϵt ) + Var γ ϵt sk = s t=0
t=k
Applying the scaling property Var(aX) = a2 Var(X) and factoring out γ 2k from the second term: ∞ k−1 X X 2 t 2 2 k j VRU F (s, s ) = Eπ,sk =s′ (γ ) σt + (γ ) Var γ ϵk+j j=0 t=0 | {z } ′
Future Potential VU (s′ )
This yields the recursive k-step Bellman form: "k−1 # X VRU F (s, s′ ) = Eπ,sk =s′ (γ 2 )t σ 2 (st+1 |st , at ) + (γ 2 )k VU (s′ ) t=0
16
Submission and Formatting Instructions for ICML 2026
B.4. Physical Interpretation: Gateway to the Global Manifold This formulation proves that VRU F (s, s′ ) is not merely a measure of “how much we learn by reaching s′ ,” but rather “how much uncertainty can be resolved globally by passing through s′ .” • Epistemic Horizon Control: The γ 2 discount (where γ 2 < γ) induces a tighter epistemic horizon. This prevents the generator G from chasing distal hallucinations by forcing it to prioritize proximal gateways that lead to high-leverage, verifiable information, addressing the compounding error problem in model-based rollouts (Janner et al., 2019). • Manifold Structural Repair: By maximizing VRU F , the agent identifies “bottleneck states” in the latent manifold. These are states that may be relatively known but act as necessary anchors to reach high-entropy, unmastered regions of the environment (Friston, 2010; Goyal et al., 2019). B.5. RUF as a Fisher Information Catalyst for Manifold Repair This section establishes the fundamental link between the Relay Uncertainty Function (RUF) and the optimization dynamics of the latent world model. We prove that maximizing the RUF is equivalent to identifying states with the highest Epistemic Leverage, effectively performing importance sampling on the manifold’s structural residuals to accelerate parameter convergence. B.5.1. L OCAL E PISTEMIC VALUE AS THE M ANIFOLD R ESIDUAL Let W denote the true environmental parameters governing the latent manifold M, and let θ be the agent’s current belief (parameters). Under the Active Inference framework, the Epistemic Value E of a transition o = (s, a, s′ ) is the information gain: E(o) = DKL [Q(θ|o) ∥ Q(θ)] To bridge the gap between abstract information gain and computable manifold residuals, we invoke the Bernstein-von Mises Theorem (Van der Vaart, 2000). Theorem B.5. Asymptotic Equivalence of Uncertainty and Residuals. In the limit of large samples, the Epistemic Value E(o) is proportional to the prediction residual RM of the world model. Proof. By the Bernstein-von Mises Theorem, the posterior Q(θ|o) converges to a Gaussian N (θ∗ , F(θ)−1 ), where θ∗ denotes the ground-truth parameter (or the Maximum Likelihood Estimate), F(θ) is the Fisher Information Matrix. Consider the world model’s log-likelihood loss L(θ) = − ln P̂ (s′ |s, a, θ). A second-order Taylor expansion of the logposterior around the MLE θ∗ gives: 1 ln Q(θ|o) ≈ ln Q(θ∗ |o) − (θ − θ∗ )T F(θ∗ )(θ − θ∗ ) 2 The KL divergence (Epistemic Value) can thus be approximated by the Fisher Information Distance (MacKay, 1992): E(o) ≈
1 EQ [∥θ − θ∗ ∥2F (θ) ] 2
The term E(o) represents the expectation regarding the parameter θ under the posterior distribution Q. By expanding ∥θ − θ∗ ∥2F (θ) into the standard form of the Mahalanobis Distance: EQ [∥θ − θ∗ ∥2F (θ) ] = EQ (θ − θ∗ )T F(θ)(θ − θ∗ ) . Since the quadratic form (θ − θ∗ )T F(θ)(θ − θ∗ ) is a scalar (1 × 1 matrix), its value is equal to its trace. Utilizing the cyclic property of the trace operator, Tr(ABC) = Tr(CAB), we can rewrite the expression as: Tr (θ − θ∗ )T F(θ)(θ − θ∗ ) = Tr F(θ)(θ − θ∗ )(θ − θ∗ )T 17
Submission and Formatting Instructions for ICML 2026
Given that both the expectation EQ and the trace Tr are linear operators, they can be commuted. Thus, the expected error can be approximated as: 1 EQ [E] ≈ Tr F(θ) · EQ [(θ − θ∗ )(θ − θ∗ )T ] 2 By the definition of the covariance matrix, Σθ = EQ [(θ − θ∗ )(θ − θ∗ )T ], which characterizes the agent’s current uncertainty regarding the parameter estimates (i.e., the spread of the distribution in the parameter space). Consequently, we arrive at the final approximation: E[E] ≈
1 Tr(F(θ)Σθ ) 2
Since F(θ) is the expected Hessian of the log-likelihood loss L(θ) = − ln P̂ (s′ |s, a, θ), and for manifold-structured data, this Hessian is bounded by the prediction residual RM = ∥T (s, a) − T̂θ (s, a)∥2 , we establish: E(o) ∝ RM (s, a, s′ ) This proves that local epistemic gain directly measures the Manifold Residual—the structural gap in the agent’s knowledge. Remark B.6. Geometric Interpretation. The spectral distribution of the Fisher Information Matrix F(θ) explicitly reflects the manifold curvature. In this context, the generator G identifies the “steepest” regions of the latent manifold—specifically, coordinates where local parameter perturbations ∆θ yield the maximal collapse of epistemic uncertainty (Amari, 1998). B.5.2. G LOBAL S TRUCTURAL R EPAIR VIA RUF I NTEGRATION Having established that local shocks ϵt correspond to residuals RM , we extend this to the path-integrated RUF. Corollary B.7. RUF as Global Residual Potential. The Relay Uncertainty Function VRU F (s, s′ ) represents the expected cumulative manifold residual unlocked by the latent anchor s′ . Proof. As RUF represents the cumulative discounted sum of future variance shocks (epistemic uncertainty) with a discount factor of γ 2 : "∞ # X ′ 2 t 2 ′ VRU F (s, s ) = Eπ (γ ) σ (st+1 |st , at ) sk = s t=0
By leveraging the epistemic value-variance equivalence (i.e., σ 2 ∝ E), we can re-map the RUF into the parameter space: "∞ # X ′ 2 t ′ VRU F (s, s ) ∝ Eπ (γ ) Tr(F(θ)t Σθ,t ) sk = s t=0
In this context, the term Tr(F(θ)Σθ ) physically signifies the constraint intensity exerted on the model parameters θ by sampling at state st .
VRU F (s, s′ ) ∝ Eπ,sk =s′
"k−1 X
# (γ 2 )t RM (st , at ) + (γ 2 )k VU (s′ )
t=0
B.5.3. C ONVERGENCE ACCELERATION : T HE C ATALYST E FFECT Finally, we demonstrate how maximizing this global residual potential translates into faster parameter optimization. Theorem B.8. Optimal Gradient Infusion. Maximizing VRU F is locally equivalent to maximizing the expected parameter update ∥∆θ∥2 of the world model. 18
Submission and Formatting Instructions for ICML 2026
Proof. According to the Cramér-Rao Inequality, the lower bound of the variance for any unbiased estimator θ̂ is determined by the inverse of the Fisher Information: Var(θ̂) ≥ F (θ)−1 When an agent selects a transition to a “Counterfactual Anchor” s′ by maximizing the VRU F objective, it is effectively performing active selection of manifold regions where the trace of the Fisher Information, Tr(F), is maximized. Parametric Convergence Each sample drawn from the peak regions of VRU F provides a Steepest Gradient Infusion upon the parameter manifold. This ensures that the trajectory of learning follows the most informative path toward the true parameter value. Information Volume Maximization Driven by the γ 2 term, the agent prioritizes the rectification of proximal bottleneck states characterized by high Fisher Leverage (Epistemic Leverage). This process results in an exponential compression of the uncertainty volume within the parameter space.
Remark B.9. The Epistemic Curriculum. The relationship can be summarized by the following implication: Z VRU F ↑ =⇒
Tr(F)dt ↑ =⇒ Var(θ̂) ↓ (via Cramér-Rao)
Consequently, VRU F acts as a catalyst for manifold rectification. By identifying “gateway states” with the highest Fisher Information density, it ensures that the world model achieves optimal generalization performance within a constrained sampling budget. The unification of the Manifold Residual and the Fisher Information Catalyst reveals the mathematical role of Mind Dreamer: it transforms the world model from a passive simulator into an Active Curriculum Generator (Bengio et al., 2009). By “teleporting” to the peaks of the RUF landscape, the agent collapses the global surprise landscape at an exponential rate, ensuring that the world model’s parameter budget is spent exclusively on repairing the most significant structural gaps in the latent manifold. B.6. Implementation: RSSM-KL as an Epistemic Proxy The framework is compatible with various Model-Based Reinforcement Learning (MBRL) paradigms. While the estimation of RUF can be instantiated through multiple statistical proxies, in this work, we provide a robust implementation tailored for the Recurrent State-Space Model (RSSM) architecture of DreamerV3 (Hafner et al., 2023). Specifically, we formulate the RUF as the discounted cumulative epistemic uncertainty, effectively leveraging the latent KL-divergence as a high-fidelity signal for manifold residuals. Proposition B.10. Implementation Consistency. The KL-divergence of the latent transition DKL [q(zt |st , ot ) ∥ p(zt |st )] is a second-order approximation of the epistemic variance σ 2 . Proof. For two Gaussian distributions P ∼ N (µp , σp2 ) and Q ∼ N (µq , σq2 ), the divergence is: DKL (Q ∥ P ) = ln
σq2 + (µq − µp )2 σp 1 + − σq 2σp2 2
Assuming the posterior and prior variances are approximately equal (σp ≈ σq ) during stable model training, the divergence simplifies to: 1 (µq − µp )2 DKL ≈ 2σp2 Since the squared residual of the means (µq − µp )2 is the unbiased estimator of the predictive variance σ 2 , we establish that DKL ∝ σ 2 . Thus, accumulating KL-divergence along a k-step imagined trajectory is mathematically equivalent to integrating the epistemic potential field (Hafner et al., 2020a). 19
Submission and Formatting Instructions for ICML 2026
C. Appendix C: Formal Derivation of the Relay Value Function (RVF) The RVF represents the pragmatic component of the Relay Expected Free Energy (R-EFE). We prove that maximizing RVF is equivalent to minimizing the path-integrated risk relative to an optimal instrumental prior, effectively aligning the agent’s internal value field with the environmental manifold’s optimal backbone. C.1. From Pragmatic EFE to Value Field Alignment In the Active Inference framework (Levine, 2018), the Pragmatic Value (or Risk) for a future state sτ is defined as the KL-divergence between the predicted observation distribution q(oτ |sτ ) and the agent’s preferred observations p(o|C). G(π, τ ) = Eq(oτ |sτ ) [ln q(oτ |sτ ) − ln p(oτ |C)] Lemma C.1. Pragmatic-Value Equivalence. Minimizing the pragmatic EFE is locally equivalent to maximizing the alignment between the current value estimate and the optimal value manifold. Proof. To recast the control problem as an inference problem, we define the agent’s preference p(o|C) as a Boltzmann Distribution over the optimal value function V ∗ , representing the “instrumental prior” of a rational agent (Botvinick & Toussaint, 2012): ∗ 1 V (s) p(o|C) = exp Z λ where V ∗ is the true optimal value and λ is the temperature. Substituting this into the pragmatic EFE term: ∗ V (sτ ) − ln Z G = Eq [ln q(oτ |sτ )] − Eq λ In the maximum entropy RL limit (ignoring constants and assuming stable entropy), minimizing G is equivalent to: min Eπ [−V ∗ (sτ )] ⇐⇒ max Eπ [V ∗ (sτ )] π
π
If the agent utilizes an approximator Vϕ , the pragmatic objective effectively minimizes the gap: Z Jpragmatic ≈ q(s) (V ∗ (s) − Vϕ (s)) ds This proves that the pragmatic term forces the agent to both select paths toward high-V ∗ regions and refine Vϕ to align with the true manifold backbone. C.2. The Relay Value Function as a Path-Constrained Potential To enable non-continuous manifold refinement, we extend the local pragmatic value to the Relay Value Function (RVF). Definition C.2. Path-Constrained RVF. VRV F (s, s′ ) represents the maximum expected discounted value of all trajectories originating at s that traverse through the counterfactual anchor s′ at step k: "k−1 # X VRV F (s, s′ ) ≜ max E π ′ γ t rt + γ k V ∗ (s′ ) π
sk =s
t=0
Unlike standard value functions that aggregate rewards over all possible futures, VRV F (s, s′ ) acts as a structural filter. It evaluates the “Pragmatic Centrality” of s′ , forcing the generator G to ignore high-reward but physically isolated states, and instead focus on the optimal manifold backbone (via-points) that sustains long-term task progression, akin to the concept of bottleneck states in hierarchical reinforcement learning (McGovern & Barto, 2001). Lemma C.3. Global Value Consistency. For any anchor s′ , VRV F (s, s′ ) ≤ V ∗ (s). The equality VRV F (s, s′ ) = V ∗ (s) holds if and only if s′ lies on an optimal trajectory ξ ∗ originating from s. Proof. Let Tall be the set of all possible trajectories from s, and Ts′ be the subset of trajectories passing through s′ at step k. Since Ts′ ⊆ Tall , the supremum over the subset cannot exceed the supremum over the entire set. If s′ ∈ ξ ∗ , then ξ ∗ ∈ Ts′ , and the maximum is attained. 20
Submission and Formatting Instructions for ICML 2026
C.3. Contraction, Uniqueness, Monotonicity and Convergence Acceleration for TV and TU Theorem C.4. Contraction and Uniqueness. The operators TV and TU are contraction mappings on the Banach space C(M × M) and possess unique fixed points. Proof. The Pragmatic Operator TV Let V and V ′ be two potential functions in C(M × M). For any state pair (s, s′ ), let Pπ denote the probability measure over trajectories induced by policy π starting at s and hitting s′ at the first hitting time τs′ . The difference between applications of the operator is given by: "τ ′ −1 # "τ ′ −1 # s s X X ′ ′ ′ t τ s′ ′ t τ s′ ′ ′ |(TV V )(s, s ) − (TV V )(s, s )| = sup Eπ γ rt + γ V (s ) − sup Eπ γ rt + γ V (s ) π
π
t=0
t=0
Using the property | sup f − sup g| ≤ sup |f − g| and the linearity of expectation: |(TV V )(s, s′ ) − (TV V ′ )(s, s′ )| ≤ sup Eπ [γ τs′ |V (s′ ) − V ′ (s′ )|] π
Since |V (s′ ) − V ′ (s′ )| ≤ ∥V − V ′ ∥∞ , we factor out the norm: |(TV V )(s, s′ ) − (TV V ′ )(s, s′ )| ≤ ∥V − V ′ ∥∞ · sup Eπ [γ τs′ ] π
On a connected manifold M where s ̸= s′ , the first hitting time τs′ ≥ 1 almost surely. For γ ∈ (0, 1), it follows that Eπ [γ τs′ ] ≤ γ < 1. Thus, TV is a κV -contraction with κV = supπ Eπ [γ τs′ ]. The Epistemic Operator TU The derivation for TU follows the variance propagation property. For the bootstrap term U (s′ ), the discount factor is squared (γ 2 ) because U represents a second-order statistic (informational variance): |(TU U )(s, s′ ) − (TU U ′ )(s, s′ )| ≤ sup Eπ γ 2τs′ |U (s′ ) − U ′ (s′ )| π
Applying the L∞ norm:
|(TU U )(s, s′ ) − (TU U ′ )(s, s′ )| ≤ ∥U − U ′ ∥∞ · sup Eπ [γ 2τs′ ] π
Since τs′ ≥ 1, we have κU = supπ Eπ [γ pragmatic operator.
2τs′
2
2
] ≤ γ < 1. As γ < γ, the epistemic operator contracts strictly faster than the
Monotonicity Both operators must be monotonic to ensure stable convergence. For TV , if V1 ≤ V2 pointwise, then for any policy π: "τ −1 # "τ −1 # X X t τ ′ t τ ′ Eπ γ rt + γ V1 (s ) ≤ Eπ γ rt + γ V2 (s ) t=0
t=0
Taking the supremum over π preserves the inequality: TV V1 ≤ TV V2 . The same logic applies to TU , given that the informational shocks I are non-negative.By the Banach Fixed-Point Theorem, since C(M × M) is a complete metric space and TV , TU are contractions, there exist unique fixed points V ∗ and U ∗ . Remark C.5. Epistemic Horizon This exponential contraction proves that MD reduces approximation errors at a significantly faster rate than standard 1-step TD (Puterman, 2014), means TV and TU effectively bypassing the slow “diffusion” of reward signals through the Markov chain and directly anchoring the value field to the most promising regions of the latent manifold. Specifically, because κU ≤ γ 2 < γ = κV (O’Donoghue et al., 2018a), the Epistemic Potential U (representing the uncertainty landscape) converges at an accelerated rate compared to the Pragmatic Potential V . This separation of scales ensures that the Generator G is guided by a stable, converged uncertainty manifold even in the early stages of training before the task-specific reward signals have fully propagated. 21
Submission and Formatting Instructions for ICML 2026
Theoretical Convergence Rate Comparison
Bellman Error (Log Scale)
100 10 5 10 10 10 15 10 20 10 25
Standard TD ( = 0.99) Mind Dreaming ( 15 0.86) Convergence Threshold
10 30 0
100
200 300 Number of Updates
400
500
Figure 5. In the ideal case, MD has an order-of-magnitude advantage over Standard TD in terms of speed of error reduction.
C.4. Global Optimality via latent gateways Theorem C.6. Global Consistency of Relay Potentials. Let V ∗ be the fixed point of the optimal Bellman operator T ∗ . Then, the maximization of the Pragmatic Relay Potential over the latent manifold recovers the optimal value function: ∗ ′ V ∗ (s) = sup VRV F (s, s )
(9)
s′ ∈M
Furthermore, the Generator’s objective J (G) = maxs′ (VRV F − V ) is zero if and only if the current value function Vϕ has converged to V ∗ globally. Proof. Direction ≥: By the Bellman Optimality Principle, for any state s, there exists an optimal trajectory ξ ∗ = P ′ ∗ ∗ ∗ ′ ∗ {s, s1 , s2 , . . . }. For any s ∈ ξ , V (s) = E[ r + γ t V ∗ (s′ )]. By definition of VRV F , we have VRV F (s, s ) = V (s) for ′ ∗ any s on the optimal path. Thus, sups′ VRV F ≥ V (s). ∗ ′ ′ ∗ Direction ≤: For any s′ , VRV F (s, s ) is the value of a specific constrained policy (must pass through s ). Since V is the ∗ ∗ ′ ′ supremum over all possible policies, V (s) ≥ VRV F (s, s ) for all s . ∗ ′ Equivalence: Combining both, V ∗ (s) = sups′ VRV F (s, s ). ∗ ′ ∗ Generator Convergence: If Vϕ = V ∗ , then for any s′ , VRV F (s, s ) ≤ V (s) = Vϕ (s), hence VRV F − V ≤ 0. The ′ ∗ maximum is 0 at s lying on optimal trajectories. If Vϕ < V , there exists an s′ (a “gateway” to a better region) such that VRV F − V > 0, providing a non-zero functional gradient for G.
C.5. RVF as a Catalyst for Value Convergence Theorem C.7. Let the value fitting error be L(ϕ) = 21 Es∼ρ [(V ∗ (s) − Vϕ (s))2 ]. The generator G maximizing ∆V = VRV F (s, s′ ) − Vϕ (s) asymptotically produces a proposal distribution qG that minimizes the variance of the parameter updates ∇ϕ L. Proof. Gradient Variance and Optimal Sampling: Consider the stochastic gradient update for ϕ: g(ϕ) = Es∼ρ [(V ∗ (s) − Vϕ (s)) ∇ϕ Vϕ (s)] | {z } ϵV (s)
and the Variance of g(ϕ) " Varq (ĝ) = Eq
ρ(s) ϵV (s)∇ϕ Vϕ (s) q(s) 22
2
# − ∥∇ϕ L∥2
Submission and Formatting Instructions for ICML 2026
Using an importance sampling proposal q, the variance of the gradient estimator ĝq is minimized (Alain et al., 2016) when (Cauchy-Schwarz): q ∗ (s) ∝ ρ(s) · |ϵV (s)| · ∥∇ϕ Vϕ (s)∥2 (10) Assuming the network sensitivity ∥∇ϕ Vϕ (s)∥2 is locally bounded and smooth on the compact manifold M, the optimal sampling density is primarily driven by the Bellman Residual |ϵV (s)|. ∗ RVF as a Multi-step Bellman Residual: By Definition 4.1, the fixed point VRV F satisfies:
∗ ′ VRV F (s, s ) = Eπ
"τ ′ −1 s X
# t
γ rt + γ
τ s′
τ s′
′
′
Vϕ (s )
t=0
The Relay Advantage ∆V can be expanded as: ′
∆V (s, s ) = Eπ
"τ ′ −1 s X
# t
γ rt + γ
Vϕ (s ) − Vϕ (s)
t=0
P For a sufficiently expressive world model, the term E[ γ t rt + γ τ Vϕ (s′ )] provides a lower bound on the optimal value V ∗ (s) via the k-step Bellman consistency. Thus: ∆V (s, s′ ) ≈ V ∗ (s) − Vϕ (s) = ϵV (s) where ϵV (s) is the true residual relative to the optimal manifold backbone. Minimizing Variance via Maximization: The generator objective in Eq. 5 maximizes Es′ ∼G [∆V (s, s′ )]. As the adversarial game between G and Vϕ reaches a Nash equilibrium, the density qG concentrates on states where ∆V is maximal: qG (s) → ϵV (smax ) =⇒ qG ∝ |ϵV (s)| By substituting qG into the variance formulation of 10, we observe that G selectively presents “hard examples” (states with the highest value-mismatch) to the critic. This collapses the χ2 -divergence between the training distribution and the optimal importance sampling distribution q ∗ , leading to a strictly monotonic increase in the value convergence rate. C.6. Amnesic Sensitivity of RVF Proposition C.8. If λ (the manifold regularization) is too small, G might over-fit to high-VRV F hallucinations. The stability of MD is guaranteed when λ ≥ L · diam(M), where L is the Lipschitz constant of the World Model. / M where the model predicts a spurious high value VRV F (sh ). Under L-Lipschitz Proof. Consider a hallucinated state sh ∈ continuity, the maximum “hallucination gain” relative to a true manifold state s∗ is bounded by ∆V ≤ L · ∥sh − s∗ ∥ (Asadi et al., 2018). To ensure G remains anchored to M, the regularization penalty λδ (where δ is the manifold deviation) must satisfy λδ > L · dist(sh , s∗ ). Extending this to the global manifold scale, the stability criterion requires λ to exceed the maximum potential gradient L across the manifold diameter, ensuring the “energy penalty” for hallucination always outweighs the spurious reward gain.
D. Appendix D: Variance Reduction and Manifold Consistency This section formalizes the Mind Dreamer (MD) framework as a Variance Reduction mechanism for stochastic manifold optimization. We prove that the generator G constructs an optimal proposal distribution that minimizes the gradient estimator’s variance, thereby accelerating convergence. D.1. Variance Reduction: R-EFE as a Variance Reducer We consider the minimization of the Relay Expected Free Energy (R-EFE) objective J (θ) = Eρ(s) [G(s; θ)] over the reachable manifold M. The efficiency of learning θ depends on the sampling distribution q(s) induced by the generator G. 23
Submission and Formatting Instructions for ICML 2026
D.1.1. T HE O PTIMIZATION P ERSPECTIVE : G RADIENT VARIANCE R EDUCTION First, we establish the optimal distribution for the parameter update process. Theorem D.1. Optimal Proposal Distribution for Gradient Estimation. The variance of the unbiased gradient estimator ∗ gq = ρ(s) q(s) ∇θ G(s; θ) is minimized if and only if the generator G induces a distribution q such that: q ∗ (s) ∝ ρ(s)∥∇θ G(s; θ)∥2 as Varq [gq ] = Eq [∥gq ∥2 ] − ∥Eq [gq ]∥2 . To minimize the variance, we Proof. The variance of the estimator gq is defined R minimize the second moment term subject to q(s)ds = 1. This is a classical result in importance sampling for stochastic gradient descent (Alain et al., 2016): Z Z ρ(s)2 ∥∇θ G(s)∥2 L(q, λ) = ds + λ q(s)ds − 1 q(s) M M By Euler-Lagrange optimality condition w.r.t. q(s) and setting it to zero: − Normalizing by
√
λ=
R
ρ(s)2 ∥∇θ G(s)∥2 ρ(s)∥∇θ G(s)∥ √ + λ = 0 =⇒ q ∗ (s) = 2 q(s) λ
ρ(s)∥∇θ G(s)∥ds yields the optimal distribution.
D.1.2. T HE S TATISTICAL P ERSPECTIVE : VALUE I MPORTANCE S AMPLING Alternatively, viewed as a Monte Carlo estimation of the global energy landscape, we obtain a Zeroth-Order optimality condition. Theorem D.2. Optimal Proposal Distribution. The variance of the Monte Carlo estimator Jˆq is minimized if and only if the generator G induces a sampling distribution qG∗ (s) ∝ ρ(s)|G(s)|. Proof. The variance of the importance sampling estimator is given by: Z 1 (ρ(s)G(s))2 2 ˆ Varq [Jq ] = ds − Jglobal N s∈M q(s) R To find the optimal q, we minimize the integral term subject to q(s)ds = 1. Using the Cauchy-Schwarz inequality or Lagrange multipliers, the optimal zero-variance estimator is achieved when the proposal is proportional to the magnitude of the function being integrated (Kahn & Marshall, 1953). Thus, q(s) ∝ ρ(s)|G(s)|. By training G to maximize R-EFE, the agent acts as an optimal importance sampler for world model refinement. D.2. Synthesis: The Potential Field as a Dual-Objective Proxy The efficacy of the Mind Dreamer objective, ηVRV F + βVRU F , stems from its ability to bridge the Statistical Perspective ∗ ∗ (qstat ) and the Optimization Perspective (qopt ). Rather than choosing one, MD performs a form of Fisher-Weighted Importance Sampling, where the two potentials play complementary roles in the learning loop (Schaul et al., 2016). ∗ D.2.1. VRU F AS THE P ROXY FOR G RADIENT S ENSITIVITY (qopt )
From the optimization perspective, the ideal generator targets regions with the highest parameter sensitivity ∥∇θ G∥. In our framework, VRU F acts as this proxy. Under the Bernstein-von Mises correspondence (Section A.5), the epistemic uncertainty is tied to the Fisher Information Matrix F(θ). • Mechanism: Regions where VRU F is maximal are high-curvature zones of the latent manifold. These are areas where the model’s parameters θ are most “unstable” and thus provide the steepest gradients for world model refinement. • Role: VRU F ensures Optimization Efficiency by identifying where the model has the most to learn. 24
Submission and Formatting Instructions for ICML 2026 ∗ D.2.2. VRV F AS THE P ROXY FOR S TATISTICAL M AGNITUDE (qstat )
From the statistical perspective, the ideal generator targets regions with the highest objective magnitude |G(s)|. Here, VRV F acts as the proxy. • Mechanism: VRV F represents the discounted path-integral of rewards and pragmatic value. Maximizing VRV F identifies “high-stakes” regions of the environment—states that are critical to the task’s success but potentially under-sampled by the current policy. • Role: VRV F ensures Global Fidelity by anchoring the generator to states with significant physical and pragmatic impact, preventing the agent from wandering into “epistemic noise” that has no task relevance. D.2.3. T HE U NIFIED “F ISHER -P RAGMATIC ” C URRICULUM The combined objective Ψ allows MD to bypass the requirement for second-order derivatives while satisfying both optimality criteria: + βVRU F (s) qG (s) ∝ ηVRV F (s) | {z } | {z } Statistical (Where is the value?)
Optimization (Where is the gradient?)
Remark D.3. Why this hybrid is superior to pure gradient maximization. Directly maximizing ∥∇θ G∥ often leads to “Gradient Vanishing or Explosion” in latent space, where the generator traps the model in numerically unstable but informatively poor regions. By using path-integrated potentials (VRV F , VRU F ), MD prioritizes Structural Reliability over instantaneous noise. This transforms the exploration problem from a local random walk into a Global Curriculum for manifold repair, ensuring that every latent intervention jump is both informatively dense and task-relevant. D.3. Efficiency Gain and Convergence Rate We quantify the speedup ν afforded by MD’s “latent jumps” relative to trajectory-based sampling qtraj , which is constrained by Markovian continuity. Proposition D.4. Speedup and χ2 -Divergence. Let σq2 be the variance of the gradient estimator. The convergence speedup ν of Mind Dreamer over traditional MBRL is: ν=
2 σtraj = 1 + χ2 (q ∗ ∥qtraj ) σq2∗ σ2
Proof. According to stochastic approximation theory, the convergence rate is E[J (θN ) − J ∗ ] = O( Nq ). The speedup ν is the ratio of samples needed to reach error ϵ. Substituting q ∗ into the variance definition: Z (p∥∇G∥)2 σq2 = ds − ∥∇J ∥2 q Normalizing the ratio leads to the second moment of the likelihood ratio w(s) = q ∗ /qtraj . By definition: Z (q ∗ )2 q∗ 2 ) ]= ds = χ2 (q ∗ ∥qtraj ) + 1 Eqtraj [( qtraj qtraj In environments with OOD regions (where qtraj (s) → 0 in high-gradient regions), χ2 → ∞, illustrating that MD collapses the sample complexity from polynomial to logarithmic scales. Lemma D.5. Conductance and Hitting Time. While trajectory-based exploration is bounded by the manifold conductance Φ (with hitting times τ ≈ O(Φ−2 )), Mind Dreamer creates “latent bridges.” This effectively transforms the manifold’s spectral gap, reducing the hitting time to critical states from O(eD ) to O(D), where D is the manifold diameter. D.4. Quantitative Analysis of the Speedup Ratio In this section, we provide a concrete derivation of the efficiency gain ν using a simplified “OOD Region” model. This case study demonstrates how Mind Dreamer (MD) bypasses the topological bottlenecks that constrain traditional trajectory-based RL. 25
Submission and Formatting Instructions for ICML 2026
D.4.1. C ASE S TUDY: T HE OOD R EGION M ODEL Consider a latent manifold M partitioned into two disjoint regions: a Mastered Region A (where the agent has dense experience but zero information gain) and an OOD Region B (a distal region containing high task-relevant rewards or high model uncertainty). Distribution Definitions: Optimal Proposal (q ∗ ): Following Theorem 4.1, the optimal sampling distribution peaks where the Expected Free Energy (EFE) is maximal. Assuming G(s) is localized in B, we define: ( 1 , s∈B ∗ q (s) = |B| 0, s∈A Trajectory Distribution (qtraj ): Traditional agents are constrained by physical transitions. Let ϵ be the probability that a trajectory-based explorer reaches B through local diffusion. For sparse-reward environments, ϵ → 0: ( ϵ , s∈B qtraj (s) = |B| 1−ϵ |A| , s ∈ A Derivation of Speedup ν: By Proposition 4.1, the speedup ratio is ν = 1 + χ2 (q ∗ ∥qtraj ). We expand the Pearson χ2 -divergence as follows: ∗ 2 Z q (s) 2 ∗ χ (q ∥qtraj ) = qtraj (s) − 1 ds qtraj (s) M R Breaking the integral into regions A and B: In Region A: Since q ∗ (s) = 0, the term contributes A qtraj (s)(−1)2 ds = 1−ϵ. ∗ q = 1ϵ . The contribution is: In Region B: The likelihood ratio is qtraj Z
ϵ |B| B
1 −1 ϵ
2
ds = ϵ
2 1 − +1 ϵ2 ϵ
=
1 −2+ϵ ϵ
Summing the terms: χ2 = (1 − ϵ) +
1 1 −2+ϵ = −1 ϵ ϵ
Substituting back into the speedup formula: ν =1+
1 −1 ϵ
=
1 ϵ
Significance: This result implies that in environments with “hard” exploration barriers, traditional MBRL requires O(1/ϵ) more samples to reach the same error bound as Mind Dreamer. MD’s ability to “jump” directly to the manifold’s frontier effectively linearizes the sample complexity relative to the mixing time of the environment, conceptually aligning with the benefits of “Deep Exploration” strategies (Osband et al., 2016). D.4.2. B EYOND THE U PPER B OUND : P RACTICAL C ONSTRAINTS ON S AMPLE E FFICIENCY While Theorem 4.3 establishes a theoretical speedup of ν ≈ 1/ϵ, empirical RL performance is constrained by the following systemic factors: • The Estimator Bias Bottleneck: The generator G relies on learned Relay Potentials (VRV F , VRU F ) to identify high-EFE regions. In early training, overestimation bias can cause G to target “illusory” informational peaks (Thrun & Schwartz, 1993). Consequently, the sampling distribution aligns with noise rather than true manifold frontiers, reducing the effective ν. • The Hallucination-Fidelity Trade-off: To reach the 1/ϵ limit, G must jump into unknown regions. However, if these regions are far out-of-distribution (OOD), the world model may produce “hallucinations”—physically impossible transitions (Talvitie, 2014). While our manifold constraints (Lcycle , Lsig ) mitigate this, they also impose a “safety trust-region” that naturally caps the maximum achievable exploration depth (Janner et al., 2019). 26
Submission and Formatting Instructions for ICML 2026
• Fragmented Policy Gradient: Trajectory-based exploration ensures a continuous gradient signal. In contrast, “latent jumping” provides sparse, high-variance anchors. If the sampling curriculum outpaces the policy’s adaptation rate, the agent may fail to learn the smooth transition logic between these anchors, leading to “Continuity Collapse” during real-world execution. • Sample vs. Compute Complexity: MD transforms environment interactions into latent imagination. In “easy” tasks (large ϵ), the adversarial optimization overhead of the generator may outweigh the savings in sample complexity. The 1/ϵ advantage is a strategic trade-off, most beneficial when environment interaction is significantly more expensive than latent synthesis (Sutton, 1991). D.5. Monotonic Improvement via Counterfactual Anchoring We quantify the impact of training on generated anchors s′ on the agent’s real-world performance, proving that MD accelerates policy improvement while maintaining stability. Theorem D.6. Generalized Policy Improvement. Let πold be the current policy. If the generator G identifies an anchor s′ with high potential Ψ(s′ ) and the policy is updated on s′ , the resulting policy πnew satisfies: J(πnew ) ≥ J(πold ) − O(ϵ) where ϵ denotes the world model’s approximation error. Proof. Applying the Performance Difference Lemma (Kakade & Langford, 2002; Schulman et al., 2015): J(πnew ) − J(πold ) = Es∼dπnew Ea∼πnew (·|s) [Aπold (s, a)] Standard trajectory-based exploration is limited by the mixing time of πold , often failing to sample regions where the advantage Aπold is large. Mind Dreamer bypasses this by utilizing G to proactively sample anchors s′ where model gaps (high VRU F ) or value gradients (high VRV F ) suggest significant untapped advantage. By optimizing π on the distribution dG , the agent performs policy improvement on a support that pre-emptively covers the support of the optimal future occupancy dπ∗ . As long as the simulation error ∥P̂ − P ∥1 ≤ δ is bounded, training on G leads to monotonic improvement (Luo et al., 2018). Remark D.7. Adversarial Equilibrium and Learning Rates. The interaction between the Generator G and the World Model WM constitutes a non-stationary game. To ensure the stability of the policy improvement in Theorem D.6, we employ a Time-Scale Separation strategy: the generator’s update frequency τG is set such that τG > τWM . This ensures that G provides a quasi-static curriculum for the world model, preventing “catastrophic forgetting” where the generator outpaces the model’s ability to repair the identified structural gaps, consistent with Two-Time-Scale Update Rules (TTUR) in adversarial learning (Heusel et al., 2017). D.6. Global Landscape Consistency: The Distributed Estimator We prove that the Mind Dreamer objective, J (G) = ηVRV F + βVRU F , is a mathematically principled proxy for the gradient of the Relay Expected Free Energy (R-EFE). R Theorem D.8. Stochastic Functional Gradient Ascent. Let Jglobal = s∈M ρ(s)G(s)ds be the global EFE objective over the manifold M. The gradient of Jglobal with respect to the generator parameters θ is effectively approximated by maximizing the potential fields VRV F and VRU F at the generated anchors. Proof. The functional gradient of the global objective J with respect to the generator G follows the reparameterization trick (Kingma & Welling, 2014): ∇θ J = Eϵ∼N (0,I) [∇s G(s) |s=Gθ (ϵ) ·∇θ Gθ (ϵ)] By defining VRV F and VRU F as the path-integral representations of the pragmatic and epistemic components of G(s) (see Section B.1), we establish that the combined potential field Ψ(s) = ηVRV F (s) + βVRU F (s) serves as a smooth potential whose local gradient ∇s Ψ(s) provides the directional derivative of the expected free energy at s. Thus, maximizing Ψ at sampled anchors allows G to perform Stochastic Functional Gradient Ascent on the global landscape, identifying regions of maximal free energy reduction without explicit integration over the entire manifold. 27
Submission and Formatting Instructions for ICML 2026
D.7. Robustness to Hallucinations: The Safety Bound To address the “Hallucination Paradox,” we quantify the safety margin provided by the Structural Self-Consistency loss Lcycle and the Lipschitz properties of the latent manifold. Theorem D.9. Hallucination Error Bound. Let δ(s′ ) = ∥s′ − ProjM (s′ )∥ be the distance of a generated anchor from the true manifold M, as measured by the cycle-reconstruction error Lcycle . If the optimal value function V ∗ is L-Lipschitz, the value estimation error at s′ is bounded by:
∥VRV F (s′ ) − V ∗ (s′ )∥ ≤
L · δ(s′ ) 1−γ
Proof. Let strue ∈ M be the closest physically realizable point to s′ . By Lipschitz continuity, |V ∗ (s′ ) − V ∗ (strue )| ≤ L∥s′ − strue ∥ = Lδ. Since s′ is used as a bootstrapping target in the Bellman recursion, the error propagates over the discounted horizon:
Total Error ≤ Lδ + γLδ + γ 2 Lδ + · · · =
Lδ 1−γ
In our architecture, the Lcycle constraint DKL [Enc(Dec(s′ ))∥s′ ] directly minimizes δ, thus constraining the value error.
Remark D.10. This bound mirrors the Simulation Lemma in traditional MBRL (Kearns & Singh, 2002) but generalizes it to non-continuous support dG . The factor (1 − γ)−1 represents the effective horizon over which a hallucinated anchor can corrupt the value field. Corollary D.11. Gradient Stability. Under the conditions of Theorem D.9, the variance of the policy gradient ∇θ J induced by generated anchors is bounded by L∇V · δ, where L∇V is the Lipschitz constant of the value gradient (Asadi et al., 2018).
Significance: This unified analysis proves that as long as the world model maintains structural self-consistency (minimizing δ), the latent jumps are mathematically guaranteed to remain anchored to physical reality. This prevents the “hallucination loops” common in adversarial model-based RL and ensures stable convergence toward the global manifold backbone. 28
Submission and Formatting Instructions for ICML 2026
E. Appendix E: Implementation Details E.1. Detailed Experimental Results
Figure 6. Comparative Evaluation on DeepMind Control (DMC) Benchmarks. We present the full training curves of Mind Dreamer variants—15 Horizon (Gold), 10 Horizon (Red), and 5 Horizon (Purple)—against state-of-the-art baselines: DreamerV3 (Green Dashed), DreamerV2 (Blue Dashed), and Plan2Explore (Cyan Dashed). Mind Dreamer consistently demonstrates superior sample efficiency and higher asymptotic performance across diverse continuous control tasks. Notably, in sparse-reward and bottleneck environments such as Hopper Hop and Quadruped Run, our method (especially the 15 Horizon variant) significantly outperforms trajectory-tethered baselines by employing Active Latent Intervention (ALI) to synthesize goal-directed latent jumps. On average, Mind Dreamer recovers 90% of peak performance with a 1.67× speedup compared to DreamerV3. Shaded regions indicate the standard deviation over 5 random seeds.
29
Submission and Formatting Instructions for ICML 2026
Figure 7. Ablation Study of Mind Dreamer. We evaluate the contribution of the Pragmatic (VRV F ) and Epistemic (VRU F ) relay functions on four representative tasks: Acrobot Swingup, Hopper Hop, Quadruped Run, and Walker Walk. (1) w/o VRU F (Green): Without epistemic guidance, the generator lacks the “curiosity” to bridge manifold discontinuities, leading to stagnation in exploration-heavy tasks (e.g., Hopper Hop). (2) w/o VRV F (Blue): Without pragmatic guidance, the generator produces high-entropy but task-irrelevant latent jumps, resulting in slower convergence despite high exploration. (3) Full Mind Dreamer (Orange): The synergistic combination ensures directed exploration toward high-value, high-uncertainty regions, significantly outperforming the component-ablated variants and the DreamerV3 baseline (Dashed).
Table 1. Performance comparison on 20 DMC Vision tasks. Scores are reported as Mean ± Std. Bold indicates the best performance, and underlined indicates the second-best. N/A denotes that the algorithm failed to run or produced invalid results.
TASK
D REAMER V2
D REAMER V3
M IND D REAMER 10 H ORIZON
M IND D REAMER 15 H ORIZON
M IND D REAMER 5 H ORIZON
P LAN 2E XPLORE
ACROBOT S WINGUP C ARTPOLE BALANCE C ARTPOLE BALANCE S PARSE C ARTPOLE S WINGUP C ARTPOLE S WINGUP S PARSE C HEETAH RUN C UP C ATCH F INGER S PIN F INGER T URN E ASY F INGER T URN H ARD H OPPER H OP H OPPER S TAND P ENDULUM S WINGUP Q UADRUPED RUN Q UADRUPED WALK R EACHER E ASY R EACHER H ARD WALKER RUN WALKER S TAND WALKER WALK
401.2 ± 4.6 975.2 ± 6.5 999.2 ± 0.9 798.9 ± 40.7 743.8 ± 22.3 734.1 ± 74.5 737.2 ± 338.3 495.7 ± 97.5 886.8 ± 32.3 848.3 ± 58.9 245.3 ± 37.2 817.4 ± 28.4 796.6 ± 20.5 474.5 ± 99.0 706.5 ± 85.3 790.7 ± 66.0 567.4 ± 73.9 684.6 ± 88.2 910.3 ± 26.5 906.6 ± 22.2
423.2 ± 0.0 996.0 ± 0.3 999.0 ± 1.3 862.2 ± 9.4 773.9 ± 34.0 866.5 ± 35.7 975.5 ± 8.4 593.7 ± 136.5 977.9 ± 18.3 965.1 ± 16.4 279.1 ± 90.4 926.1 ± 13.1 871.4 ± 49.3 639.4 ± 96.1 827.8 ± 121.8 962.1 ± 25.8 925.7 ± 35.2 691.6 ± 109.5 986.5 ± 8.7 974.6 ± 0.0
426.3 ± 44.9 998.8 ± 0.0 1000.0 ± 0.0 861.1 ± 0.0 795.5 ± 0.0 921.4 ± 0.0 978.7 ± 0.0 802.4 ± 0.0 976.7 ± 0.0 954.8 ± 0.0 401.2 ± 0.0 946.0 ± 0.0 864.6 ± 45.7 821.4 ± 0.0 869.8 ± 0.0 974.0 ± 0.0 788.0 ± 55.4 777.7 ± 0.0 981.8 ± 0.0 963.1 ± 0.0
437.0 ± 0.0 996.9 ± 0.0 1000.0 ± 0.0 876.9 ± 0.0 846.1 ± 0.0 894.1 ± 0.0 971.3 ± 0.0 878.1 ± 0.0 973.4 ± 0.0 962.9 ± 0.0 446.0 ± 67.1 945.8 ± 0.0 880.7 ± 5.8 833.1 ± 49.7 926.0 ± 3.9 978.8 ± 0.0 748.9 ± 19.9 788.5 ± 0.0 969.8 ± 0.0 941.4 ± 17.5
474.2 ± 0.0 998.5 ± 0.0 1000.0 ± 0.0 869.6 ± 0.0 839.6 ± 0.0 913.4 ± 0.0 972.8 ± 0.0 582.3 ± 1.1 967.2 ± 3.5 960.4 ± 0.0 344.3 ± 0.0 940.7 ± 0.0 896.8 ± 0.0 718.0 ± 135.0 939.3 ± 0.0 973.7 ± 0.0 748.0 ± 60.6 793.3 ± 4.3 967.6 ± 0.0 964.6 ± 0.0
298.1 ± 112.4 902.2 ± 80.2 815.4 ± 136.6 826.5 ± 26.6 403.9 ± 19.3 416.7 ± 10.5 953.3 ± 12.0 901.3 ± 44.3 959.1 ± 7.4 959.8 ± 1.7 141.3 ± 51.9 383.7 ± 33.3 797.5 ± 39.3 418.5 ± 22.2 958.3 ± 6.8 972.9 ± 9.5 954.9 ± 4.7 436.1 ± 6.3 958.6 ± 7.3 956.4 ± 4.0
668.3
780.3
820.5
831.1
801.2
720.7
AVG. S CORE
30
Submission and Formatting Instructions for ICML 2026 Table 2. Sample Efficiency: Number of environment steps (in thousands, k) required to reach 90% of DreamerV3’s maximum return. Results are Mean ± Std. Bold indicates the fastest convergence (fewest steps), and underlined is the second fastest. N/A indicates the threshold was not reached.
TASK ACROBOT S WINGUP C ARTPOLE BALANCE C ARTPOLE BALANCE S PARSE C ARTPOLE S WINGUP C ARTPOLE S WINGUP S PARSE C HEETAH RUN C UP C ATCH F INGER S PIN F INGER T URN E ASY F INGER T URN H ARD H OPPER H OP H OPPER S TAND P ENDULUM S WINGUP Q UADRUPED RUN Q UADRUPED WALK R EACHER E ASY R EACHER H ARD WALKER RUN WALKER S TAND WALKER WALK
D REAMER V2
D REAMER V3
M IND D REAMER 10 H ORIZON
M IND D REAMER 15 H ORIZON
M IND D REAMER 5 H ORIZON
P LAN 2E XPLORE
763.5 K 732.9 K 400.0 K 895.2 K 804.5 K N/A N/A N/A 975.8 K N/A N/A N/A 522.6 K N/A N/A N/A N/A 589.8 K 895.2 K 768.4 K
985.4 K 271.9 K 180.2 K 446.3 K 810.5 K 698.2 K 219.7 K 257.0 K 552.6 K 579.9 K 984.1 K 652.0 K 760.7 K 925.0 K 806.6 K 503.7 K 767.2 K 523.3 K 170.6 K 266.5 K
724.9 K 259.1 K 174.4 K 379.4 K 452.6 K 245.7 K 165.0 K 245.0 K 624.9 K 593.2 K 411.6 K 285.3 K 374.6 K 745.7 K 844.9 K 441.5 K N/A 504.9 K 184.9 K 325.6 K
715.6 K 277.4 K 178.5 K 394.5 K 403.0 K 392.6 K 132.5 K 112.1 K 462.7 K 694.9 K 168.3 K 317.8 K 86.1 K 485.0 K 455.0 K 373.1 K N/A 353.9 K 181.3 K 174.8 K
864.9 K 320.7 K 182.1 K 484.5 K 769.8 K 295.0 K 112.3 K 343.3 K 465.0 K 784.9 K 351.7 K 380.4 K 265.8 K 694.9 K 386.3 K 405.0 K N/A 285.0 K 180.0 K 244.3 K
N/A N/A N/A 2110.0 K N/A N/A 2023.3 K 1656.7 K 1433.3 K 1806.7 K 416.7 K N/A 246.7 K N/A 713.3 K 636.7 K 1780.0 K N/A 846.7 K 1000.0 K
Table 3. Sample Efficiency: Number of environment steps (in thousands, k) required to reach 85% of DreamerV3’s maximum return. Results are Mean ± Std. Bold indicates the fastest convergence (fewest steps), and underlined is the second fastest. N/A indicates the threshold was not reached.
TASK ACROBOT S WINGUP C ARTPOLE BALANCE C ARTPOLE BALANCE S PARSE C ARTPOLE S WINGUP C ARTPOLE S WINGUP S PARSE C HEETAH RUN C UP C ATCH F INGER S PIN F INGER T URN E ASY F INGER T URN H ARD H OPPER H OP H OPPER S TAND P ENDULUM S WINGUP Q UADRUPED RUN Q UADRUPED WALK R EACHER E ASY R EACHER H ARD WALKER RUN WALKER S TAND WALKER WALK
D REAMER V2
D REAMER V3
M IND D REAMER 10 H ORIZON
M IND D REAMER 15 H ORIZON
M IND D REAMER 5 H ORIZON
P LAN 2E XPLORE
589.8 K 661.8 K 381.7 K 785.1 K 730.0 K N/A N/A N/A 879.3 K 840.7 K 821.4 K 900.8 K 465.8 K N/A 975.8 K N/A N/A 512.6 K 730.1 K 560.4 K
985.4 K 252.2 K 170.4 K 367.4 K 780.9 K 609.5 K 180.2 K 168.1 K 166.4 K 372.8 K 984.1 K 474.0 K 454.2 K 875.7 K 727.8 K 394.9 K 698.2 K 414.3 K 141.0 K 197.3 K
714.9 K 251.4 K 166.3 K 332.6 K 405.0 K 195.6 K 155.0 K 235.0 K 504.9 K 533.4 K 376.9 K 275.2 K 108.3 K 685.6 K 844.9 K 431.8 K 967.8 K 494.9 K 175.4 K 294.3 K
645.6 K 262.3 K 178.5 K 377.5 K 393.1 K 295.7 K 112.4 K 102.3 K 422.9 K 554.9 K 158.4 K 302.2 K 76.6 K 415.0 K 445.0 K 293.6 K N/A 324.0 K 173.7 K 145.8 K
794.9 K 305.3 K 174.4 K 443.1 K 769.8 K 245.0 K 112.3 K 204.0 K 415.0 K 624.9 K 313.2 K 372.5 K 265.8 K 614.9 K 386.3 K 295.0 K N/A 235.0 K 180.0 K 164.5 K
N/A N/A N/A 1646.7 K N/A N/A 1570.0 K 1503.3 K 1193.3 K 1493.3 K 286.7 K N/A 230.0 K N/A 666.7 K 573.3 K 1516.7 K N/A 656.7 K 706.7 K
31
Submission and Formatting Instructions for ICML 2026 Table 4. Sample Efficiency: Number of environment steps (in thousands, k) required to reach 80% of DreamerV3’s maximum return. Results are Mean ± Std. Bold indicates the fastest convergence (fewest steps), and underlined is the second fastest. N/A indicates the threshold was not reached.
TASK ACROBOT S WINGUP C ARTPOLE BALANCE C ARTPOLE BALANCE S PARSE C ARTPOLE S WINGUP C ARTPOLE S WINGUP S PARSE C HEETAH RUN C UP C ATCH F INGER S PIN F INGER T URN E ASY F INGER T URN H ARD H OPPER H OP H OPPER S TAND P ENDULUM S WINGUP Q UADRUPED RUN Q UADRUPED WALK R EACHER E ASY R EACHER H ARD WALKER RUN WALKER S TAND WALKER WALK
D REAMER V2
D REAMER V3
M IND D REAMER 10 H ORIZON
M IND D REAMER 15 H ORIZON
M IND D REAMER 5 H ORIZON
P LAN 2E XPLORE
531.9 K 608.5 K 363.3 K 693.4 K 692.7 K 821.4 K N/A 399.3 K 763.5 K 782.8 K 647.7 K 825.2 K 446.9 K N/A 937.2 K 749.5 K N/A 435.4 K 638.4 K 503.6 K
985.4 K 222.7 K 160.5 K 308.3 K 751.3 K 520.7 K 170.4 K 128.6 K 77.3 K 333.4 K 915.1 K 414.7 K 454.2 K 875.7 K 708.0 K 385.0 K 698.2 K 394.5 K 131.1 K 187.4 K
694.9 K 243.7 K 158.3 K 317.0 K 395.5 K 165.5 K 145.0 K 205.0 K 465.0 K 473.5 K 376.9 K 265.2 K 98.5 K 675.6 K 774.9 K 412.4 K 877.5 K 375.0 K 165.9 K 224.0 K
525.5 K 254.7 K 171.3 K 326.7 K 393.1 K 269.3 K 105.7 K 102.3 K 412.9 K 544.9 K 153.5 K 263.1 K 76.6 K 375.0 K 335.0 K 283.6 K 917.6 K 324.0 K 135.3 K 125.1 K
704.9 K 289.9 K 166.7 K 377.0 K 760.0 K 185.0 K 104.7 K 134.3 K 415.0 K 624.9 K 284.3 K 341.3 K 265.8 K 554.9 K 358.4 K 205.0 K 964.9 K 235.0 K 180.0 K 164.5 K
3256.7 K 1903.3 K 986.7 K 1273.3 K N/A N/A 1320.0 K 1276.7 K 1096.7 K 1376.7 K 186.7 K N/A 206.7 K N/A 616.7 K 526.7 K 1320.0 K N/A 566.7 K 566.7 K
Table 5. Sample Efficiency: Number of environment steps (in thousands, k) required to reach 75% of DreamerV3’s maximum return. Results are Mean ± Std. Bold indicates the fastest convergence (fewest steps), and underlined is the second fastest. N/A indicates the threshold was not reached.
TASK ACROBOT S WINGUP C ARTPOLE BALANCE C ARTPOLE BALANCE S PARSE C ARTPOLE S WINGUP C ARTPOLE S WINGUP S PARSE C HEETAH RUN C UP C ATCH F INGER S PIN F INGER T URN E ASY F INGER T URN H ARD H OPPER H OP H OPPER S TAND P ENDULUM S WINGUP Q UADRUPED RUN Q UADRUPED WALK R EACHER E ASY R EACHER H ARD WALKER RUN WALKER S TAND WALKER WALK
D REAMER V2
D REAMER V3
M IND D REAMER 10 H ORIZON
M IND D REAMER 15 H ORIZON
M IND D REAMER 5 H ORIZON
P LAN 2E XPLORE
512.6 K 555.2 K 345.0 K 656.8 K 636.9 K 686.3 K 968.5 K 361.7 K 724.9 K 763.5 K 570.5 K 730.6 K 428.0 K N/A 879.3 K 711.7 K N/A 377.5 K 546.7 K 446.9 K
985.4 K 203.1 K 150.7 K 278.8 K 573.6 K 451.7 K 150.7 K 89.1 K 77.3 K 333.4 K 717.9 K 345.5 K 414.7 K 777.1 K 619.3 K 345.5 K 589.7 K 335.0 K 121.3 K 167.6 K
574.9 K 243.7 K 158.3 K 309.2 K 385.9 K 145.4 K 135.0 K 165.0 K 465.0 K 473.5 K 351.0 K 245.2 K 98.5 K 585.5 K 684.9 K 354.2 K 707.0 K 335.0 K 147.0 K 184.9 K
475.4 K 239.6 K 171.3 K 309.8 K 383.1 K 181.2 K 105.7 K 92.6 K 253.8 K 544.9 K 153.5 K 255.2 K 76.6 K 345.0 K 335.0 K 253.8 K 827.4 K 244.3 K 120.0 K 116.8 K
475.0 K 266.8 K 166.7 K 360.5 K 750.2 K 185.0 K 104.7 K 114.5 K 415.0 K 514.9 K 236.1 K 333.4 K 256.0 K 465.0 K 330.5 K 205.0 K 804.9 K 205.0 K 171.7 K 164.5 K
3020.0 K 1003.3 K 896.7 K 1123.3 K N/A N/A 1200.0 K 1163.3 K 1020.0 K 1293.3 K 156.7 K N/A 193.3 K N/A 573.3 K 490.0 K 1196.7 K N/A 510.0 K 490.0 K
32
Submission and Formatting Instructions for ICML 2026 Table 6. Comprehensive list of mathematical symbols and notations.
Symbol
Description
First Appearance
State Spaces and World Model X,o
High-dimensional observation space and raw observation o ∈ X
Sec. 3.1
M, s
Low-dimensional latent manifold and latent state s ∈ M
Sec. 3.1
eψ , pψ
Observation encoder and transition world model parameterized by ψ
Sec. 3.1
at , rt
Action and extrinsic reward at time step t
Sec. 3.1, Eq. 3
Parametric scalar value function field and the optimal value field
Sec. 3.1
Parametric Bellman uncertainty function
Eq. 4
Vϕ , V
∗
Uϕu
Active Latent Intervention (ALI) s′
Synthesized counterfactual anchor (destination of a latent jump)
Def. 3.1
′
pgen
Learned generator distribution for initializing imagination at s
Def. 3.1
Gθ
Adversarial state generator parameterized by θ
Sec. 3.2
ϵ
Standard Gaussian noise vector for the generator ϵ ∼ N (0, I)
Eq. 5, Alg. 1
ξ
An imagined trajectory or rollout path
Eq. 2
Expected Free Energy and Objectives G(π, t)
Local Expected Free Energy (EFE) for policy π at step t
′
Def. 3.2, Eq. 1 ′
Ψ(s, s )
Relay Expected Free Energy (R-EFE) functional bridging s to s
Eq. 2
P, E
Pragmatic Value (goal-alignment) and Epistemic Value (info-gain)
Eq. 1
I
Mutual Information quantifying epistemic uncertainty reduction
Eq. 1, Eq. 4
VRV F
Relay Value Function (Pragmatic Proxy)
Eq. 3
VRU F
Relay Uncertainty Function (Epistemic Proxy)
Eq. 4
Lmf
Manifold structural consistency constraint (Cycle/Dynamics coherence)
Eq. 5
β, η
Weighting coefficients for Epistemic (RUF) and Pragmatic (RVF) potentials
Eq. 1, Eq. 5
γ
Pragmatic temporal discount factor
Eq. 3
Quadratic epistemic discount factor establishing the Epistemic Horizon
Eq. 4
γ
2
Theoretical Analysis and Operators τs′ TV , TU qG , q
∗
F(θ) ′
First hitting time to the synthesized anchor s′
Def. 4.1
Pragmatic and Epistemic Relay Operators
Def. 4.1
Sampling distribution induced by G and optimal proposal distribution
Thm. 4.4
Fisher Information Matrix of the world model parameters θ
Prop. 4.5
∆V (s, s )
Variational Bellman Residual (Relay Advantage); sups′ ∆V (s, s′ )
ϵV (s)
Value residual V ∗ (s) − Vϕ (s) (Bellman error)
Thm. 4.6
Φ
Conductance of the latent manifold M
Thm. 4.8
Convergence speedup ratio relative to trajectory-bound sampling
Thm. 4.8
Hallucination error / Euclidean deviation from the true manifold M
Thm. 4.9
ν ′
δ(s ) L
Lipschitz constant of the optimal value field V
33
∗
∆V (s)
=
Thm. 4.6
Thm. 4.9
Submission and Formatting Instructions for ICML 2026 Table 7. Hyperparameters for Mind Dreamer and DreamerV3 Baseline. We retain the default parameters of DreamerV3 to ensure a fair comparison, while introducing MD-specific parameters for the generator and relay potentials.
C ATEGORY
H YPERPARAMETER
VALUE
DreamerV3 Base (DMC Vision) E NVIRONMENT
ACTION R EPEAT I MAGE S IZE
2 64 × 64
W ORLD M ODEL
RSSM D ETERMINISTIC / S TOCHASTIC D ISCRETE L ATENT C ATEGORIES E NCODER /D ECODER CNN D EPTH MLP L AYERS / U NITS KL F REE NATS / DYN S CALE / R EP S CALE
512 / 32 32 32 5 / 1024 1.0 / 0.5 / 0.1
B EHAVIOR
ACTOR / C RITIC L AYERS I MAGINATION H ORIZON (H ) D ISCOUNT FACTOR (γ) / L AMBDA (λ) ACTOR E NTROPY C OEFFICIENT
2 15 0.997 / 0.95 3 × 10−4
O PTIMIZATION
BATCH S IZE × L ENGTH M ODEL / ACTOR / C RITIC L EARNING R ATE G RADIENT C LIPPING
16 × 64 1 × 10−4 / 3 × 10−5 / 3 × 10−5 1000.0
Mind Dreamer Specific (Ours) G ENERATOR G
MLP L AYERS (E NC / DYN ) I NFO NCE T EMPERATURE I NFO NCE S CALE C YCLE C ONSISTENCY S CALE (λmf ) DYNAMICS C ONSISTENCY S CALE
2/2 0.1 1.0 0.1 0.01
I NFO NCE W EIGHTS
R EAL B UFFER N EGATIVES (wreal ) G ENERATED N EGATIVES (wg ) C ROSS -BATCH N EGATIVES (wcross ) E LITE P OOL N EGATIVES (welite )
1.0 0.4 0.2 1.2
R ELAY P OTENTIALS (RVF / RUF)
k-H ORIZON EMA TARGET U PDATE F REQ . / F RACTION P RAGMATIC (SVF) W EIGHT (wv ) E PISTEMIC (SUF) W EIGHT (wu )
34
15 100 / 0.05 1.0 1.0