Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control

Observation-Grounded Self-Predictive Reinforcement Learning for Visual Continuous Control Xinwei Liu1 , Junyuan Liang1∗ , Jianting Zhang2 , Wuhui Chen1 1

Sun Yat-sen University 2 Purdue University liuxw73@mail2.sysu.edu.cn, liangjy53@mail2.sysu.edu.cn, zhan4674@purdue.edu, chenwuh@mail.sysu.edu.cn

arXiv:2608.05989v1 [cs.LG] 6 Aug 2026

Abstract Sample-efficient policy learning from pixels is a long-standing challenge in reinforcement learning (RL). Recent dynamicsbased representation learning methods have significantly improved the sample efficiency of model-free visual RL by learning dynamics-aware representations through auxiliary prediction performed either in latent space (self-prediction) or observation space (observation prediction). However, state-ofthe-art methods from both categories still struggle on challenging visual control tasks when training data is limited. We posit that relying on either predictive objective alone may be insufficient. Self-prediction encourages latent representations to be temporally predictive over multiple future steps, but does not explicitly require latent transitions to align with observation-level dynamics. In contrast, observation prediction grounds learned representations in observation-level dynamics, but does not directly regularize the temporal predictability of latent representations over extended horizons. In this paper, we propose Observation-Grounded Self-Predictive Representations (OG-SPR), a model-free visual RL algorithm for continuous control that learns representations that are both temporally predictive in latent space and grounded in observation-level dynamics. Building on an actor-critic framework, OG-SPR incorporates two core auxiliary objectives: multi-step latent self-prediction and next-observation prediction. In addition, OG-SPR uses short-term value prediction as an auxiliary task to further stabilize value learning. We empirically show that directly imposing latent self-prediction on the shared representation may over-constrain it and does not necessarily improve performance. To address this issue, OG-SPR introduces two lightweight adapters for latent selfprediction, allowing the shared representation to benefit from temporally predictive signals without being forced to directly satisfy the self-prediction objective. Experiments on 28 visual control tasks from the DeepMind Control Suite show that OG-SPR improves aggregate performance over state-of-theart self-predictive and observation-predictive RL methods, with particularly pronounced gains in challenging domains such as dog and humanoid.

Introduction Learning policies from pixel inputs is an important problem in reinforcement learning (RL). While prior work has shown the potential of mastering visual continuous control by model-free methods, such as DrQ (Yarats, Kostrikov, and Fergus 2021) and DrQ-v2 (Yarats et al. 2022), sample ∗

Corresponding author.

Figure 1: Performance improvements of OG-SPR over stateof-the-art baselines on the DeepMind Control Suite at 500k environment steps. For each task, the baseline score is defined as the stronger result between the recent self-predictive method MR.Q (Fujimoto et al. 2025) and observationpredictive method AnonMethod (Anonymous 2026).

efficiency remains a major challenge. Recently, dynamicsbased representation learning has made significant progress in data-efficient visual RL (Schwarzer et al. 2021, 2023; Fujimoto et al. 2025; Kim, Ha, and Kim 2025; Anonymous 2026). These methods learn dynamics-aware representations for model-free RL algorithms, without using a learned dynamics model for planning or value estimation, as done in model-based RL methods (Hansen, Su, and Wang 2022, 2024; Hafner et al. 2025). Two representative lines of work are self-predictive methods and observation-predictive methods. Self-predictive approaches (Zheng et al. 2023; Scannell

et al. 2024; Fujimoto et al. 2025), inspired by self-supervised learning (Grill et al. 2020), learn representations by predicting latent embeddings of future observations over multiple time steps. In contrast, observation-predictive methods (Gelada et al. 2019; Anonymous 2026) train representations by predicting next observations. Both lines of work have recently achieved performance matching or surpassing state-of-the-art model-based methods, such as DreamerV3 (Hafner et al. 2025) and TD-MPC2 (Hansen, Su, and Wang 2024), on diverse tasks (Fujimoto et al. 2025). This suggests that learning meaningful representations is a key enabler for data-efficient visual RL. Despite this progress, the success of these methods remains uneven across tasks. Strong recent methods from both lines still struggle in challenging visual control domains under limited data budgets, such as 500k environment steps (Fujimoto et al. 2025; Anonymous 2026). We posit that relying on either latent self-prediction or next-observation prediction alone may be insufficient for data-efficient visual continuous control. Self-prediction encourages latent representations to be temporally predictive over multiple future steps, but does not explicitly require latent transitions to align with observation-level dynamics. In contrast, observation prediction grounds learned representations in observation-level dynamics through future observation reconstruction, but does not directly enforce long-horizon temporal predictability in latent space. These two objectives therefore impose complementary inductive biases, motivating the following question: Can a model-free visual RL agent become more dataefficient in continuous control by learning representations that are both temporally predictive in latent space and grounded in observation-level dynamics? In this paper, we provide a positive answer to this question by exploring how to leverage the complementary strengths of latent self-prediction and observation prediction. Specifically, we present Observation-Grounded Self-Predictive Representations (OG-SPR), a model-free RL algorithm that learns representations satisfying both objectives. Building on an off-policy actor-critic framework (Sutton, Barto et al. 1998), OG-SPR learns state-action representations through two dynamics-oriented auxiliary tasks: multi-step latent selfprediction and next-observation prediction. In addition to these dynamics-oriented objectives, OG-SPR also includes short-term value prediction as an auxiliary objective to stabilize value learning (Anonymous 2026). However, as shown in our ablation study, naively combining these auxiliary objectives does not effectively improve performance. We hypothesize that directly imposing the latent self-prediction objective on the shared representation may over-constrain it. To mitigate this issue, OG-SPR introduces two lightweight adapters for latent self-prediction, with one placed after the observation encoder and the other after the state-action encoder. These adapters form an adapter-mediated self-prediction branch, so that the shared representation can receive temporally predictive learning signals without being forced to directly satisfy the self-prediction objective. We evaluate OG-SPR on 28 visual control tasks from the DeepMind Control Suite (DMControl) (Tassa et al. 2018) under limited data budgets. Experimental results show that OG-

SPR achieves better average performance than state-of-theart self-predictive and observation-predictive RL methods (Fujimoto et al. 2025; Anonymous 2026), with particularly strong gains in challenging domains such as dog and humanoid, as shown in Figure 1. OG-SPR represents an initial step toward data-efficient RL for complex visual continuous control tasks by integrating self-predictive and observationpredictive methods. We hope this work can inspire further research on their intersection to improve the data efficiency of model-free visual RL.

Related Work Representation Learning for Data-Efficient RL Representation learning is widely used to improve the sample efficiency of RL. Prior work has explored a wide range of representation learning approaches for RL (Echchahed and Castro 2025). Metric-based methods (Liao, Zhang, and Yu 2023; Zhang et al. 2025) shape the embedding space by enforcing task-relevant similarities among state representations. Dataaugmentation methods (Yarats et al. 2022; Ma et al. 2024) encourage the learned representations to be invariant to irrelevant visual changes. Contrastive learning methods (Eysenbach et al. 2022; Liu, Tang, and Eysenbach 2025) learn representations by contrasting positive pairs against negative pairs. Our work focuses on dynamics-based representation learning, which learns dynamics-aware representations for model-free RL through dynamics prediction (Gelada et al. 2019; Fujimoto et al. 2025).

Dynamics-Based Representation Learning Recent dynamics-based methods have achieved performance competitive with or better than model-based methods, such as DreamerV3 (Hafner et al. 2025) and TD-MPC2 (Hansen, Su, and Wang 2024), on diverse tasks with less algorithmic and computational complexity (Fujimoto et al. 2025; Anonymous 2026). This branch of research can be broadly divided into self-predictive and observation-predictive methods. Self-predictive methods learn representations by predicting latent embeddings of future observations. TD7 (Fujimoto et al. 2023) learns state-action representations in the latent dynamics space for low-dimensional continuous control tasks. Building on TD7, MR.Q (Fujimoto et al. 2025) is a state-of-the-art self-predictive method that augments latent self-prediction with additional predictive objectives, such as reward and termination prediction. Observation-predictive methods also train representations through dynamics modeling, but ground the prediction objective in observation space. OFENet (Ota et al. 2020) is a representative observation-predictive approach for lowdimensional settings. AnonMethod (Anonymous 2026) is a recent observation-predictive method. It addresses a bottleneck of observation-predictive representation learning in low-dimensional settings, where reconstruction losses tend to be dominated by observation dimensions with large value ranges. To mitigate this issue, AnonMethod performs both representation learning and RL in a normalized observation space, making observation-predictive learning applicable across diverse domains. However, existing dynamics-

based methods typically focus on prediction either in latent space or in observation space, and these two directions have largely evolved as separate lines of work. The potential of training representations simultaneously in latent and observation spaces for model-free RL remains underexplored. OG-SPR takes an initial step toward addressing this gap.

Model-Based Reinforcement Learning Model-based methods, such as the Dreamer series (Hafner et al. 2020, 2021, 2025), also learn latent dynamics with observation prediction. Our work differs from these methods in both learning objectives and how the learned dynamics are used. In terms of learning objectives, OG-SPR overlaps with these methods only in observation prediction, while the remaining objectives are distinct. In terms of usage, OG-SPR does not maintain a learned dynamics model for RL training. Instead, it learns representations through dynamics modeling in both latent and observation spaces for model-free RL.

Reinforcement Learning Reinforcement learning (RL) addresses the problem of sequential decision making, usually formulated as a Markov Decision Process (MDP). An MDP can be represented by a tuple (S, A, P, r, γ), where S and A denote the state and action spaces, respectively; P (st+1 |st , at ) denotes the transition probability of the next state st+1 given the current state st and action at ; r : S × A → R is the reward function; γ ∈ [0, 1) is the discount factor. The objective of RL is to learn a policy π : P S → A that maximizes the discounted ∞ cumulative reward t=0 γ t rt . Actor-critic methods (Sutton, Barto et al. 1998) typically learn an action-value function P∞ Qπ (s, a) = Eπ [ t=0 γ t rt |s0 = s, a0 = a] which models the expected return, from an initial state s and action a. In low-dimensional settings, the observation an agent receives at time step t, denoted as ot , is usually treated as the state st (st := ot ). In complex settings (e.g., visual RL), ot does not necessarily satisfy the Markov property and therefore may not be directly treated as st . This induces a partially observable MDP (POMDP). Following common practice (Mnih et al. 2013; Yarats et al. 2022), we approximate the Markov state by stacking three consecutive prior observations into a state st := [ot−2 , ot−1 , ot ].

Methodology As shown in Figure 2, OG-SPR is built on an actor-critic framework with value and policy networks. An image-based state s is first passed through an online encoder f followed by Adapter 1 u1 , producing a low-dimensional latent state representation h. The latent state is then combined with the action a and fed into a state-action encoder g to obtain a state-action representation z: z = g(h, a).

Value Learning The value target is constructed based on TD3 (Fujimoto, van Hoof, and Meger 2018). Specifically, we train two separate long-term value predictors Q1 and Q2 , with identical architectures. Both predictors take the same state-action representations as inputs. To stabilize bootstrapping, we maintain target copies of the networks used for target value estimation. Target networks and their outputs are denoted by adding a prime to their online counterparts (e.g., f ′ and h′ ). Target parameters are not updated via gradient descent, but periodically synchronized with their online counterparts (Fujimoto ′ et al. 2025). The smoothed target action aπ is produced by ′ the target policy network π with clipped Gaussian noise: ′

aπ = clip(a′ , −1, 1), a′ = π ′ (h′ ) + clip(ϵ, −c, c),

Preliminaries

h = u1 (f (s)) ,

representation z is used for long-term value learning and three auxiliary tasks.

(1)

Adapter 1 maps encoder features into the latent state space on which policy learning, value learning, and latent selfprediction are built. Similar to AnonMethod, the policy network takes h with gradients stopped, while the state-action

ϵ ∼ N (0, σ 2 ),

(2)

where h′ = u′1 (f ′ (s)). We use n-step returns for value learning (Yarats et al. 2022). Given a sampled transition sequence τ = (st , at , rt:t+n−1 , st+n ), the value target y is: y=

n−1 X

′ γ i rt+i + γ n min Q′j (zt+n ), j∈{1,2}

i=0

(3)

  ′ ′ zt+n = g ′ h′t+n , aπt+n , ′

where h′t+n = u′1 (f ′ (st+n )), and aπt+n is computed based on st+n via Equation 2. We use the Huber loss instead of mean squared error (MSE) for value learning to eliminate bias from prioritized sampling (Fujimoto, Meger, and Precup 2020; Fujimoto et al. 2025): LValue = Huber (Q1 (zt ), y) + Huber (Q2 (zt ), y) .

(4)

Policy Learning The policy network π is optimized using the deterministic policy gradient (Silver et al. 2014). An L2 regularization term is added to the pre-activation policy outputs ãπ , which helps avoid local minima when rewards and value estimates are sparse (Schrittwieser et al. 2020; Fujimoto et al. 2025): aπ = tanh(ãπ ), 1 X LPolicy = − Qi (z) + λpre-activ ∥ãπ ∥22 , 2

(5)

i∈{1,2}

where z = g (u1 (f (s)), aπ ). aπ is computed from s using the online encoder, Adapter 1, and policy network without policy smoothing noise.

Auxiliary Tasks Latent Self-Prediction. Starting from the current latent state ht , future latent states are predicted recursively by applying the state-action encoder g and Adapter 2 u2 with the corresponding actions from the sampled transition sequence: ĥt+1 = u2 (g(ht , at )), ĥt+i+1 = u2 (g(ĥt+i , at+i )),

i = 1, . . . , K − 1.

(6)

(a) Value network with two long-term value prediction heads and three auxiliary prediction heads.

(b) Policy network with detached inputs. Figure 2: An illustration of the full OG-SPR method. Adapter 2 maps state-action representations back to the latent space defined by Adapter 1. Both adapters use the same output activation function to maintain consistent output spaces. By inserting this adapter between the state-action representation and the latent prediction target, this design prevents the self-prediction objective from being directly imposed on the shared state-action representation, thereby alleviating the constraint on it. The targets for latent self-prediction are computed using target networks (Schwarzer et al. 2021; Fujimoto et al. 2025). Given the target latent embeddings (h′ t+1 , . . . , h′ t+K ) produced by the target encoder f ′ and target Adapter 1 u′1 , the self-prediction loss is computed using the mean squared error (MSE): K   X LSelf-predictive = MSE ĥt+k , h′ t+k . (7) k=1

Next-Observation Prediction. The second auxiliary task predicts the next observation from the state-action representation. Specifically, a decoder Dec takes zt as input and reconstructs the next observation ot+1 using the MSE loss: 2 (8) LRec = ∥Dec(zt ) − ot+1 ∥2 . Short-Term Value Prediction. We adopt short-term value prediction as an auxiliary stabilizing objective, which provides lower-variance training signals than bootstrapped value targets and stabilizes representation learning when value estimates are noisy (Anonymous 2026). A predictor q takes state-action representations z as input and predicts the cumulative discounted reward over the next n steps. The auxiliary

loss is computed as the cross-entropy (CE) loss between the predicted logits and a two-hot encoding of the target: Rt =

n−1 X

γ j rt+j ,

j=0

(9)

Ln-step = CE (q(zt ), TwoHot(Rt )) , where Rt is the non-bootstrapped term in the value target in Equation 3. We use a non-uniform return support for the two-hot encoding, with bin locations generated by the symexp transform, symexp(x) = sign(x)(exp(|x|) − 1) (Hafner et al. 2025). During training, gradients from the policy loss are not propagated to the online encoder and Adapter 1, whereas all components of the value network are jointly optimized using the combined loss L: L = LValue + λRec LRec + λn-step Ln-step +λSelf-predictive LSelf-predictive ,

(10)

where λRec , λn-step and λSelf-predictive denote the auxiliary loss weights for next-observation prediction, short-term value prediction, and latent self-prediction, respectively. The pseudocode of OG-SPR is provided in the appendix.

Experiments In this section, we first compare OG-SPR with baselines in terms of performance and computational efficiency on continuous control tasks. We then conduct ablation studies to

analyze the key components of OG-SPR, followed by a representation analysis to examine how self-prediction and observation prediction affect the learned state and state-action representations. Finally, we provide an additional evaluation on discrete-action domains to examine the behavior of OGSPR beyond continuous-control settings.

Experimental Setup Environments. We primarily evaluate OG-SPR on the DMControl benchmark (Tassa et al. 2018), a collection of continuous control tasks built on the MuJoCo simulator (Todorov, Erez, and Tassa 2012). The maximum episode score for each task is 1000. We consider 28 tasks used in prior work (Fujimoto et al. 2025; Anonymous 2026). Agents are trained for 500k environment steps, equivalent to 1M frames in the original environment under an action repeat of 2. The input state is constructed by stacking three previous frames, which are resized to 84 × 84 pixels in RGB format. Baselines. We compare OG-SPR with representative dataefficient RL methods, including both model-based and model-free approaches: (1) TD-MPC2 (Hansen, Su, and Wang 2024), a strong model-based method for continuous control tasks, (2) DreamerV3 (Hafner et al. 2025), a generalpurpose model-based RL algorithm that performs well across diverse domains, (3) MR.Q (Fujimoto et al. 2025), a stateof-the-art self-predictive method that achieves performance competitive with model-based RL approaches, DreamerV3 and TD-MPC2, (4) AnonMethod (Anonymous 2026), a recent observation-predictive method achieving performance competitive with or better than MR.Q, (5) DrQ-v2 (Yarats et al. 2022), a simple yet strong model-free RL baseline for visual continuous control tasks based on data augmentation. Implementation Details. We follow MR.Q in constructing the actor-critic backbone of OG-SPR. Each adapter is implemented as a linear layer. We apply random shift augmentation to pixel observations (Yarats et al. 2022), but use different augmentation strategies for the observation-predictive and self-predictive objectives. For next-observation prediction, we apply the same sampled spatial shift to both the input observation and the prediction target to preserve spatial alignment. For latent self-prediction, we apply independently sampled shifts to future target observations. The prediction horizon for latent self-prediction is set to 5, following prior selfpredictive methods (Schwarzer et al. 2021; Fujimoto et al. 2025). The auxiliary loss weights are kept fixed across tasks. We adopt the same default settings as AnonMethod for the observation-predictive and short-term value objectives, setting λRec = 0.1 and λn-step = 1.0. To determine λSelf-predictive , we start from the default weight used in SPR (Schwarzer et al. 2021), 2.0, although SPR uses a cosine-similarity loss while OG-SPR uses MSE. We then select λSelf-predictive using a small subset containing only quadruped-run and dog-run, searching over candidate values spaced by 1.0. The resulting value, 5.0, is used for all 28 tasks without per-task tuning. Additional details are provided in the appendix. Evaluation Protocol. All experiments are run for 5 seeds. For each seed, we evaluate the agent every 5k environment

steps over 10 episodes and report the average episode return as the evaluation score. Results for all baselines are obtained by re-running their official implementations, except for DrQv2. For DrQ-v2, we use the results reported in MR.Q. Most prior work reports results on DMControl using raw, unnormalized scores (Yarats, Kostrikov, and Fergus 2021; Yarats et al. 2022; Zheng et al. 2023; Hansen, Su, and Wang 2024; Hafner et al. 2025; Fujimoto et al. 2025). However, aggregate statistics over raw scores can be dominated by relatively easy tasks with high returns, such as cartpolebalance and reacher-easy. This may obscure improvements on more challenging tasks, such as dog-run and humanoidwalk, where raw scores are typically much lower. To provide a complementary view of aggregate performance, we additionally report DrQv2-normalized scores, which measure relative improvement over DrQ-v2, a strong model-free baseline for visual continuous control. The metric is computed in a way similar to the human-normalized score (Wang et al. 2016) commonly used for Atari games (Bellemare et al. 2013): DrQv2-Normalized(x) =

x − random score . DrQ-v2 score − random score

This normalization reduces the influence of tasks where DrQ-v2 already performs well and highlights improvements on tasks where DrQ-v2 remains weak. Benchmark-level aggregate performance is reported using the mean and interquartile mean (IQM) over task-level scores, where each task-level score is averaged across seeds.

Main Results As shown in Table 1, OG-SPR achieves the best aggregate performance on both raw scores and DrQv2-normalized scores. Notably, OG-SPR substantially outperforms both MR.Q and AnonMethod in terms of DrQv2-normalized scores. Its mean DrQ-v2-normalized score is approximately 1.8× that of MR.Q and 1.9× that of AnonMethod. Figure 1 shows the percentage improvement of OG-SPR over the stronger of MR.Q and AnonMethod on each task, computed using raw scores. OG-SPR brings particularly large relative gains in challenging visual control domains, achieving 20.9–63.8% and 61.7–333.7% improvements in the dog and humanoid domains, respectively. Under the default 500kstep budget, all methods obtain relatively low scores in the humanoid domain, making it difficult to fully reveal performance differences in this challenging domain. We therefore additionally evaluate the humanoid tasks with a relaxed training budget of 2M environment steps in Table 1. Under this setting, OG-SPR still achieves substantial gains over all baselines. We also provide aggregate learning curves computed using raw scores and DrQv2-normalized scores in Figures 3. Full per-task results are provided in the appendix. Since OG-SPR incorporates both observation-predictive and self-predictive objectives, it incurs higher training cost than AnonMethod and MR.Q. OG-SPR requires 18.3% more training time than MR.Q, and trains substantially faster than DreamerV3 (see the appendix) when all methods are implemented in PyTorch (Paszke et al. 2019).

Ablation Study We conduct ablation studies for two purposes:

Tasks

DrQ-v2

TD-MPC2

DreamerV3

MR.Q

AnonMethod

OG-SPR (ours)

8 [5, 11] 161 [147, 175] 16 [12, 18] 14 [12, 17] 1 [1, 1] 1 [1, 1] 6 [6, 7] 7 [6, 7] 2 [1, 2] 2 [2, 2]

29 [23, 36] 118 [78, 157] 44 [34, 54] 42 [31, 52] 1 [1, 2] 4 [1, 9] 7 [3, 10] 11 [7, 17] 2 [1, 3] 2 [1, 3]

51 [45, 56] 251 [233, 269] 70 [61, 80] 90 [79, 101] 1 [1, 2] 27 [4, 50] 8 [7, 8] 79 [8, 200] 3 [2, 5] 48 [3, 93]

55 [46, 69] 239 [214, 263] 61 [59, 63] 81 [72, 91] 1 [1, 1] 1 [1, 1] 6 [5, 7] 31 [8, 55] 2 [2, 3] 54 [3, 153]

90 [64, 115] 357 [267, 448] 105 [79, 128] 108 [102, 118] 3 [2, 4] 100 [94, 106] 12 [8, 20] 339 [284, 401] 15 [7, 22] 316 [299, 333]

485 [467, 505] 489 [453, 524]

536 [507, 563] 571 [516, 622]

599 [591, 606] 685 [670, 699]

607 [602, 612] 702 [692, 711]

626 [620, 632] 730 [718, 741]

Raw Scores of Selected Challenging Tasks dog-run dog-stand dog-trot dog-walk humanoid-run humanoid-run (2M) humanoid-stand humanoid-stand (2M) humanoid-walk humanoid-walk (2M)

10 [9, 12] 43 [37, 49] 14 [11, 18] 22 [18, 29] 1 [1, 1] 2 [2, 3] 6 [6, 7] 9 [8, 11] 2 [2, 2] 19 [2, 52]

Aggregate Raw Results of 28 Tasks Mean IQM

510 [497, 523] 545 [519, 564]

Aggregate DrQv2-Normalized Results of 28 Tasks Mean IQM

1.00 1.00

1.35 [1.27, 1.42] 1.94 [1.64, 2.24] 2.84 [2.72, 2.96] 2.65 [2.51, 2.80] 5.05 [4.40, 5.71] 0.97 [0.92, 1.01] 1.21 [1.14, 1.32] 1.29 [1.24, 1.36] 1.25 [1.20, 1.31] 1.61 [1.50, 1.76]

Table 1: Results on selected challenging DMControl tasks and aggregate results over all 28 tasks at default 500k environment steps. 2M denotes results evaluated at 2M environment steps. See the appendix for humanoid-domain results evaluated at 1M and 1.5M environment steps. Brackets denote 95% bootstrap confidence intervals. Bold numbers indicate the best performance.

Methods

Raw Mean IQM

Figure 3: Aggregate learning curves on DMControl, computed as the mean over 28 tasks using raw scores and DrQv2-normalized scores. Shaded area captures a 95% bootstrap confidence interval.

1. To evaluate the effectiveness of learning representations that are both temporally predictive in latent space and grounded in observation-level dynamics. 2. To examine the importance of the adapter design. The aggregate ablation results are summarized in Table 2. We denote by OG-SPR - OP the variant that removes the observation-predictive objective LRec . OG-SPR - SP indicates the variant that removes the self-predictive objective LSelf-predictive . OG-SPR - SVP denotes removing the shortterm value prediction loss Ln-step . Overall, removing any auxiliary objective degrades the performance of OG-SPR. The performance drop is especially pronounced when removing the self-predictive objective. To study the adapter design, we compare OG-SPR with three variants. In Shared Adapter, the two adapters share parameters, reducing the flexibility of the mappings used for recursive self-prediction. In No Adapter, both adapters

DrQv2-Normalized Mean

IQM

OG-SPR (complete) 626

730

5.05

1.61

OG-SPR - OP OG-SPR - SP OG-SPR - SVP

615 608 613

712 704 706

4.29 2.62 4.60

1.63 1.26 1.59

Shared Adapter No Adapter Adapter 2 Only

611 597 603

702 676 691

4.38 3.73 3.54

1.48 1.39 1.41

Table 2: Aggregate ablation results on the DMControl benchmark. The mean and IQM are reported for both raw scores and DrQv2-Normalized scores. Bold numbers indicate the best performance.

are removed, and the self-prediction objective is directly imposed on the shared representation. These two variants progressively reduce the flexibility of the adapter-mediated selfprediction branch. Both Shared Adapter and No Adapter underperform complete OG-SPR, indicating that reducing or removing adapter flexibility hurts performance. We also consider Adapter 2 Only, where Adapter 1 is removed, which is the design used in MR.Q. Compared with the full OG-SPR, Adapter 2 Only performs worse because removing Adapter 1 makes the state encoder output space inconsistent with the output space of Adapter 2, as the former uses an ELU activation whereas the latter uses no activation function. These results support the importance of using two separate adapters with matched output spaces for latent self-prediction.

Analyzing State and State-Action Representations

Metrics

To understand how self-prediction and observation prediction shape representations, we analyze the state and stateaction representations learned by OG-SPR and its two ablated variants, OG-SPR - OP and OG-SPR - SP. We freeze the learned representation networks and train lightweight linear predictors to recover the proprioceptive states corresponding to the visual observations. We consider two prediction tasks. The first predicts the current proprioceptive state from the latent state representation, with the prediction loss denoted by Lstate . The second predicts the next proprioceptive state from the state-action representation, with the loss denoted by Lnext-state . Details and results are provided in the appendix. The two ablated variants exhibit distinct strengths. The observation-predictive variant OG-SPR - SP generally achieves lower Lnext-state than the self-predictive variant OGSPR - OP, suggesting that observation prediction helps state-action representations capture more accurate dynamics information. In contrast, the self-predictive variant tends to achieve lower Lstate , suggesting that self-prediction learns state representations that are more informative for recovering the underlying proprioceptive state. These observations are consistent with the different inductive biases of the two predictive objectives. Multi-step latent self-prediction primarily regularizes the state representation to be temporally predictive over horizons, without explicitly constraining latent transitions to align with observation-level dynamics. In contrast, next-observation prediction learns the state-action representation through next-observation reconstruction, encouraging it to capture observation-grounded dynamics information, while not directly imposing a multi-step predictability constraint on the latent state representation. OG-SPR inherits the characteristics of both objectives. Its Lstate is close to that of OG-SPR - OP, while its Lnext-state often lies between those of OG-SPR - OP and OG-SPR SP. Overall, OG-SPR strikes a balance between extracting useful information from the current pixel input and capturing accurate dynamics information.

Human-Normalized Results

Evaluation on Discrete-Action Tasks Although our main focus is visual continuous control, we also evaluate OG-SPR on Atari100k (Kaiser et al. 2020), a benchmark for data-efficient discrete-action visual RL, to examine how the proposed design behaves when adapted to a discreteaction domain. We adopt a lightweight discrete-action adaptation following MR.Q and AnonMethod, by modifying the output activation of the policy network. The implementation details and hyperparameters are provided in the appendix. We report aggregated human-normalized scores, the standard evaluation metric for the Atari benchmark. Similar to the DrQv2-normalized score, we also consider SPR-normalized scores, where each game score is normalized by the corresponding score of SPR (Schwarzer et al. 2021), a strong selfpredictive baseline for this benchmark. As shown in Table 3, OG-SPR remains competitive on Atari100k, ranking first or second on three of the four aggregate metrics. Together with the main DMControl results, these findings suggest that OGSPR is particularly effective in continuous-control domains while remaining competitive in discrete-action settings.

Mean IQM

SPR 0.66 0.43

MR.Q 0.90 0.40

AnonMethod

OG-SPR

0.93 0.40

0.81 0.42

1.25 1.10

1.24 1.11

SPR-Normalized Results Mean IQM

1.00 1.00

1.12 0.99

Table 3: Aggregate results on Atari100k. We report the mean and IQM under both human-normalized and SPRnormalized scores. Bold and underline indicate the best and second-best results, respectively.

One possible explanation for the less pronounced improvements on Atari100k is that adapting continuous-control algorithms to discrete-action domains may require nontrivial algorithmic modifications for cross-domain adaptation. The Dreamer family provides a representative example. DreamerV2 (Hafner et al. 2021) introduced discrete latent representations when extending Dreamer (Hafner et al. 2020) to Atari, while DreamerV3 (Hafner et al. 2025) further introduced a set of cross-domain stabilization techniques for stable learning across diverse domains. Our Atari experiments instead use a minimal discrete-action adaptation, allowing us to examine whether OG-SPR transfers to discrete-action domains without substantial cross-domain modifications. We leave the development of robust cross-domain mechanisms for model-free visual RL to future work.

Conclusion and Limitations In this paper, we presented Observation-Grounded SelfPredictive Representations (OG-SPR), a model-free visual RL algorithm that improves data efficiency by integrating multi-step latent self-prediction, next-observation prediction, and short-term value prediction within an actor-critic framework. To alleviate the direct constraint imposed by latent selfprediction on the shared representation, OG-SPR introduces two lightweight adapters that form an adapter-mediated selfprediction branch. Experiments on DMControl show that OG-SPR improves sample efficiency and achieves new stateof-the-art aggregate performance on the benchmark, with particularly pronounced gains in challenging domains. Despite the considerable room for improvement in these challenging domains under limited data budgets, OG-SPR provides an initial step toward data-efficient RL for complex visual continuous control tasks. Several limitations remain. Although OG-SPR can be adapted to discrete-action domains, its gains are less consistent than those observed in visual continuous control. Further work is needed to make the method robust across different action spaces. Besides, OG-SPR uses fixed auxiliary loss weights across tasks, which may not be optimal for all tasks. Future work could explore adaptive weighting strategies to better coordinate multiple auxiliary objectives.

References Anonymous. 2026. Title Withheld for Anonymous Review. Citation suppressed to preserve double-blind review. Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res., 47: 253–279. Echchahed, A.; and Castro, P. S. 2025. A Survey of State Representation Learning for Deep Reinforcement Learning. Trans. Mach. Learn. Res., 2025. Eysenbach, B.; Zhang, T.; Levine, S.; and Salakhutdinov, R. 2022. Contrastive Learning as Goal-Conditioned Reinforcement Learning. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. Fujimoto, S.; Chang, W.; Smith, E. J.; Gu, S.; Precup, D.; and Meger, D. 2023. For SALE: State-Action Representation Learning for Deep Reinforcement Learning. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Fujimoto, S.; D’Oro, P.; Zhang, A.; Tian, Y.; and Rabbat, M. 2025. Towards General-Purpose Model-Free Reinforcement Learning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Fujimoto, S.; Meger, D.; and Precup, D. 2020. An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Fujimoto, S.; van Hoof, H.; and Meger, D. 2018. Addressing Function Approximation Error in Actor-Critic Methods. In Dy, J. G.; and Krause, A., eds., Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, 1582–1591. PMLR. Gelada, C.; Kumar, S.; Buckman, J.; Nachum, O.; and Bellemare, M. G. 2019. DeepMDP: Learning Continuous Latent Space Models for Representation Learning. In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, 2170–2179. PMLR. Grill, J.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P. H.; Buchatskaya, E.; Doersch, C.; Pires, B. Á.; Guo, Z.; Azar, M. G.; Piot, B.; Kavukcuoglu, K.; Munos, R.; and Valko, M. 2020. Bootstrap Your Own Latent - A New Approach to Self-Supervised Learning. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Hafner, D.; Lillicrap, T. P.; Ba, J.; and Norouzi, M. 2020. Dream to Control: Learning Behaviors by Latent Imagination. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Hafner, D.; Lillicrap, T. P.; Norouzi, M.; and Ba, J. 2021. Mastering Atari with Discrete World Models. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.

Hafner, D.; Pasukonis, J.; Ba, J.; and Lillicrap, T. 2025. Mastering diverse control tasks through world models. Nature, 1–7. Hansen, N.; Su, H.; and Wang, X. 2022. Temporal Difference Learning for Model Predictive Control. In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvári, C.; Niu, G.; and Sabato, S., eds., International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, 8387–8406. PMLR. Hansen, N.; Su, H.; and Wang, X. 2024. TD-MPC2: Scalable, Robust World Models for Continuous Control. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Kaiser, L.; Babaeizadeh, M.; Milos, P.; Osinski, B.; Campbell, R. H.; Czechowski, K.; Erhan, D.; Finn, C.; Kozakowski, P.; Levine, S.; Mohiuddin, A.; Sepassi, R.; Tucker, G.; and Michalewski, H. 2020. Model Based Reinforcement Learning for Atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Kim, K.; Ha, J.; and Kim, Y. 2025. Self-Predictive Dynamics for Generalization of Vision-based Reinforcement Learning. CoRR, abs/2506.05418. Liao, W.; Zhang, Z.; and Yu, Y. 2023. Policy-Independent Behavioral Metric-Based Representation for Deep Reinforcement Learning. In Williams, B.; Chen, Y.; and Neville, J., eds., Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, 8746–8754. AAAI Press. Liu, G.; Tang, M.; and Eysenbach, B. 2025. A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoals. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Ma, G.; Li, L.; Zhang, S.; Liu, Z.; Wang, Z.; Chen, Y.; Shen, L.; Wang, X.; and Tao, D. 2024. Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stages. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. A. 2013. Playing Atari with Deep Reinforcement Learning. CoRR, abs/1312.5602. Ota, K.; Oiki, T.; Jha, D. K.; Mariyama, T.; and Nikovski, D. 2020. Can Increasing Input Dimensionality Improve Deep Reinforcement Learning? In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, 7424– 7433. PMLR. Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; Desmaison, A.; Köpf, A.; Yang, E. Z.; DeVito, Z.; Raison, M.; Tejani, A.; Chilamkurthy, S.; Steiner, B.; Fang, L.; Bai, J.; and Chintala, S. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Wallach, H. M.; Larochelle, H.; Beygelzimer, A.; d’Alché-Buc, F.; Fox, E. B.; and Garnett, R., eds., Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 8024–8035. Scannell, A.; Kujanpää, K.; Zhao, Y.; Nakhaei, M.; Solin, A.; and Pajarinen, J. 2024. iQRL - Implicitly Quantized Representations for Sample-efficient Reinforcement Learning. CoRR, abs/2406.02696.

Schrittwieser, J.; Antonoglou, I.; Hubert, T.; Simonyan, K.; Sifre, L.; Schmitt, S.; Guez, A.; Lockhart, E.; Hassabis, D.; Graepel, T.; Lillicrap, T. P.; and Silver, D. 2020. Mastering Atari, Go, chess and shogi by planning with a learned model. Nat., 588(7839): 604–609. Schwarzer, M.; Anand, A.; Goel, R.; Hjelm, R. D.; Courville, A. C.; and Bachman, P. 2021. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Schwarzer, M.; Obando-Ceron, J. S.; Courville, A. C.; Bellemare, M. G.; Agarwal, R.; and Castro, P. S. 2023. Bigger, Better, Faster: Human-level Atari with human-level efficiency. In Krause, A.; Brunskill, E.; Cho, K.; Engelhardt, B.; Sabato, S.; and Scarlett, J., eds., International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, 30365–30380. PMLR. Silver, D.; Lever, G.; Heess, N.; Degris, T.; Wierstra, D.; and Riedmiller, M. A. 2014. Deterministic Policy Gradient Algorithms. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, 387– 395. JMLR.org. Sutton, R. S.; Barto, A. G.; et al. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. Tassa, Y.; Doron, Y.; Muldal, A.; Erez, T.; Li, Y.; de Las Casas, D.; Budden, D.; Abdolmaleki, A.; Merel, J.; Lefrancq, A.; Lillicrap, T. P.; and Riedmiller, M. A. 2018. DeepMind Control Suite. CoRR, abs/1801.00690. Todorov, E.; Erez, T.; and Tassa, Y. 2012. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, 5026–5033. IEEE. Wang, Z.; Schaul, T.; Hessel, M.; van Hasselt, H.; Lanctot, M.; and de Freitas, N. 2016. Dueling Network Architectures for Deep Reinforcement Learning. In Balcan, M.; and Weinberger, K. Q., eds., Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, 1995– 2003. JMLR.org. Yarats, D.; Fergus, R.; Lazaric, A.; and Pinto, L. 2022. Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. Yarats, D.; Kostrikov, I.; and Fergus, R. 2021. Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Zhang, L.; Wang, Z.; Li, X.; and Li, Y. 2025. Revisiting Bisimulation Metric for Robust Representations in Reinforcement Learning. CoRR, abs/2507.18519. Zheng, R.; Wang, X.; Sun, Y.; Ma, S.; Zhao, J.; Xu, H.; III, H. D.; and Huang, F. 2023. TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023.

Supplementary Material Algorithm 1: Training Procedure of OG-SPR Let θV denote the parameters of the online value network, including the encoder, adapters, state-action encoder, value predictors, and auxiliary prediction heads (i.e., the decoder and the short-term value predictor). Let θπ denote the parameters of the online policy network. Let θV′ and θπ′ denote the target parameters used for bootstrapped value estimation and latent self-prediction targets. Let Ttarget denote the update frequency of target networks. 1: Initialize replay buffer B and current training step t 2: while Training do 3: Collect experience (s, a, r, s′ ) and add to buffer B 4: Sample a minibatch of sequences (si , ai , ri:i+n−1 , oi+1 , si+n ) ∼ B 5: Compute latent states hi and state-action representations zi 6: Compute LValue , Ln-step and LRec based on the same zi 7: Compute LPolicy using the same detached hi 8: Sample another minibatch of sequences (si:i+K , ai:i+K−1 ) ∼ B 9: Compute LSelf-predictive 10: Compute the combined loss L 11: Update θV by minimizing L 12: Update θπ by minimizing LPolicy 13: t←t+1 14: if t mod Ttarget = 0 then 15: {θV′ , θπ′ } ← {θV , θπ } 16: end if 17: end while

A

Pseudocode of OG-SPR

Algorithm 1 summarizes the training procedure of OG-SPR.

B

Overview of AnonMethod

AnonMethod (Anonymous 2026) is an observationpredictive model-free RL method that learns representations by predicting dynamics in observation space. It learns representations through next-observation prediction in a normalized observation space, in contrast to self-predictive methods that predict future latent embeddings. In AnonMethod, observations are first normalized and then encoded into latent state representations. This latent state is combined with the action to produce a state-action representation, which is fed to three predictors for value learning, short-term value prediction, and next-observation prediction. The policy network takes the latent state as input with gradients stopped. AnonMethod introduces two auxiliary tasks. The first is short-term value prediction, which predicts the discounted reward accumulated over a short horizon and provides a stabilizing training signal for representation learning. The second is next-observation prediction, where a decoder reconstructs the next normalized observation from the state-action representation. This objective encourages the learned representation to capture observation-grounded dynamics information.

For reproducibility, we include the implementation used for AnonMethod in the supplementary material.

C

Additional Details for Experimental Setup

Computing Infrastructure All experiments were conducted on a server running Ubuntu 24.04.4 LTS (Noble Numbat) on an x86_64 architecture. The server was equipped with two Intel Xeon Platinum 8368Q CPUs operating at 2.60 GHz, providing 76 physical CPU cores in total, 251 GiB of system memory, and four NVIDIA GeForce RTX 4090 GPUs. The software environment consisted of Python 3.9.23 and PyTorch 2.6.0.

Randomness Control For each method or variant on each task, we conduct five independent runs using the random seeds {42, 99, 123, 520, 668}. We control the random number generators used by Python, NumPy, and PyTorch and enable deterministic behavior for supported cuDNN operations.

D

Implementation Details

Network Architecture We follow MR.Q in constructing the actor-critic backbone of OG-SPR. The observation encoder consists of four convolutional (Conv) layers, each with 32 output channels, a kernel size of 3, strides (2, 2, 2, 1) across the four layers, and ELU activations. The convolutional layers are followed by a linear layer applied to the flattened output, layer normalization (LayerNorm), and a final ELU activation. The state-action encoder is a three-layer MLP, with LayerNorm and ELU applied after each of the first two layers. Each long-term value predictor is a four-layer MLP, with LayerNorm and ELU applied after each of the first three layers. Similar to AnonMethod, the decoder consists of a linear layer that projects the input vector to a feature map, followed by four transposed convolutional (TransConv) layers. Spectral normalization is applied to the first layer of the decoder. The short-term value predictor is a two-layer MLP, where the first layer is followed by LayerNorm and an ELU activation. The policy network is a three-layer MLP with LayerNorm followed by ReLU after the first two layers. The source code for OG-SPR is provided in the supplementary material.

Other Design Choices For design choices that are orthogonal to our main representation-learning contribution, we align OG-SPR with dynamics-based baselines MR.Q and AnonMethod. This controlled setup helps attribute performance differences more reliably to the key design differences introduced by OGSPR. Specifically, we use the LAP (Fujimoto, Meger, and Precup 2020) replay buffer to perform prioritized sampling during training. All networks are trained with the AdamW optimizer (Loshchilov and Hutter 2019).

E

Modifications for Atari100k Atari100k (Bellemare et al. 2013; Kaiser et al. 2020) is a data-efficiency benchmark consisting of 26 Atari games for discrete control. Agents are allowed 100k steps of environment interaction with an action repeat of 4, corresponding to 400k frames and approximately two hours of game time. We use the no-sticky-action setting. The input state is constructed by stacking the most recent four frames, which are gray-scaled, resized to 84 × 84 pixels, and set to the max between the 3rd and 4th frame (Mnih et al. 2015). When aggregating scores, we compute human-normalized scores using the human scores reported in (Wang et al. 2016): Human-Normalized(x) =

x − random score . (1) Human score − random score

To adapt OG-SPR to discrete-action domains, we modify the policy network using discrete-action adaptations adopted by MR.Q and AnonMethod. Specifically, the action is computed as follows: ′

aπ = argmax(a′ ), a′ = π ′ (h′ ) + clip(ϵ, −c, c),

ϵ ∼ N (0, σ 2 ).

(2)

Discrete actions are represented by a one-hot encoding, with the Gaussian noise added to each dimension. The activation function applied after the pre-activation policy outputs is Gumbel-Softmax in discrete-action settings. For the Atari100k experiments, we keep the observationprediction and short-term value prediction weights fixed at λRec = 1.0 and λn-step = 1.0, which are the default settings in AnonMethod. To determine λSelf-predictive , we use the default self-prediction loss weight of SPR, 2.0, as the starting point and perform a grid search with a step size of 1.0 on a small subset of games containing only Alien and Amidar. The selected value is λSelf-predictive = 2.0, which is then fixed for all games. While OG-SPR performs one gradient update for both the value network and the policy network per environment step in DMControl, we perform two gradient updates for both networks per environment step on Atari100k following SPR (Schwarzer et al. 2021). OG-SPR clips rewards to {−1, 0, +1} according to their signs, which is a preprocessing method commonly used for Atari games (Mnih et al. 2015; Hessel et al. 2018; Schwarzer et al. 2021).

Observation Prediction Targets The observation for pixel-based benchmarks is composed of several previous frames (three for DMControl and four for Atari100k). This raises a question of how to construct learning targets for the next observation prediction. A naive approach is to construct a complete form of the observation. However, such a target contains redundant information from the current observation and may hinder the model from efficiently learning dynamics-aware representations. We thus remove the overlapping frames between the next and the current observations when constructing targets.

Hyperparameters Table 2 summarizes the default hyperparameters of OG-SPR.

Computational Efficiency Comparison

Table 1 compares the training wall-clock time of OG-SPR with baselines on the cheetah-run task. For this runtime comparison only, we use a smaller DreamerV3 configuration instead of its default 10M-parameter configuration, making its model size comparable to those of the other methods. Since the official DreamerV3 implementation is based on JAX (Bradbury et al. 2018), we use a PyTorch implementation of DreamerV31 for a framework-consistent comparison with the other PyTorch-based methods. All methods are evaluated on a single NVIDIA GeForce RTX 4090 GPU. Methods DreamerV3 OG-SPR (ours) MR.Q TD-MPC2 AnonMethod DrQ-v2

Params (M)

Wall-Clock Time (h)

5.3 5.4 4.2 5.4 4.9 4.4

32.7 14.2 12.0 9.4 7.7 5.7

Table 1: Comparison of training wall-clock time for 500k environment steps on the cheetah-run task.

F

Additional Representation Analysis

This section provides additional details and complete results for the linear probing analysis introduced in the main text. The goal of this analysis is to examine how latent selfprediction and observation prediction affect the information encoded in the learned latent state and state-action representations. We conduct this analysis on four DMControl tasks: dog-trot, humanoid-walk, quadruped-walk, and cheetah-run. The first three tasks are selected as representative cases where OG-SPR outperforms both ablated variants, OG-SPR - SP and OG-SPR - OP, which remove self-prediction (SP) and observation prediction (OP), respectively. The fourth task, cheetah-run, is selected as a case where OG-SPR does not achieve the best performance among the compared variants. For each method, selected task, and final checkpoint, we freeze the learned representation modules, including the visual encoder f , the online Adapter 1 u1 , and the state-action encoder g. We then train lightweight linear predictors on top of the frozen representations to evaluate how much proprioceptive information is encoded in the learned latent state and state-action representations. Let st denote the visual input state and xt denote the corresponding proprioceptive state at time step t. We normalize the proprioceptive states using the online normalization method proposed in AnonMethod (Anonymous 2026) and use the normalized states x̄ as prediction targets. This normalization reduces the influence of proprioceptive dimensions with larger numerical scales on the aggregated prediction loss. We consider two prediction tasks. The first predicts the current proprioceptive state from the latent state representation, with the prediction loss denoted by Lstate . The second predicts the next proprioceptive state from the state-action 1

https://github.com/NM512/dreamerv3-torch/

representation, with the loss denoted by Lnext-state . For the current-state prediction, we train a linear predictor pstate on top of the latent state representation: ht = u1 (f (st )),

(3)

and minimize the prediction loss: 2

Lstate = ∥pstate (ht ) − x̄t ∥2 .

(4)

For the next-state prediction, we train another linear predictor pnext on top of the state-action representation: zt = g(ht , at ),

(5)

and minimize the following loss: 2

Lnext-state = ∥pnext (zt ) − x̄t+1 ∥2 .

(6)

The representation networks are kept fixed throughout training, and only the linear predictors are updated. Thus, lower prediction losses indicate that the corresponding proprioceptive information is more easily recoverable from the frozen representations by a simple linear model.

Training Details All linear predictors are trained with the Adam optimizer using a learning rate of 3 × 10−4 . We apply gradient clipping with a maximum norm of 20. For each prediction task, the linear predictor is trained for 100k gradient updates with data collected by running a fully trained policy. Since different policies may induce different state-action data distributions, we further control for the data-collection policy. For each method, selected task, final checkpoint, and prediction task, we train three linear predictors, each using data collected by executing the trained actor from one of the three methods: OG-SPR, OG-SPR - SP, or OG-SPR OP. For example, in the next-state prediction experiment on dog-trot with a random seed of 42, we train linear predictors using data collected by the OG-SPR actor, the OG-SPR SP actor, and the OG-SPR - OP actor, respectively. For each fully trained run, we average the resulting prediction losses Lstate and Lnext-state over the three data-collection policies, and then report the average across seeds. This protocol reduces the potential advantage from distribution matching. If each method’s linear predictor were trained and evaluated only on data collected by its own policy, its prediction loss could be lower than it would be under a shared data distribution, simply because the data used for linear prediction matches that method’s induced state-action distribution. Averaging over data collected by all three policies therefore provides a fairer comparison of representation quality. Table 3 summarizes the linear prediction results.

G

Complete Results

Tables 4, 5, and 6 report the complete per-task results corresponding to the aggregate results in the main results, ablation study and Atari100k benchmark, respectively.

Components

Hyperparameter

Value

n-step returns Discount factor γ

3 DMControl: (0.1, 1.0, 5.0) Atari100k: (1.0, 1.0, 2.0) 0.99

TD3

Target policy noise σ Target policy noise clipping c

N (0, 0.22 ) (-0.3, 0.3)

LAP

Probability smoothing α Minimum priority

0.4 1

Auxiliary Tasks

Horizon of latent self-prediction K

5

Optimization

Optimizer Learning rate Weight decay Mini-batch size Target update frequency Gradient updates per training step (DMControl) Gradient updates per training step (Atari100k)

AdamW 3e-4 1e-4 256 250 value network: 1 , policy network: 1 value network: 2 , policy network: 2

Value Learning

Auxiliary loss weights (λRec , λn-step , λSelf-predictive )

Initial random exploration time steps Exploration Exploration noise

DMControl: 10k Atari100k: 2k DMControl: N (0, 0.12 ) Atari100k: N (0, 0.22 )

Observation Encoder

Structure State dimension Activation function Gradient clip norm

Conv + MLP 512 ELU 20

State-Action Encoder

Hidden dimension State-action representation dimension Activation function Gradient clip norm

580 512 ELU 20

Long-term value predictor

Hidden dimension Activation function Gradient clip norm

512 ELU 20

Short-term value predictor

Hidden dimension Reward bins Activation function Gradient clip norm Reward range

512 65 ELU 20 [−10, 10] (effective: [−22k, 22k])

Structure Latent channels Activation function Gradient clip norm

MLP + TransConv 128 DMControl: 3 Atari100k: 1 ReLU 20

Hidden dimension Activation function Gumbel-Softmax τ (Atari100k)

512 ReLU 10

Decoder

Policy Network

Output channels

Table 2: Default hyperparameters of OG-SPR.

Methods

Lstate

Lnext-state

Lstate + Lnext-state

Scores

0.802 [0.768, 0.834] 0.803 [0.768, 0.844] 0.984 [0.947, 1.019]

1.042 [1.023, 1.057] 1.053 [1.019, 1.075] 0.908 [0.872, 0.930]

1.845 [1.815, 1.891] 1.856 [1.815, 1.919] 1.892 [1.858, 1.949]

105 [79, 128] 79 [70, 88] 62 [59, 66]

0.666 [0.507, 0.765] 0.666 [0.504, 0.767] 1.119 [1.024, 1.260]

1.184 [1.151, 1.229] 1.193 [1.146, 1.256] 1.142 [1.089, 1.225]

1.850 [1.679, 1.955] 1.860 [1.760, 1.913] 2.261 [2.136, 2.485]

15 [7, 22] 12 [7, 17] 2 [2, 3]

0.565 [0.544, 0.602] 0.563 [0.543, 0.597] 1.848 [1.575, 2.125]

1.185 [1.135, 1.276] 1.189 [1.139, 1.250] 1.088 [1.054, 1.155]

1.750 [1.679, 1.825] 1.752 [1.721, 1.800] 2.936 [2.629, 3.181]

837 [805, 884] 679 [458, 900] 795 [752, 849]

0.0350 [0.0292, 0.0413] 0.0399 [0.0298, 0.0508] 3.729 [3.382, 4.032]

1.600 [1.540, 1.663] 1.541 [1.464, 1.614] 1.587 [1.525, 1.645]

1.635 [1.567, 1.700] 1.581 [1.495, 1.654] 5.316 [4.998, 5.638]

780 [772, 789] 684 [558, 760] 854 [823, 881]

dog-trot OG-SPR OG-SPR - OP OG-SPR - SP humanoid-walk OG-SPR OG-SPR - OP OG-SPR - SP quadruped-walk OG-SPR OG-SPR - OP OG-SPR - SP cheetah-run OG-SPR OG-SPR - OP OG-SPR - SP

Table 3: Linear prediction results on four selected DMControl tasks. Brackets denote 95% bootstrap confidence intervals.

Task

DrQ-v2

TD-MPC2

DreamerV3

MR.Q

AnonMethod

OG-SPR

acrobot-swingup 168 [127, 219] 177 [131, 223] 260 [162, 337] 303 [266, 340] 282 [207, 356] 443 [397, 494] ball_in_cup-catch 909 [821, 973] 746 [494, 942] 968 [965, 971] 974 [970, 977] 975 [973, 979] 980 [977, 983] cartpole-balance 993 [990, 996] 883 [665, 995] 992 [992, 992] 998 [997, 999] 997 [996, 999] 999 [998, 999] cartpole962 [887, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] balance_sparse cartpole-swingup 864 [854, 873] 790 [676, 851] 851 [847, 857] 865 [859, 872] 873 [867, 878] 878 [871, 883] cartpole774 [741, 805] 158 [0, 475] 285 [0, 570] 777 [734, 818] 840 [833, 845] 784 [769, 814] swingup_sparse cheetah-run 728 [701, 753] 548 [452, 612] 733 [694, 772] 759 [752, 768] 835 [814, 867] 780 [772, 789] dog-run 10 [9, 12] 8 [5, 11] 29 [23, 36] 51 [45, 56] 55 [46, 69] 90 [64, 115] dog-stand 43 [37, 49] 161 [147, 175] 118 [78, 157] 251 [233, 269] 239 [214, 263] 357 [267, 448] dog-trot 14 [11, 18] 16 [12, 18] 44 [34, 54] 70 [61, 80] 61 [59, 63] 105 [79, 128] dog-walk 22 [18, 29] 14 [12, 17] 42 [31, 52] 90 [79, 101] 81 [72, 91] 108 [102, 118] finger-spin 860 [787, 922] 987 [985, 989] 843 [704, 981] 835 [682, 982] 987 [985, 989] 981 [973, 986] finger-turn_easy 503 [399, 615] 757 [647, 883] 732 [245, 981] 955 [918, 975] 934 [889, 975] 947 [907, 976] finger-turn_hard 223 [121, 340] 816 [770, 862] 560 [177, 943] 948 [910, 970] 927 [878, 976] 928 [853, 972] fish-swim 84 [65, 107] 67 [52, 85] 126 [61, 223] 74 [67, 82] 68 [66, 70] 86 [58, 115] hopper-hop 224 [170, 278] 234 [220, 247] 294 [289, 299] 236 [183, 289] 276 [256, 295] 293 [263, 325] hopper-stand 917 [903, 931] 755 [624, 886] 873 [850, 896] 925 [919, 930] 899 [848, 930] 945 [935, 955] humanoid-run 1 [1, 1] 1 [1, 1] 1 [1, 2] 1 [1, 2] 1 [1, 1] 3 [2, 4] humanoid1 [1, 2] 1 [1, 1] 1 [1, 2] 1 [1, 2] 1 [1, 2] 13 [5, 25] run (1.0M) humanoid2 [2, 3] 1 [1, 1] 1 [1, 2] 3 [2, 5] 1 [1, 2] 67 [60, 71] run (1.5M) humanoid2 [2, 3] 1 [1, 1] 4 [1, 9] 27 [4, 50] 1 [1, 1] 100 [94, 106] run (2.0M) humanoid-stand 6 [6, 7] 6 [6, 7] 7 [3, 10] 8 [7, 8] 6 [5, 7] 12 [8, 20] humanoid8 [7, 9] 7 [7, 7] 8 [6, 10] 7 [7, 8] 9 [7, 11] 100 [50, 151] stand (1.0M) humanoid8 [8, 9] 7 [6, 7] 6 [3, 8] 14 [9, 20] 8 [7, 9] 248 [195, 300] stand (1.5M) humanoid9 [8, 11] 7 [6, 7] 11 [7, 17] 79 [8, 200] 31 [8, 55] 339 [284, 401] stand (2.0M) humanoid-walk 2 [2, 2] 2 [1, 2] 2 [1, 3] 3 [2, 5] 2 [2, 3] 15 [7, 22] humanoid3 [2, 3] 2 [2, 2] 2 [1, 3] 3 [2, 3] 3 [2, 4] 86 [44, 143] walk (1.0M) humanoid5 [2, 10] 2 [2, 2] 2 [1, 2] 5 [2, 10] 9 [2, 23] 226 [220, 232] walk (1.5M) humanoid19 [2, 52] 2 [2, 2] 2 [1, 3] 48 [3, 93] 54 [3, 153] 316 [299, 333] walk (2.0M) pendulum-swingup 838 [813, 861] 735 [506, 875] 845 [717, 936] 840 [830, 853] 836 [819, 855] 856 [830, 877] quadruped-run 459 [412, 507] 280 [158, 369] 475 [433, 498] 503 [474, 518] 460 [458, 463] 543 [490, 599] quadruped-walk 750 [699, 796] 338 [282, 393] 512 [293, 709] 865 [799, 921] 839 [817, 857] 837 [805, 884] reacher-easy 938 [903, 973] 961 [928, 979] 778 [392, 978] 978 [974, 981] 935 [894, 976] 956 [909, 983] reacher-hard 705 [580, 831] 644 [490, 772] 965 [954, 978] 953 [913, 974] 956 [915, 979] 946 [899, 971] walker-run 546 [475, 612] 710 [681, 738] 717 [664, 748] 562 [502, 619] 683 [669, 705] 707 [678, 736] walker-stand 980 [977, 984] 914 [852, 957] 982 [977, 987] 985 [984, 986] 988 [987, 990] 982 [977, 985] walker-walk 766 [489, 957] 897 [832, 925] 968 [957, 981] 964 [959, 969] 963 [958, 968] 968 [964, 971]

Table 4: Final raw episode returns on DMControl after 500k environment steps (default), corresponding to 1M frames under action repeat of 2. 1.0M, 1.5M, and 2.0M denote results evaluated at 1.0M, 1.5M, and 2.0M environment steps, respectively. The [bracketed values] represent a 95% bootstrap confidence interval.

Figure 1: Per-task learning curves on the DMControl benchmark for OG-SPR and baselines (Part 1). Solid lines indicate average performance over 5 seeds, and shaded areas indicate the 95% bootstrap confidence interval.

Figure 2: Per-task learning curves on the DMControl benchmark for OG-SPR and baselines (Part 2). Solid lines indicate average performance over 5 seeds, and shaded areas indicate the 95% bootstrap confidence interval.

Figure 3: Per-task learning curves on the humanoid tasks from the DMControl benchmark for OG-SPR and baselines under the 2M-environment-step training budget. Solid lines indicate mean performance over 5 seeds, and shaded areas indicate 95% bootstrap confidence intervals.

Task

OG-SPR - OP

OG-SPR - SP

OG-SPR - SVP

Shared Adapter

No Adapter

Adapter 2 Only

acrobot-swingup 455 [412, 511] 268 [221, 277] 391 [360, 431] 412 [343, 460] 318 [250, 365] 419 [414, 426] ball_in_cup-catch 979 [977, 980] 977 [975, 980] 978 [975, 981] 975 [970, 981] 922 [825, 973] 973 [969, 976] cartpole-balance 998 [996, 999] 998 [996, 999] 999 [997, 999] 999 [999, 1000] 999 [999, 1000] 998 [998, 999] cartpole1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 1000 [1000, 1000] 999 [998, 1000] 1000 [1000, 1000] balance_sparse cartpole-swingup 870 [866, 878] 881 [880, 882] 877 [870, 881] 872 [862, 882] 873 [867, 879] 880 [878, 883] cartpole840 [831, 847] 841 [822, 856] 821 [789, 845] 675 [360, 835] 814 [778, 838] 781 [671, 844] swingup_sparse cheetah-run 684 [558, 760] 854 [823, 881] 682 [575, 739] 700 [585, 761] 726 [698, 754] 791 [756, 843] dog-run 71 [52, 90] 62 [50, 80] 78 [58, 99] 84 [77, 91] 78 [57, 100] 57 [49, 67] dog-stand 324 [276, 380] 240 [212, 271] 431 [329, 534] 398 [295, 503] 401 [276, 527] 284 [232, 356] dog-trot 79 [70, 88] 62 [59, 66] 83 [76, 88] 100 [78, 123] 76 [65, 86] 69 [63, 75] dog-walk 103 [100, 106] 81 [73, 90] 114 [99, 129] 101 [88, 114] 114 [102, 131] 98 [85, 111] finger-spin 981 [979, 982] 984 [983, 986] 983 [977, 986] 983 [981, 987] 982 [978, 987] 984 [979, 988] finger-turn_easy 924 [880, 968] 912 [887, 948] 875 [749, 969] 874 [870, 881] 945 [892, 976] 948 [900, 974] finger-turn_hard 943 [895, 971] 920 [882, 959] 960 [952, 969] 940 [902, 962] 747 [590, 858] 728 [507, 916] fish-swim 87 [53, 120] 66 [64, 71] 101 [66, 141] 78 [56, 115] 91 [48, 134] 64 [57, 70] hopper-hop 275 [260, 298] 287 [260, 310] 297 [272, 321] 280 [274, 287] 301 [275, 334] 287 [263, 316] hopper-stand 930 [913, 941] 903 [866, 939] 933 [928, 939] 919 [895, 936] 912 [858, 948] 905 [864, 929] humanoid-run 3 [2, 4] 1 [1, 1] 3 [2, 4] 1 [1, 2] 2 [1, 2] 3 [2, 4] humanoid-stand 9 [6, 12] 7 [5, 8] 12 [8, 17] 11 [6, 16] 12 [8, 16] 8 [7, 8] humanoid-walk 12 [7, 17] 2 [2, 3] 7 [3, 12] 15 [7, 23] 3 [2, 3] 5 [3, 7] pendulum-swingup 828 [785, 889] 844 [822, 859] 836 [793, 882] 830 [812, 852] 852 [837, 868] 762 [593, 852] quadruped-run 545 [469, 620] 481 [461, 502] 520 [478, 602] 519 [424, 614] 414 [360, 464] 441 [410, 473] quadruped-walk 679 [458, 900] 795 [752, 849] 724 [508, 917] 833 [779, 902] 561 [436, 753] 840 [737, 918] reacher-easy 975 [974, 977] 978 [975, 981] 957 [908, 985] 979 [978, 981] 978 [974, 983] 977 [976, 978] reacher-hard 949 [905, 974] 939 [894, 964] 898 [869, 947] 970 [967, 973] 972 [971, 975] 948 [900, 974] walker-run 734 [721, 746] 685 [651, 710] 646 [550, 719] 610 [562, 656] 690 [631, 749] 674 [648, 705] walker-stand 983 [981, 985] 984 [979, 990] 984 [982, 987] 979 [970, 985] 982 [981, 984] 986 [983, 989] walker-walk 965 [961, 969] 972 [961, 982] 968 [966, 972] 965 [961, 969] 957 [944, 965] 964 [957, 971]

Table 5: Final raw episode returns of the ablation variants on DMControl after 500k environment steps, corresponding to 1M frames under action repeat of 2. Results are averaged over 5 seeds. The [bracketed values] represent a 95% bootstrap confidence interval. Due to table-width constraints, the full OG-SPR results are not included in this table and are instead reported in Table 4.

Task

Human Random

SPR

MR.Q

AnonMethod

OG-SPR

Alien 7127.7 227.8 903 [747, 1060] 880 [821, 948] 1158 [973, 1366] 947 [853, 1047] Amidar 1719.5 5.8 203 [183, 228] 174 [154, 196] 155 [121, 190] 201 [155, 247] Assault 742.0 222.4 699 [662, 733] 604 [587, 625] 687 [659, 714] 708 [672, 743] Asterix 8503.3 210.0 922 [871, 974] 1110 [1058, 1161] 1286 [1124, 1447] 1284 [1126, 1509] BankHeist 753.1 14.2 108 [32, 242] 24 [21, 27] 41 [27, 56] 55 [39, 71] BattleZone 37187.5 2360.0 9028 [7130, 11218] 6539 [5122, 7982] 4886 [2718, 7894] 4716 [4346, 5130] Boxing 12.1 0.1 41 [30, 53] 72 [69, 76] 76 [72, 80] 60 [47, 73] Breakout 30.5 1.7 14 [12, 15] 19 [17, 21] 17 [15, 19] 22 [20, 23] ChopperCommand 7387.8 811.0 1295 [1125, 1481] 1381 [1282, 1494] 1741 [1613, 1869] 2041 [1905, 2211] CrazyClimber 35829.4 10780.5 23109 [21408, 24950] 57113 [53805, 60048] 69018 [61868, 75604] 68404 [59739, 79846] DemonAttack 1971.0 152.1 1502 [1375, 1638] 834 [755, 917] 1952 [1449, 2464] 1614 [1407, 1851] Freeway 29.6 0.0 24 [18, 29] 19 [10, 29] 16 [6, 25] 6 [1, 9] Frostbite 4334.7 65.2 2938 [2732, 3114] 2488 [1905, 2937] 1982 [1285, 2601] 1286 [259, 2452] Gopher 2412.5 257.6 550 [448, 649] 746 [661, 826] 810 [713, 905] 698 [594, 783] Hero 30826.4 1027.0 7956 [7587, 8314] 8129 [7216, 9403] 5487 [4613, 6360] 6976 [4846, 8725] Jamesbond 302.8 29.0 405 [368, 446] 411 [374, 447] 376 [334, 410] 434 [382, 498] Kangaroo 3035.0 52.0 4282 [2849, 5552] 1345 [659, 2586] 422 [276, 563] 2634 [944, 5846] Krull 2665.5 1598.0 3923 [3629, 4308] 7571 [7116, 8056] 7570 [7226, 7860] 5467 [5146, 5915] KungFuMaster 22736.3 258.5 15946 [13995, 17624] 16099 [13545, 18704] 14670 [12813, 16451] 9729 [9343, 10336] MsPacman 6951.6 307.3 1529 [1402, 1715] 1740 [1478, 2011] 1353 [1264, 1444] 1578 [1236, 1971] Pong 14.6 -20.7 -6 [-10, 0] 13 [9, 17] 6 [0, 13] 10 [5, 16] PrivateEye 69571.3 24.9 80 [59, 100] 2 [0, 5] 68 [38, 95] 94 [83, 100] Qbert 13455.0 163.9 3240 [2534, 4029] 2397 [1455, 3467] 766 [624, 955] 2740 [1507, 3974] RoadRunner 7845.0 11.5 13353 [9767, 16281] 13898 [11547, 16591] 15285 [11914, 18792] 11274 [8647, 15859] Seaquest 42054.7 68.4 568 [531, 603] 591 [566, 607] 1040 [901, 1179] 736 [541, 895] UpNDown 11693.2 533.4 7266 [5169, 10190] 5974 [5126, 6778] 8508 [5620, 13591] 7545 [5378, 10857]

Table 6: Final raw game scores on Atari100k after 100k environment steps, corresponding to 400k frames under an action repeat of 4. Results are averaged over 5 seeds. The [bracketed values] represent a 95% bootstrap confidence interval.

References Anonymous. 2026. Title Withheld for Anonymous Review. Citation suppressed to preserve double-blind review. Bellemare, M. G.; Naddaf, Y.; Veness, J.; and Bowling, M. 2013. The Arcade Learning Environment: An Evaluation Platform for General Agents. J. Artif. Intell. Res., 47: 253–279. Bradbury, J.; Frostig, R.; Hawkins, P.; Johnson, M. J.; Katariya, Y.; Leary, C.; Maclaurin, D.; Necula, G.; Paszke, A.; VanderPlas, J.; Wanderman-Milne, S.; and Zhang, Q. 2018. JAX: composable transformations of Python+NumPy programs. Fujimoto, S.; Meger, D.; and Precup, D. 2020. An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay. In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. Hessel, M.; Modayil, J.; van Hasselt, H.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M. G.; and Silver, D. 2018. Rainbow: Combining Improvements in Deep Reinforcement Learning. In McIlraith, S. A.; and Weinberger, K. Q., eds., Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 3215–3222. AAAI Press. Kaiser, L.; Babaeizadeh, M.; Milos, P.; Osinski, B.; Campbell, R. H.; Czechowski, K.; Erhan, D.; Finn, C.; Kozakowski, P.; Levine, S.; Mohiuddin, A.; Sepassi, R.; Tucker, G.; and Michalewski, H. 2020. Model Based Reinforcement Learning for Atari. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Loshchilov, I.; and Hutter, F. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M. A.; Fidjeland, A.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015. Human-level control through deep reinforcement learning. Nat., 518(7540): 529–533. Schwarzer, M.; Anand, A.; Goel, R.; Hjelm, R. D.; Courville, A. C.; and Bachman, P. 2021. Data-Efficient Reinforcement Learning with Self-Predictive Representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net. Wang, Z.; Schaul, T.; Hessel, M.; van Hasselt, H.; Lanctot, M.; and de Freitas, N. 2016. Dueling Network Architectures for Deep Reinforcement Learning. In Balcan, M.; and Weinberger, K. Q., eds., Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, 1995– 2003. JMLR.org.

Related documents

• W2Rep: Learning Visual Representations by Watching the World Change#1130048
• PSR: Predictive Sensorimotor Representation Learning for Contact-Rich Manipulation#1036143
• Seeing Conflict: Identifying Self-Control Conflict Attenuates Early Allocation of Visual Attention to Positive Stimuli.#323127
• Reward as Observation: Learning Reward-Based Policies for Rapid Adaptation#1155664
• ForeDrive: Foresight-Guided End-to-End Autonomous Driving with a Planning-Relevant Latent World Model#1063004