arXiv CSopen access

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

arXiv:2606.16933v1 [cs.LG] 15 Jun 2026

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning ARDIANTO WIBOWO∗ , IMT Atlantique, France, Flinders University, Australia, and IRL Crossing, Australia PAULO E. SANTOS, Priori Analytica, IRL Crossing, Australia AMER BAGHDADI, IMT Atlantique, France MATTHEW STEPHENSON, Flinders University, IRL Crossing, Australia KARL SAMMUT, Flinders University, IRL Crossing, Australia JEAN-PHILIPPE DIGUET, CNRS, IRL Crossing, Australia Background: Reinforcement learning (RL) systems often degrade when operating conditions differ from those previously encountered, reflecting changes in the underlying data-generating process. These changes manifest as distributional shifts. They may arise between training and evaluation (as in In-Distribution (ID) and Out-of-Distribution (OOD) generalization), or within non-stationary settings, where environment dynamics evolve over time. However, the literature lacks a unified structural account of distributional shift: some works frame it as train-test mismatch, others as non-stationarity, yet their formal relationship remains unclear. Moreover, existing research primarily focuses on mitigation strategies, rather than analyzing the causal origin of distributional shift within the agent–environment interaction. Objectives: We aim to develop a unified causal-origin taxonomy that characterizes the sources of distributional shift in reinforcement learning and systematically relates distributional shift across ID/OOD generalization and non-stationary settings. Methods: We transfer the classical dataset-shift principle from supervised learning to reinforcement learning by reformulating distributional shift in terms of the generative interaction process. Adopting a Partially Observable Markov Decision Process (POMDP) setting to reflect realistic environments, we decompose the interaction into its structural components—including the state distribution, observation process, policy, reward, and transition dynamics—together with the shifted-time boundary. We then analyze how changes in each component alter the underlying data-generating mechanism. Results: This leads to a causal-origin taxonomy that distinguishes internal (agent-driven) and external (environment-driven) distributional shifts based on the generative interaction process. The shifted-time boundary perspective further characterizes explicit, implicit, and hybrid shifts. This formulation unifies ID/OOD generalization and non-stationarity as structured changes in the underlying process. We also introduce an evaluation framework that quantifies shift impact and agent adaptation through performance degradation and recovery metrics. Conclusions: By grounding distributional shift in the causal-origin structure of RL, this work provides a principled foundation that unifies component-level shift in supervised learning with the generative interaction process in RL, and connects both ∗ Corresponding Author.

Authors’ Contact Information: Ardianto Wibowo, orcid: 0000-0002-0257-7239, [email protected] r, IMT Atlantique, Brest, Finistère, France and Flinders University, Adelaide, South Australia, Australia and IRL Crossing, Adelaide, South Australia, Australia; Paulo E. Santos, orcid: 0000-0001-8484-0354, [email protected], Priori Analytica, IRL Crossing, Adelaide, South Australia, Australia; Amer Baghdadi, orcid: 0000-0002-6181-6500, [email protected], IMT Atlantique, Brest, Finistère, France; Matthew Stephenson, orcid: 0000-0002-3867-5842, [email protected], Flinders University, IRL Crossing, Adelaide, South Australia, Australia; Karl Sammut, orcid: 0000-0002-7654-9978, [email protected], Flinders University, IRL Crossing, Adelaide, South Australia, Australia; Jean-Philippe Diguet, orcid: 0000-0003-0728-6040, [email protected], CNRS, IRL Crossing, Adelaide, South Australia, Australia.

This work is licensed under a Creative Commons Attribution International 4.0 License.

© 0000 Copyright held by the owner/author(s). doi: 10.1613/jair.1.xxxxx Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:2

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

ID/OOD generalization and non-stationarity adaptation under a common perspective. This establishes a systematic basis for analyzing robustness under distributional shift in reinforcement learning. JAIR Associate Editor: Insert JAIR AE Name JAIR Reference Format: Ardianto Wibowo, Paulo E. Santos, Amer Baghdadi, Matthew Stephenson, Karl Sammut, and Jean-Philippe Diguet. 0000. A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning. Journal of Artificial Intelligence Research 00, Article 00 ( 0000), 36 pages. doi: 10.1613/jair.1.xxxxx

1

Introduction

Modern machine learning systems are typically developed under the assumption that training and test data are drawn from the same distribution, formalized as the independent and identically distributed (i.i.d.) assumption (Shalev-Shwartz and Ben-David 2014; Vapnik 1999). In real-world evaluation, however, this assumption rarely holds. Data-generating processes may evolve due to changes in sensors, environmental conditions, domain representations, or task specifications, creating discrepancies between training and evaluation distributions (Amodei et al. 2016; Dulac-Arnold et al. 2021). In supervised learning, such discrepancies are studied under the framework of dataset shift (Quiñonero-Candela et al. 2010), formally defined as a mismatch between training and test joint distributions, 𝑃train (𝑥, 𝑦) ≠ 𝑃test (𝑥, 𝑦). This framework is grounded in the factorization 𝑃 (𝑥, 𝑦) = 𝑃 (𝑥)𝑃 (𝑦 | 𝑥), which enables systematic reasoning about where mismatch occurs, leading to well-established categories such as covariate shift, label shift, and concept drift (J. Lu et al. 2018; Sugiyama et al. 2007). Complementarily, uncertainty-based approaches quantify shift through predictive behavior; for instance, methods such as Prior Networks (Malinin and Gales 2018) model a distribution over predictive distributions to distinguish data noise from distributional shift. Empirical studies further show that predictive uncertainty and model confidence can degrade substantially under such perturbations (Ovadia et al. 2019). Reinforcement learning differs fundamentally from supervised learning. Rather than receiving independent samples from a fixed distribution, a reinforcement learning agent generates data through sequential interaction with an environment (Sutton and Barto 2018). This interaction forms a closed-loop system: policy updates influence state visitation distributions, while changes in transition dynamics or observation mappings affect subsequent decisions. Consequently, the data distribution in RL is jointly determined by the agent and the environment, and distributional shift may arise from changes in state distributions, transition dynamics, observation processes, reward functions, or policy parameters. In realistic settings, this interaction is more accurately modeled as a Partially Observable Markov Decision Process (POMDP), where the agent does not directly observe the true state, but instead receives partial observations (Kaelbling et al. 1998; Q. Liu et al. 2022). Robustness under such shifts has been widely studied. Existing work addresses out-of-distribution (OOD) generalization (Cobbe et al. 2019; Korkmaz 2023; Packer et al. 2019; Tamang et al. 2025), non-stationary environments (Padakandla 2022; Papoudakis, Christianos, et al. 2019), and continual learning (Khetarpal et al. 2022). Methodological approaches include domain randomization (Tobin et al. 2017), meta-reinforcement learning (Finn et al. 2017), robust value estimation (Fujimoto et al. 2023), and invariant representation learning (Wang et al. 2025). These works improve empirical robustness under varying conditions. Despite this extensive literature, a conceptual gap remains. In reinforcement learning, distributional shift is often studied through specific manifestations, including state or transition perturbations (Fujimoto et al. 2023; Haider et al. 2021), offline-to-online mismatch (Lee et al. 2020), and trajectory-level divergence (Luo et al. 2025). These perspectives analyze discrepancies in observed behavior or data, but do not explicitly identify which component of the underlying generative interaction process has changed. As a result, environment-driven changes (e.g., altered dynamics or initial-state distributions) and agent-driven changes (e.g., policy updates Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:3

or representation variations) are frequently treated under a common label, despite corresponding to distinct modifications of this process. This lack of structural separation obscures the causal origin of shift and complicates both diagnosis and evaluation. We address this gap by grounding distributional shift in the causal-origin structure of reinforcement learning. Identifying the causal origin of a shift is important because robustness and adaptation depend on what component of the interaction process has changed. If distributional shift is treated only as a generic phenomenon to be adapted to, the source of mismatch remains unclear, and adaptation strategies may be misaligned with the underlying change. Recent benchmark studies provide empirical motivation for this concern. Robust-Gymnasium (Gu et al. 2025) evaluates agents under multiple disturbance types, and shows that performance degradation differs across these disturbance types. Complementarily, RL-ViGen (Yuan et al. 2023) evaluates visual generalization across several categories, such as visual appearance, camera view, lighting condition, scene structure, and crossembodiment changes, and shows that no single method consistently performs best across all categories. These findings suggest that policy robustness is shift-type dependent. Unlike these benchmark-oriented works, our objective is not to introduce another evaluation suite, but to provide a unified taxonomy that locates the causal origin of distributional shift within the RL generative process. Our Contributions. The main contributions of this work are as follows: • We provide a unified structural formulation of distributional shift in reinforcement learning by restating the probabilistic foundation of dataset shift (Quiñonero-Candela et al. 2010) and connecting it to the Markov decision process (MDP) formulation (Sutton and Barto 2018). We show that both admit a shared chain-rule factorization, which allows distributional shift in RL to be described not only by its empirical effect on performance, but also by the component of the generative process in which the mismatch originates. • We derive a causal-origin taxonomy that distinguishes shifts in the state distribution, observation-generation process, policy-induced action/visitation distribution, transition dynamics, and reward mechanism. This taxonomy explains how standard RL settings, such as In-Distribution (ID)/Out-of-Distribution (OOD) generalization and non-stationarity, arise from changes in specific parts of the RL generative process. • We demonstrate that the proposed taxonomy provides a unifying framework to compare and categorize existing approaches to distributional shift in reinforcement learning. • We introduce an evaluation framework composed of several complementary metrics to quantify the impact of distributional shift and agent adaptation performance.

2 Background 2.1 Classical Dataset Shift in Machine Learning In supervised learning, data are assumed to be independently and identically distributed (i.i.d.) between training and evaluation. Formally, the joint distribution of inputs and outputs factors as 𝑃 (𝑥, 𝑦) = 𝑃 (𝑦|𝑥) 𝑃 (𝑥),

(1)

where 𝑃 (𝑥) denotes the input (covariate) distribution and 𝑃 (𝑦|𝑥) the conditional labeling mechanism. Dataset shift is defined as any situation in which the joint distribution differs between training and evaluation, i.e., 𝑃train (𝑥, 𝑦) ≠ 𝑃 deploy (𝑥, 𝑦) (Quiñonero-Candela et al. 2010). The importance of Eq. 1 lies in its ability to localize the source of distributional discrepancy within a probabilistic decomposition. Shift can be categorized according to which component of the factorization changes: • Covariate shift (𝑃 (𝑥) changes). The input distribution varies while the conditional mapping 𝑃 (𝑦|𝑥) remains fixed. This arises in domain adaptation settings, sensor changes, sampling bias, or demographic shifts. The predictive function remains valid, but it is evaluated under a new input distribution. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:4

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

• Prior-probability shift (𝑃 (𝑦) changes). The marginal label distribution changes while class-conditional distributions remain stable. This occurs under changing class prevalence or population drift, affecting calibration and decision thresholds. • Conditional shift / Concept drift (𝑃 (𝑦|𝑥) changes). The mapping from inputs to outputs evolves. Concept drift (J. Lu et al. 2018) emphasizes temporal evolution in the predictive relationship, often reflecting changes in the underlying causal mechanism or decision boundary. • Domain or representation shift. Domain shift occurs when the measurement system or representation changes. An underlying latent covariate 𝑥 0 is assumed, but only an observable representation 𝑥 = 𝑓 (𝑥 0 ) is available. Variations in the measurement function 𝑓 —due to sensor differences, lighting changes, calibration variation, or preprocessing— alter the observed input space while leaving the latent structure intact. The decomposition in Eq. 1 provides a principled way to attribute distributional change to specific components of the data-generating process. This structural clarity is central to theoretical analyses, adaptation guarantees, and robustness design in supervised learning. In this research, this factorization is used as a conceptual baseline to highlight the structural clarity available in supervised learning. Specifically, the decomposition into 𝑃 (𝑥) and 𝑃 (𝑦|𝑥) serves as a reference point for identifying how distributional shift can be localized within a causal-origin process. Our work extends this perspective to reinforcement learning, where an analogous decomposition is not explicitly defined, motivating a reformulation of shift in terms of the underlying interaction dynamics.

2.2

Markov Decision Processes in Reinforcement Learning

A Markov Decision Process (MDP) is defined by the tuple (S, A, 𝑃, 𝑅, 𝛾), where S is the state space, A the action space, 𝑃 (𝑠 ′ | 𝑠, 𝑎) the transition probabilities, 𝑅(𝑠, 𝑎) the expected reward function, and 𝛾 ∈ [0, 1] the discount factor (Howard 1960; Puterman 2005). In the infinite-horizon formulation, it is standard to assume that 𝑃 and 𝑅 do not depend on the decision stage, corresponding to a time-homogeneous controlled Markov process. Finite-horizon formulations allow stagedependent transitions 𝑃𝑡 (𝑠 ′ | 𝑠, 𝑎) and rewards 𝑅𝑡 (𝑠, 𝑎) (Howard 1960; Puterman 2005). When these quantities vary with 𝑡, the resulting controlled process is non-homogeneous. Within this framework, agent–environment interaction unfolds sequentially: at each time step 𝑡, the agent selects an action 𝐴𝑡 ∈ A in state 𝑆𝑡 ∈ S according to a policy 𝜋 (𝑎 | 𝑠) = Pr{𝐴𝑡 = 𝑎 | 𝑆𝑡 = 𝑠}, after which the environment produces a reward 𝑅𝑡 +1 and transitions to a new state 𝑆𝑡 +1 according to 𝑝 (𝑠 ′, 𝑟 | 𝑠, 𝑎) = Pr{𝑆𝑡 +1 = 𝑠 ′, 𝑅𝑡 +1 = 𝑟 | 𝑆𝑡 = 𝑠, 𝐴𝑡 = 𝑎} (Sutton and Barto 2018). In real-world settings, full state observability is rarely satisfied, making the Partially Observable Markov Decision Process (POMDP) formulation more appropriate for modeling agent–environment interaction (Åström 1965; Kaelbling et al. 1998). In this setting, the true state 𝑆𝑡 is not directly accessible. Instead, the agent receives an observation governed by 𝑝 (𝑜 | 𝑠) = Pr{𝑂𝑡 = 𝑜 | 𝑆𝑡 = 𝑠}.

(2)

This interaction induces a stochastic process over trajectories (𝑆 0, 𝐴0, 𝑅1, 𝑆 1, 𝐴1, . . . ), whose distribution is jointly determined by the policy 𝜋 and the environment dynamics 𝑃. Unlike supervised learning, samples are not independently drawn from a fixed distribution, but arise from the recursive composition of policy and transition mechanisms, leading to path-dependent data. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:5

This research focuses on the POMDP setting, where the interaction process is interpreted as a generative mechanism that produces data through the composition of underlying processes. This perspective enables us to characterize distributional shift in RL as changes in specific components of this generative structure.

2.3

In-Distribution (ID) and Out-of-Distribution (OOD)

Classical statistical learning theory assumes that training and evaluation samples are drawn from a fixed but unknown probability distribution 𝑃 (𝑋, 𝑌 ). Under this assumption, empirical risk minimization provides guarantees on generalization because both the training and evaluation data are realizations of the same datagenerating mechanism (Devroye et al. 2008; Vapnik 1999). Formally, let 𝑃train (𝑋, 𝑌 ) denote the joint distribution governing the training data and 𝑃 deploy (𝑋, 𝑌 ) the distribution encountered at evaluation. The classical framework presumes 𝑃 train (𝑋, 𝑌 ) = 𝑃 deploy (𝑋, 𝑌 ). When this equality does not hold, the i.i.d. assumption is violated. This situation is known in the statistical literature as dataset shift (Quiñonero-Candela et al. 2010), and is characterized by 𝑃 train (𝑋, 𝑌 ) ≠ 𝑃 deploy (𝑋, 𝑌 ). Within this probabilistic formulation, the terminology in-distribution (ID) refers to inputs drawn from 𝑃train (𝑋, 𝑌 ), i.e., from the same distribution under which the predictor was learned. Conversely, out-of-distribution (OOD) refers to inputs governed by a distinct distribution 𝑃deploy ≠ 𝑃 train . Thus, OOD designates the violation of the fixed-distribution assumption underlying standard generalization theory. In this research, the ID–OOD distinction is reinterpreted beyond static dataset mismatch. Instead of defining OOD solely as a discrepancy between fixed training and evaluation distributions, we relate it to changes in the underlying generative process governing agent–environment interaction. This allows us to unify classical ID–OOD generalization with dynamic, interaction-driven shifts in reinforcement learning.

2.4

Stationarity and Non-Stationarity

In probability theory, a stochastic process {𝑍𝑡 }𝑡 ≥0 is said to be strictly stationary if its finite-dimensional distributions are invariant under time shifts (Athanasios and Pillai 2002; Doob 1990). Formally, for any integer 𝑘 ≥ 1, any time indices 𝑡 1, . . . , 𝑡𝑘 , and any shift 𝜏, 𝑃 (𝑍𝑡1 , . . . , 𝑍𝑡𝑘 ) = 𝑃 (𝑍𝑡1 +𝜏 , . . . , 𝑍𝑡𝑘 +𝜏 ). If this invariance condition does not hold, the process is said to be non-stationary. A Markov chain is a particular type of stochastic process satisfying the Markov property. For a Markov chain {𝑋𝑡 }𝑡 ≥0 , stationarity is closely related to whether its transition probabilities depend on time. The chain is called time-homogeneous if 𝑃 (𝑋𝑡 +1 = 𝑥 ′ | 𝑋𝑡 = 𝑥) = 𝑃 (𝑥 ′ | 𝑥), independent of 𝑡 (Levin and Peres 2009; Norris 1997). If the transition probabilities depend explicitly on time, written 𝑃𝑡 (𝑥 ′ | 𝑥), the chain is referred to as non-homogeneous. Such time dependence generally results in a non-stationary stochastic process. In this research, stationarity is interpreted as invariance of the agent–environment interaction process over time, while non-stationarity corresponds to temporal variation in this process. Such variation implies changes in the underlying mechanism generating data, thereby inducing distributional shift. This perspective aligns with our objective of identifying the causal origin of distributional shift in reinforcement learning. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:6

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

3

Related Work and Gap

Distributional shift in reinforcement learning has been examined from multiple perspectives, including survey taxonomies, dynamic-environment analysis, and trajectory-level mismatch formulations. However, these perspectives are often developed in parallel, resulting in fragmented definitions and methodological classifications. In this section, we organize the literature into four parts: (1) survey and taxonomy literature, (2) dynamic environments and ID/OOD perspectives together with mitigation strategies, (3) existing definitions of distributional shift in RL, and (4) the resulting conceptual gap. This structure clarifies how existing work characterizes distributional shift and motivates the need for a causal-origin decomposition grounded in the generative structure of reinforcement learning.

3.1

Survey and Taxonomy Literature

Research on distributional shift originates in statistical learning theory, where classical methods assume that training and test data are drawn from the same underlying distribution (Vapnik 1999). Within this framework, dataset-shift taxonomies distinguish covariate shift, prior-probability shift, and domain/representation shift when 𝑃train (𝑥, 𝑦) ≠ 𝑃test (𝑥, 𝑦), as well as conditional shift (concept drift), which captures temporal changes in the data-generating process (J. Lu et al. 2018; Quiñonero-Candela et al. 2010). Large-scale empirical studies further show that predictive uncertainty can deteriorate under such distributional perturbations (Ovadia et al. 2019). Surveys in supervised learning systematize OOD generalization and shift-related methodologies (J. Liu et al. 2023; Tamang et al. 2025), as well as broader machine learning safety and robustness taxonomies (Mohseni et al. 2023). Within reinforcement learning, surveys examine generalization and representation under distributional variation, including zero-shot and task generalization in DeepRL (Kirk et al. 2023), state representation learning under observation variability (Echchahed and Castro 2025), overfitting in deep reinforcement learning (Zhang et al. 2018), sim-to-real transfer in robotics (Zhao et al. 2020), dynamically varying environments (Padakandla 2022), continual reinforcement learning (Khetarpal et al. 2022), and generalization benchmarks (Korkmaz 2023; Packer et al. 2019). In multi-agent settings, (Papoudakis, Christianos, et al. 2019) surveys moving-target learning problems caused by concurrently adapting agents. Recent robustness and generalization benchmarks further motivate this gap. (Gu et al. 2025) shows that degradation patterns differ across disturbance types, and (Yuan et al. 2023) shows that no single method is consistently best across visual generalization categories. These benchmarks are primarily designed to evaluate algorithmic robustness and generalization under different experimental conditions. In contrast, they do not aim to provide a generative account of where the underlying mismatch originates in the RL interaction process. Our work addresses this complementary problem by locating distributional shift within the components of the RL generative process, thereby clarifying why an agent’s policy performance under distributional shift can vary depending on the type of underlying change being evaluated.

3.2

Dynamic Environments and Mitigation Strategies

Within reinforcement learning, distributional variation in dynamic environments is commonly studied through the lenses of non-stationarity and ID/OOD generalization. These perspectives capture how the underlying interaction process may evolve over time or differ between training and evaluation conditions. Non-stationarity arises when the environment dynamics change during interaction. In single-agent settings, this typically corresponds to variations in transition functions or reward mechanisms over time. Existing approaches address such changes through mechanisms such as change-point detection (Padakandla et al. 2020), behavioraware adaptation (Z. Liu et al. 2024), context detection (Da Silva et al. 2006), and stability-oriented corrections (J. Jiang and Z. Lu 2022; Li et al. 2022). In multi-agent settings, non-stationarity further emerges from evolving agent populations and concurrently adapting policies. Representative approaches include smooth Q-learning Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:7

(Amhraoui and Masrour 2023), multi-timescale updates (Emami et al. 2023), dynamic belief modeling (Zhai et al. 2023), adaptive partner modeling (C. Xu et al. 2024), continuous adaptation via meta-learning (Al-Shedivat et al. 2018), and clustering-based strategies (Du et al. 2024). ID/OOD generalization considers the case where evaluation environments differ from those encountered during training (Korkmaz 2023). Empirical studies consistently report performance degradation under such conditions, including state, transition, visual, or domain perturbations (Fujimoto et al. 2023; Haider et al. 2021). Related work on offline-to-online mismatch further highlights instability arising from discrepancies between training datasets and the distributions induced during policy execution (Lee et al. 2020). To address these challenges, a range of mitigation strategies has been proposed. Meta-reinforcement learning develops policies capable of rapid adaptation to new tasks drawn from a meta-distribution (Beck et al. 2024), with representative approaches such as MAML (Finn et al. 2017), RL2 (Duan et al. 2016), and distributionally adaptive meta-learning (Ajay et al. 2022; T. Xu et al. 2024). Domain randomization broadens the training distribution over visual and physical parameters to improve robustness across environments (Tobin et al. 2017), including dynamics randomization (Muratore et al. 2021; Tiboni et al. 2023) and hybrid strategies combining randomization with real-world fine-tuning (Shakerimov et al. 2023). Additional approaches include shift-aware value correction (Chen 2024), robust transition modeling (Herremans et al. 2024), evaluation-time policy switching (Neggatu et al. 2025), and latent distribution alignment (Wang et al. 2025). While these approaches improve empirical robustness, they primarily operate at the level of mitigation after distributional shift occurs, without explicitly identifying which component of the agent–environment interaction process has changed. This motivates the need for a structural characterization of distributional shift based on its causal origin.

3.3

Existing Definitions of Distributional Shift in RL

Existing definitions of distributional shift in reinforcement learning typically characterize shift through observable discrepancies in interaction data, such as trajectory distributions, state–action visitation frequencies, dataset composition, or empirical performance variation. Trajectory-ratio formulations. A representative example is provided by (Luo et al. 2025), who define distributional shift through the ratio between two trajectory distributions: 𝑐

𝑞 𝜋 (𝜏) Ö 𝑞(𝑠𝑡 +1 | 𝑠𝑡 , 𝑎𝑡 ) Ö 𝜋 𝑐 (𝑎𝑡 | 𝑠𝑡 ) . = 𝑝 𝜋 (𝜏) 𝑝 (𝑠𝑡 +1 | 𝑠𝑡 , 𝑎𝑡 ) 𝑡 𝜋 (𝑎𝑡 | 𝑠𝑡 ) 𝑡 | {z } | {z }

(3)

policy shift

model bias 𝑐

where 𝜏 = (𝑠 0, 𝑎 0, 𝑠 1, . . . ) denotes a trajectory, and 𝑝 𝜋 (𝜏) and 𝑞 𝜋 (𝜏) are trajectory distributions induced by policies 𝜋 and 𝜋 𝑐 , respectively. The terms 𝑝 (𝑠𝑡 +1 | 𝑠𝑡 , 𝑎𝑡 ) and 𝑞(𝑠𝑡 +1 | 𝑠𝑡 , 𝑎𝑡 ) represent transition dynamics, while 𝜋 (𝑎𝑡 | 𝑠𝑡 ) and 𝜋 𝑐 (𝑎𝑡 | 𝑠𝑡 ) denote action-selection probabilities. This decomposition attributes trajectory-level discrepancy to model bias in the dynamics and policy shift in action selection. However, it remains at the level of observable behavior, as it does not identify which component of the generative interaction process has changed or whether the shift originates from the agent or the environment. State–action visitation perspectives. Another common view characterizes distributional shift through changes in empirical state–action visitation frequencies (Wang et al. 2025). Datasets generated by different policies, or by the same policy across different time periods, can induce substantially different action distributions. Such differences are typically visualized using density estimates or visitation overlays, revealing discrepancies in empirical distributions without identifying their underlying cause. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:8

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Rollout–training distribution mismatch. (Chen 2024) characterize shift through mismatches between the training distribution and the rollout distribution induced by an updated policy: guide

rollout

𝑑 train −−−−−→ 𝜋𝑘+1 −−−−−−→ 𝑑 𝜋𝑘+1 ,

𝜂 (𝜋𝑘+1 ).

(4)

where 𝑑 train denotes the training data distribution, 𝜋𝑘+1 is the updated policy, and 𝑑 𝜋𝑘+1 is the state–action distribution induced by rolling out 𝜋𝑘+1 in the environment. The operator guide represents the update from data to policy, while rollout denotes interaction with the environment. The term 𝜂 (𝜋𝑘+1 ) is the expected return of the updated policy. This formulation captures mismatch between training and rollout distributions, but does not identify which component of the interaction process has changed or where the shift originates. Offline–online dataset mismatch. Offline reinforcement learning highlights distributional mismatch between datasets collected under different policies or interaction regimes (Lee et al. 2020). Such mismatches are often illustrated through low-dimensional embeddings, which reveal partial overlap between datasets but do not explain whether the discrepancy arises from environment variation, policy drift, observation changes, or dataset bias. Environment-domain perspectives. Other work frames distributional shift as domain perturbations in the environment itself (Haider et al. 2021), such as changes in lighting, terrain, or sensor conditions. These perspectives describe variation across environments without formally relating such changes to the underlying interaction process. Across these perspectives, distributional shift is consistently identified through discrepancies in observed data or behavior, but without specifying where the change originates. We next formalize this gap and position our framework accordingly.

3.4

Gap and Our Position

Existing definitions of distributional shift in reinforcement learning primarily describe observable discrepancies in interaction data, such as trajectory divergence, state–action visitation mismatch, dataset differences, environmental perturbations, or performance degradation. These perspectives describe what differs, but do not specify which component changes or where the change originates in the agent–environment interaction. This limitation stems from a fundamental difference with supervised learning. In supervised learning, distributional shift is analyzed through causal-origin factors (e.g., changes in inputs, labels, or their mapping). In reinforcement learning, data is generated through interaction, making such decomposition less explicit. This motivates a causal-origin reinterpretation in RL. From Distributional Shift in Supervised to Reinforcement Learning. Figure 1 illustrates this difference. In supervised learning, the dataset is fixed, and shift is defined through changes in its factorization (e.g., covariate, label, or concept shift). In reinforcement learning, data is generated through agent–environment interaction, governed by a generative process formalized by the MDP. Distributional shift must therefore be defined at the level of this process, not only through observed discrepancies. Existing formulations exhibit three main limitations: • Lack of generative decomposition. Current definitions do not identify which component of the interaction process has changed (e.g., state distribution, observation mapping, policy, transition dynamics, or reward generation). As a result, environment-driven and agent-driven changes are not distinguished. • Ambiguous problem framing. Distributional shift is variably treated as (i) training–evaluation mismatch (ID/OOD), (ii) non-stationarity over time, or (iii) a mixture of both, without a unifying formulation. • Post-shift, solution-centered views. Existing taxonomies organize methods by mitigation strategies or observed behavior after the shift, rather than by the factors that generate it. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:9

Fig. 1. Conceptual bridge from supervised learning to reinforcement learning. In supervised learning, distributional shift is defined over a given dataset through changes in its factorization. In reinforcement learning, data is generated through interaction and governed by a generative process. This motivates a causal-origin reinterpretation of distributional shift in RL.

This lack of identification has direct consequences. As indicated by recent benchmark studies, different disturbance types can induce different degradation patterns, and the best-performing method can vary across generalization categories (Gu et al. 2025; Yuan et al. 2023). Thus, treating distributional shift only through post-shift performance or method families can obscure the source of mismatch. Our causal-origin perspective addresses this limitation by locating the shift within the components of the RL generative process. Conceptual Position of Our Framework. Figure 2 positions our taxonomy. The dashed line marks the shift at time 𝑡. Existing works focus on post-shift behavior and solution-level categories, while our framework analyzes the causal-origin factors that precede and generate the shift. Solution Taxonomies

causal-origin

Sub-Problem Taxonomies

(𝑡 − ) Before the shift: causal-origin factors

Distributional Shift (𝑡)

(𝑡 + )

After the shift: observable behaviour + methods

Fig. 2. Conceptual position of our taxonomy relative to existing taxonomies. The dashed vertical line marks the occurrence of distributional shift at time 𝑡. Existing surveys focus on observable behavior and solution-level characterizations after the shift (right), whereas our framework analyzes the causal-origin factors that give rise to the shift before it occurs (left). Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:10

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

We define distributional shift at the level of the causal-origin structure, by mapping changes in the interaction process to components of the MDP. A principled formulation requires identifying: • the component of the interaction process that changes, • the origin of the change (agent or environment), • and the point in time at which the change occurs. This provides a structural basis for analyzing distributional shift, separating its origin from its observable effects. In the next section, we introduce a taxonomy based on these factors.

4

Causal-origin Taxonomy of Distributional Shift in Reinforcement Learning

Understanding distributional shift in reinforcement learning requires moving beyond observable trajectory differences and instead exposing the causal structure of the agent–environment interaction itself. Unlike supervised learning—where dataset shift can be localized within the factorization 𝑃 (𝑥, 𝑦) = 𝑃 (𝑦 | 𝑥)𝑃 (𝑥)—reinforcement learning involves a closed-loop generative process in which state, observation, action, transition, and reward are interdependent. To systematically characterize distributional shift in this setting, we propose a causal-origin taxonomy that decomposes shift along two complementary dimensions. First, we identify where the shift originates within the underlying generative process, leading to the MDP component aspect. Second, we characterize how the shift manifests across interaction regimes over time, leading to the boundary shift aspect. Figure 3 shows the proposed taxonomy. Under the MDP component aspect, distributional shift is localized according to which component of the RL generative process changes. From this perspective, shifts arise either from internal (agent-driven) or external (environment-driven) factors. Internal factors include changes in perception and decision-making, namely the observation mapping 𝑝 (𝑜 | 𝑠) and the policy 𝜋 (𝑎 | 𝑜). External factors include changes in the state distribution 𝑝 (𝑠), transition dynamics 𝑝 (𝑠 ′ | 𝑠, 𝑎), and reward generation 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ). Complementarily, the boundary shift aspect characterizes how distributional changes are expressed across interaction regimes over time. Specifically, it concerns the presence and observability of the boundary that separates distinct generative regimes. From this perspective, shifts may be categorized as explicit, implicit, or explicit-implicit (hybrid). Explicit shifts arise when a clear and operationally defined boundary between regimes is imposed (e.g., via policy freezing). Implicit shifts occur when such a boundary is not directly observable or explicitly defined, as in non-stationary or continual learning settings, where changes may be gradual or unknown. Hybrid shifts combine both characteristics, where a boundary may be partially defined while changes continue to evolve across regimes. The following subsections elaborate each aspect in detail, followed by a formal definition of distributional shift in RL and its extension to decentralized multi-agent reinforcement learning (MARL).

4.1

Internal and External Shift under the MDP formulation

We begin by formalizing the generative structure of the agent–environment interaction at a single step in a partially observable setting. At each interaction, experience is generated through the following ordered process: 𝑠 ∼ 𝑝 (𝑠),

𝑜 ∼ 𝑝 (𝑜 | 𝑠),

𝑎 ∼ 𝜋 (𝑎 | 𝑜),

𝑠 ′ ∼ 𝑝 (𝑠 ′ | 𝑠, 𝑎),

𝑟 ∼ 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ).

Each component plays a distinct causal role: • 𝑝 (𝑠) determines how the environment distributes the state. • 𝑝 (𝑜 | 𝑠) determines how the agent perceives the state through its observation mechanism. • 𝜋 (𝑎 | 𝑜) determines how the agent selects actions based on observations. • 𝑝 (𝑠 ′ | 𝑠, 𝑎) specifies the state transition dynamics of the environment. • 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) specifies the reward mechanism given by the environment. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

(5)

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:11

Fig. 3. Causal-origin taxonomy of distributional shift in reinforcement learning. The taxonomy combines two complementary perspectives: (i) the MDP component aspect, which localizes shift according to the underlying generative component that changes, and (ii) the boundary shift aspect, which characterizes how the before-after shift boundary is expressed.

This ordered process induces the joint distribution 𝑝 (𝑠, 𝑜, 𝑎, 𝑠 ′, 𝑟 ) = 𝑝 (𝑠) 𝑝 (𝑜 | 𝑠) 𝜋 (𝑎 | 𝑜) 𝑝 (𝑠 ′ | 𝑠, 𝑎) 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ),

(6)

making explicit that the RL interaction is governed by five interdependent generative components. To relate this formulation to the classical dataset-shift decomposition 𝑃 (𝑥, 𝑦) = 𝑃 (𝑦 | 𝑥)𝑃 (𝑥) in supervised learning, we interpret a single RL interaction step as an input–output pair and group variables as 𝑋 = (𝑠, 𝑜),

𝑌 = (𝑎, 𝑠 ′, 𝑟 ).

Here, 𝑋 represents all information available to the agent prior to decision-making, analogous to the input in supervised learning. In contrast, 𝑌 represents the outcome of the interaction given 𝑋 , including both the selected action and its consequences in the environment, namely the next state and reward. Under this interpretation, each interaction step in reinforcement learning can be viewed as a structured input–output sample, where the output is not a single label but a tuple capturing both the agent’s decision and the resulting environmental response. Starting from the standard decomposition, we obtain: 𝑝 (𝑋, 𝑌 ) = 𝑝 (𝑌 | 𝑋 ) 𝑝 (𝑋 ) = 𝑝 (𝑎, 𝑠 ′, 𝑟 | 𝑠, 𝑜) 𝑝 (𝑠, 𝑜)

(7)

From the generative process defined in Eq.5 , each variable depends only on a specific subset of preceding variables. In particular, the action 𝑎 depends only on the observation 𝑜, the next state 𝑠 ′ depends only on the state–action pair (𝑠, 𝑎), and the reward 𝑟 depends only on (𝑠, 𝑎, 𝑠 ′ ). Therefore, the conditional distribution simplifies as: 𝑝 (𝑋, 𝑌 ) = 𝜋 (𝑎 | 𝑜) 𝑝 (𝑠 ′ | 𝑠, 𝑎) 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) 𝑝 (𝑜 | 𝑠) 𝑝 (𝑠) = 𝑝 (𝑠, 𝑜, 𝑎, 𝑠 ′, 𝑟 ).

(8)

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:12

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Thus, the classical decomposition 𝑃 (𝑥, 𝑦) = 𝑃 (𝑦 | 𝑥)𝑃 (𝑥) is fully consistent with the generative structure of reinforcement learning. In particular, 𝑝 (𝑌 | 𝑋 ) = 𝜋 (𝑎 | 𝑜) 𝑝 (𝑠 ′ | 𝑠, 𝑎) 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), 𝑝 (𝑋 ) = 𝑝 (𝑜 | 𝑠) 𝑝 (𝑠), (9) where the factorization of 𝑝 (𝑌 | 𝑋 ) follows directly from the dependency structure implied by the generative process. Building on this factorization, we now separate the generative components according to their causal origin. The joint process can be decomposed as: 𝑝 (𝑠, 𝑜, 𝑎, 𝑠 ′, 𝑟 ) = 𝑝 (𝑠) 𝑝 (𝑠 ′ | 𝑠, 𝑎) 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) 𝑝 (𝑜 | 𝑠) 𝜋 (𝑎 | 𝑜) . | {z } | {z } External (Environment-driven)

(10)

Internal (Agent-driven)

This decomposition follows directly from the source of control in the interaction process: • External (environment-driven) factors are those not determined by the agent: – State distribution 𝑝 (𝑠), – Transition dynamics 𝑝 (𝑠 ′ | 𝑠, 𝑎), – Reward mapping 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ). • Internal (agent-driven) factors arise from the agent’s internal perception and decision mechanisms: – Observation mapping 𝑝 (𝑜 | 𝑠), – Policy 𝜋 (𝑎 | 𝑜). While the MDP component aspect identifies where distributional shift originates in terms of the underlying generative components, it does not specify how such changes unfold over time during agent–environment interaction. In practice, shifts may occur abruptly or gradually, and the point at which the interaction transitions between regimes may or may not be clearly identifiable. To capture this temporal dimension, we introduce the boundary shift aspect, which characterizes how distributional changes are expressed with respect to a shifted-time boundary.

4.2

Explicit and Implicit Shifted-Time Boundaries

In supervised learning, distributional shift is defined relative to a clear training–testing split: a model is trained on one dataset and evaluated on another. In reinforcement learning, however, such separation is not always explicit. Learning may continue online, and environmental changes may occur without freezing the policy. To formalize distributional shift in reinforcement learning independently of the training protocol, we introduce the notion of a shifted-time boundary— a temporal point that separates two generative regimes of interaction. We distinguish these three cases below: (1) explicit-boundary shift (frozen-policy setting) In the explicit case, the boundary is defined by design. The agent is trained under a fixed generative regime up to time 𝑇 , after which learning is stopped and the policy is frozen. Evaluation then proceeds under either the same or a modified environment generative process. Thus, the separation between regimes is operationally defined by halting parameter updates, characterized as follows: • The agent is trained up to time 𝑇 . • Parameters are frozen. • Evaluation occurs under either identical (ID) or altered (OOD) environment generative processes. Because learning stops at the boundary, adaptation is not possible. Consequently, performance depends entirely on how well the frozen policy generalizes to the post-boundary regime. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:13

Fig. 4. Explicit shifted-time boundary: the model is frozen before evaluation. Distributional shift (DS) occurs if the environment generative process at test time differs from that during training.

Figure 4 visualizes this setting. The upper portion of the figure shows the training phase, where interaction occurs under a single environment generative process. The dashed horizontal line represents the shiftedtime boundary, after which the model is no longer updated. Below this boundary, evaluation proceeds using a saved policy. If the environment generative process at test time matches the one used during training, the evaluation remains in-distribution (ID). If it differs, the evaluation occurs under an out-of-distribution (OOD) regime. This setting corresponds to: 𝑓 (𝑥)train = 𝑓 (𝑥)test

⇒

In-Distribution (ID)

𝑓 (𝑥)train ≠ 𝑓 (𝑥)test

⇒

Out-of-Distribution (OOD).

The figure also emphasizes that no further learning occurs after the boundary; thus, performance depends entirely on the properties of the frozen policy. Under this explicit shifted boundary, distributional shift may arise in two distinct ways. First, if the test-time environment generative process differs from that during training (OOD case), an external distributional shift is necessarily present. Second, even in the ID case (𝑓 (𝑥)train = 𝑓 (𝑥)test ), distributional shift may still occur internally if components of the agent (e.g., observation mapping or policy representation) differ across regimes. (2) implicit-boundary shift (continuous learning setting) In contrast, the implicit case does not rely on freezing the policy. The agent continues to update its parameters during interaction. However, the environment’s generative process may change at some time 𝑡 + Δ, thereby creating a new interaction regime. The boundary is therefore defined structurally by a modification of the generative mechanism rather than by halting learning, characterized as follows: • Learning continues online. • No explicit training–evaluation phase separation. • The environment generative process changes at time 𝑡 + Δ. • Because learning continues across the boundary, performance depends on continual adaptation capability rather than purely on static generalization. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:14

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Fig. 5. Implicit shifted-time boundary: learning continues, but the environment generative process changes over time. Distributional shift (DS) corresponds to non-stationarity in the generative process.

Figure 5 illustrates this scenario. Unlike the explicit case, there is no frozen model and no operational separation between training and evaluation. Learning continues throughout the interaction. The dashed horizontal marker in the figure indicates the shifted-time boundary, defined not by halting updates but by a change in the environment generative process. Before this point, the generative process remains stable and the interaction is stationary. After the change, the underlying generator differs, creating a new regime within the same ongoing learning process. The figure highlights that parameter updates continue across the boundary, so the agent must adapt online to the new generative conditions. This setting corresponds to: 𝑓 (𝑥)𝑡 = 𝑓 (𝑥)𝑡 +Δ

⇒

Stationary

𝑓 (𝑥)𝑡 ≠ 𝑓 (𝑥)𝑡 +Δ

⇒

Non-Stationary.

Under the implicit shifted-time boundary, distributional shift arises whenever the underlying generative mechanism changes across time. If the environment generative process differs at 𝑡 + Δ, an external distributional shift occurs, manifested as non-stationarity in the environment-driven factors. Even if the environment generative process remains unchanged, internal distributional shift may still occur because learning continues, and policy updates or representation changes alter the agent-driven components of the generative process. Thus, in the implicit case, non-stationarity reflects a structural change in at least one component of the joint generative model. Sustained performance therefore depends on adaptation capability, rather than solely on static generalization. (3) Explicit-Implicit (hybrid) boundary shift The explicit–implicit (hybrid) boundary shift refers to a setting where a defined training–evaluation boundary coexists with implicit, ongoing distributional changes. Initially, the agent is trained under a given generative regime, and a policy is saved at a designated boundary, creating a clear separation between training and evaluation phases. At this stage, the shift occurring at the boundary is relatively predictable, as it corresponds to a controlled transition between regimes. However, after evaluation, learning continues. The agent keeps updating its parameters through interaction with the environment, making the boundary only partially meaningful. While it exists operationally, it does not prevent further changes in the underlying data-generating process. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:15

Fig. 6. Explicit-implicit shifted-time boundary: There is clear separation between training and evaluation phase, but the distributional shift happens during the training and/or evaluation phase.

As illustrated in Fig. 6, distributional shifts are not confined to the boundary itself. Although the transition from training to evaluation may introduce a predictable shift, additional shifts can emerge dynamically during both training and evaluation. These shifts are implicit, as they occur without explicit signaling and may arise at arbitrary time steps due to changes in the underlying generative process. Consequently, the agent must rely on both generalization and adaptation as complementary capabilities. This setting therefore represents a hybrid regime, in which an explicit training–evaluation boundary coexists with implicit shifts that may occur at any time. The agent must not only generalize across the boundary but also adapt online to ongoing distributional changes.

4.3

Formal Definition of Distributional Shift in RL

Building on the Internal and External shift decomposition under the MDP formulation (Subsection 4.1) and the notion of shifted-time boundaries (Subsection 4.2), we now provide a structural definition of distributional shift in reinforcement learning from a causal-origin perspective. Before giving the formal definition, Table 1 summarizes the five generative components used to locate the origin of shift across pre-shift and post-shift regimes. Table 1. Causal-origin components of distributional shift in reinforcement learning. Shift name State-distribution shift

Generative component 𝑝 (𝑠 )

Aspect External

Observation-mapping shift

𝑝 (𝑜 | 𝑠 )

Internal

Policy-induced shift

𝜋 (𝑎 | 𝑜 )

Internal

Transition-dynamics shift

𝑝 (𝑠 ′ | 𝑠, 𝑎)

External

Reward-mechanism shift

𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

External

Description The distribution of states encountered by the agent changes, for example due to different initial-state distributions, starting regions, or object placements at reset. The mapping from the underlying state to the agent’s observation changes, for example due to altered perception, sensor noise, camera configuration, visual distortion, or representation changes. The action distribution changes because the agent’s policy, decision rule, exploration behavior, or learned representation changes across regimes. The environment evolution changes, so the next-state distribution differs under the same state–action condition, for example due to changed layout, moving obstacles with new movement patterns, altered physical dynamics, wind, or currents. The reward generated from a transition changes, for example due to a modified task objective, reward function, cost structure, penalty rule, or success criterion.

Based on Table 1, the following definitions formalize each shift type in terms of a mismatch between the pre-shift and post-shift regimes. Definition 1 (Distributional shift in RL). Let pre and post denote the regimes before and after the shifted-time boundary, respectively. A distributional shift occurs whenever the joint generative model differs across these regimes: Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:16

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

𝑝 pre (𝑜, 𝑠, 𝑎, 𝑠 ′, 𝑟 ) ≠ 𝑝 post (𝑜, 𝑠, 𝑎, 𝑠 ′, 𝑟 ). This mismatch must arise from a change in at least one of: 𝑝 (𝑠),

𝑝 (𝑜 | 𝑠),

𝜋 (𝑎 | 𝑜),

𝑝 (𝑠 ′ | 𝑠, 𝑎),

𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ).

This definition is structural: it characterizes distributional shift as a change in the underlying generative interaction process, rather than through its empirical manifestations such as performance degradation, trajectory divergence, or dataset mismatch. The decomposition above provides a direct link to the causal origin of shift. Changes in 𝑝 (𝑠), 𝑝 (𝑠 ′ | 𝑠, 𝑎), or 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) correspond to external shifts arising from the environment, while changes in 𝑝 (𝑜 | 𝑠) or 𝜋 (𝑎 | 𝑜) correspond to internal shifts arising from the agent. This establishes a principled bridge between the formal definition of distributional shift and the Internal–External taxonomy introduced earlier. Definition 2 (External distributional shift). External shift corresponds to changes in environment-driven factors: 𝑝 pre (𝑠) ≠ 𝑝 post (𝑠) or 𝑝 pre (𝑠 ′ | 𝑠, 𝑎) ≠ 𝑝 post (𝑠 ′ | 𝑠, 𝑎) or 𝑝 pre (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) ≠ 𝑝 post (𝑟 | 𝑠, 𝑎, 𝑠 ′ ). Definition 3 (Internal distributional shift). Internal shift corresponds to changes in agent-driven factors: 𝑝 pre (𝑜 | 𝑠) ≠ 𝑝 post (𝑜 | 𝑠)

or 𝜋 pre (𝑎 | 𝑜) ≠ 𝜋 post (𝑎 | 𝑜).

Definitions 1–3 formalize distributional shift in reinforcement learning and specify (i) when a shift occurs, and (ii) whether the change originates from environment-driven (external) or agent-driven (internal) factors. In the next section, we extend this structural formulation to the decentralized multi-agent setting.

4.4

Distributional Shift Extension to Decentralized MARL

Extending the proposed distributional shift taxonomy to decentralized multi-agent reinforcement learning (MARL) requires particular care, because multiple learning processes evolve simultaneously. In fully decentralized MARL, each agent has access only to its own local observations and does not observe the internal states, policy parameters, or update dynamics of other agents. Consequently, even when the physical environment remains unchanged, the interaction process appears intrinsically non-stationary. This phenomenon is well documented in decentralized MARL settings (Emami et al. 2023; Li et al. 2022; Papoudakis, Christianos, et al. 2019). To situate decentralized non-stationarity within our causal-origin taxonomy, we distinguish two complementary viewpoints. (1) Environment generative viewpoint. From the environment generative perspective, the transition kernel and reward function depend on the joint action of all agents. The true underlying dynamics can therefore be written as: 𝑝 (𝑠 ′ | 𝑠, 𝑎, 𝑎 −1 ),

𝑝 (𝑟 | 𝑠, 𝑎, 𝑎 −1, 𝑠 ′ ),

where 𝑎 denotes the action of the considered agent and 𝑎 −1 the joint action of all other agents. In decentralized learning, the actions 𝑎 −1 are generated according to the policies 𝜋𝑎 −1 (𝑎 −1 | 𝑜 −1 ). Marginalizing over these policies yields the effective transition dynamics experienced by agent 𝑎: Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

𝑝 (𝑠 ′ | 𝑠, 𝑎, 𝜋𝑎 −1 ) =

∑︁

•

00:17

𝑝 (𝑠 ′ | 𝑠, 𝑎, 𝑎 −1 ) 𝜋𝑎 −1 (𝑎 −1 | 𝑜 −1 ),

𝑎 −1

and analogously for the reward distribution. Therefore, whenever the policies 𝜋𝑎 −1 evolve, the effective environment-side components 𝑝 (𝑠), 𝑝 (𝑠 ′ | 𝑠, 𝑎), 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) change as well. Even if the physical world remains fixed, the generative process governing an agent’s experience is altered through teammates’ learning updates. (2) Agent-centric viewpoint. From the perspective of a single agent, the evolving policies of other agents are not internally controllable factors. Agent 𝑎 cannot access or directly modify the parameter updates of agent 𝑗; it only observes their behavioral consequences through its own trajectory. Accordingly, within our causal-origin taxonomy, the behavioral evolution of other agents must be classified as an external source of shift. Although the cause originates in another learning agent, it manifests as a modification of the environment-driven generative factors governing the considered agent’s interaction. These viewpoints show that evolving teammate policies alter the effective environment-driven generative factors experienced by a considered agent, while remaining externally uncontrollable from its local perspective. We therefore formalize this effect as follows. Definition 4 (External shift induced by other agents in fully decentralized multi-agent setting). For an agent 𝑎, let 𝑎 −1 denote the set of all other agents in the system. The influence of their evolving policies 𝜋𝑎 −1 induces an external shift for agent 𝑎 whenever Agent perspective

Environment perspective

𝑝 (𝑠)

𝑝 (𝑠 | 𝜋𝑎 −1 ),

𝑝 (𝑠 ′ | 𝑠, 𝑎)

𝑝 (𝑠 ′ | 𝑠, 𝑎, 𝜋𝑎 −1 ),

𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′, 𝜋𝑎 −1 ).

That is, evolving teammate policies alter the effective environment-side generative factors experienced by agent 𝑎, and therefore constitute an external distributional shift under our taxonomy. Under this interpretation, policy updates of other agents are treated as modifications to the environment-driven generative regime from the perspective of agent 𝑎. Although the physical environment may remain unchanged, the effective transition dynamics, reachable state distribution, and reward structure experienced by agent 𝑎 are altered through the joint action dependence of the system. This clarification establishes that decentralized MARL non-stationarity corresponds to a formally identifiable instance of external generative change within our distributional-shift framework. In other words, the nonstationarity observed in decentralized MARL can be structurally characterized as external distributional shift according to the same causal-origin decomposition introduced earlier.

5

Implications for Performance and Evaluation

Our internal–external decomposition suggests that different evaluation failures correspond to changes in specific factors of the generative process introduced in Subsection 4.1. To illustrate these distinctions empirically, we conduct a controlled study in a gridworld environment using the Deep Q-Network (DQN) algorithm (Mnih et al. 2013). This study evaluates five designed distributional shifts: three affecting the external components (𝑝 (𝑠), 𝑝 (𝑠 ′ | 𝑠, 𝑎), and 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )), and two affecting the internal components (𝑝 (𝑜 | 𝑠) and 𝜋 (𝑎 | 𝑜)). The purpose is not to Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:18

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

introduce a new algorithm, but to provide empirical evidence that perturbations in different causal-origin factors produce distinct and diagnosable degradation patterns. To support this analysis, we present the gridworld environment, the observation model, explicit and implicit training–evaluation boundaries, perturbation mechanisms for each shift type, DQN hyperparameters, and the evaluation metrics used throughout.

5.1

Environment Layout

The base environment is a deterministic gridworld with the following properties: • Grid size: 𝑁 × 𝑁 (we use 30 × 30). • Obstacles: A fixed set of 10 static obstacles. • Start-state distribution 𝑝 train (𝑠 0 ): During training, agents begin in the bottom region of the map. For experiments involving an external 𝑝 (𝑠) shift, the agent’s starting location is moved to the right region. • Goal: The goal is fixed at position (6, 12) during training. Under external 𝑝 (𝑠 ′ | 𝑠, 𝑎) shifts, the goal location is changed. • Transition model: The environment uses deterministic, grid-based movement. At each step the agent selects one of five discrete actions: up, down, left, right, or stay. The agent moves exactly one cell in the chosen direction unless that move would cross a boundary or collide with an obstacle, in which case the agent remains in its current cell. Selecting stay keeps the agent in its current position regardless. • Observation model 𝑝 (𝑜 | 𝑠): The agent receives a local field of view (FOV) of size 𝐹 × 𝐹 centered on its current position. Observations are encoded in three channels: {walls, obstacles, goal}. The DQN agent is trained on this environment until convergence, producing the experience distribution used during training, denoted 𝑝 train (𝑜, 𝑠, 𝑎, 𝑠 ′ ). Figure 7 shows the layout used during training, where represents static obstacles; ▲ marks the goal position; • indicates the agent; the light–green highlighted region shows the agent’s FOV.

Fig. 7. Training layout of the gridworld environment. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

5.2

•

00:19

Shifted-time Boundaries

As discussed in Subsection 4.2, distributional shift can be evaluated under explicit, implicit, or hybrid shifted-time boundaries. In this experimental study, we implement only the explicit and implicit settings. The hybrid setting is not evaluated separately because our goal is to isolate the minimal effect of each causal-origin factor. That is, if a single internal or external shift is sufficient to degrade performance, then a hybrid setting that combines multiple such changes is expected to inherit this source of degradation. We therefore focus on controlled single-factor shifts rather than mixed shift configurations. • explicit-boundary (frozen policy). The agent is trained exclusively on the base environment. After convergence, the DQN policy is frozen and evaluated in the shifted environments without any further updates. This setting assesses the agent’s ability to generalize to unseen distributional conditions under a strict train–then–deploy protocol. • implicit-boundary (environment changes during learning). Training proceeds in two sequential phases: (1) Phase 1 (Base MDP): The agent is trained for the first 𝐸 1 episodes on the base generative model. (2) Phase 2 (Shifted MDP): At episode 𝐸 1 + 1, a distributional shift is introduced by modifying exactly one of the five causal-origin factors described in Subsection 4.3. The agent then continues training for an additional 𝐸 2 episodes under the shifted generative process. Results under the implicit-boundary setting measure the agent’s ability to adapt online to a shifted environment.

5.3

External Shift Design

5.3.1 State-distribution shift (𝑝 (𝑠)). This shift modifies only the initial-state distribution: • The new initial-state distribution 𝑝 test (𝑠 0 ) places the agent in a right-side region of the grid that is never visited during training. • Obstacle positions remain identical to the base environment. • The transition model and observation mapping are unchanged. Figure 8 shows the shifted start region. 5.3.2 Transition-dynamics shift (𝑝 (𝑠 ′ | 𝑠, 𝑎)). This shift modifies the environment transition dynamics while keeping the initial-state distribution and observation model unchanged. We apply two modifications: (1) Obstacle relocation. All obstacle coordinates are moved to new positions that differ from the training layout. (2) Target relocation. The goal position is moved to a different region of the grid. Figure 9 illustrates an example of the shifted environment dynamics. 5.3.3 Reward-mechanism shift (𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )). This shift modifies only the reward function while keeping the initial-state distribution, transition dynamics, and observation model unchanged. Before shift, the agent is optimized under a sparse reward: ( +𝑅𝑔 , if 𝑠𝑡 +1 reaches the goal, 𝑅train (𝑠𝑡 , 𝑎𝑡 , 𝑠𝑡 +1 ) = (11) −1, otherwise. When the shift occurs, we introduce a deadline-based reward constraint without altering the environment layout or dynamics. Let 𝑇max denote the maximum allowed number of steps (e.g., 𝑇max = 26). The shifted reward is defined as: Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:20

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Fig. 8. Shifted initial-state region for External Shift of state distribution (𝑝 (𝑠)). The agent starts in the right area, which does not overlap with its training start distribution.

𝑅test (𝑠𝑡 , 𝑎𝑡 , 𝑠𝑡 +1, 𝑡) =

 +𝑅𝑔 ,      −𝜆,       −1, 

if 𝑠𝑡 +1 reaches the goal and 𝑡 ≤ 𝑇max, if the episode terminates without success or if success occurs at 𝑡 > 𝑇max, otherwise.

(12)

Here, 𝜆 > 0 denotes a large penalty applied when the agent fails to reach the goal within the allowed step budget. The shift therefore constitutes a pure reward-side external distributional shift, in which the optimization objective changes while the generative mechanism of state transitions remains identical.

5.4

Internal Shift Design

5.4.1 Observation-mapping shift (𝑝 (𝑜 | 𝑠)). This shift modifies only the observation mapping 𝑝 (𝑜 | 𝑠). In our implementation, the agent’s local field-of-view (FOV) tensor is perturbed by a cyclic spatial shift. Let 𝑜 ∈ R𝐹 ×𝐹 ×𝐶 denote the original observation. The shifted observation 𝑜˜ is obtained by rolling the tensor by 𝑛 cells along both spatial dimensions:
𝑜˜ = roll roll(𝑜, 𝑛, axis = 0), 𝑛, axis = 1 . A positive value of 𝑛 shifts the observation down and right, whereas a negative 𝑛 shifts it up and left. The roll operator wraps values around at the boundaries, producing a synthetic perturbation of the observation channel while leaving the underlying MDP unchanged. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:21

Fig. 9. Example environment modification for External Shift of environment dynamics (𝑝 (𝑠 ′ | 𝑠, 𝑎)), where obstacles and/or the goal are relocated.

As an Illustrative example, consider a single-channel 3 × 3 observation: 1  𝑜 = 4 7 

2 5 8

3 6 . 9

Applying a shift of 𝑛 = 1 first rolls the matrix downward: 7  roll(𝑜, 1, 0) = 1 4 

8 2 5

9 3 , 6

and then rolls it rightward: 
9 𝑜˜ = roll roll(𝑜, 1, 0), 1, 1 = 3 6 

7 1 4

8 2 . 5 Thus the FOV content is shifted one cell down and one cell right, with entries wrapping around at the edges. 5.4.2 Policy-induced shift (𝜋 (𝑎 | 𝑜)). In this setting, the environment and observation model remain unchanged. Internal Shift B perturbs only the policy computation by applying a synthetic quantization distortion to intermediate network activations. This construction is motivated by prior work on learned and hardware-friendly quantization schemes, particularly the learned step-size quantization framework of Esser et al. (Esser et al. 2020) and the integer-arithmetic-only inference formulation introduced by Jacob et al. (Jacob et al. 2018). Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:22

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Let 𝑥 denote an activation tensor produced by the policy network. We define a fake-quantization operator that aggressively compresses 𝑥 into a low-bit integer grid and then maps it back to real values: 𝑥˜ = 𝑄 (𝑥; 𝑛 bits, 𝑐), where 𝑛 bits denotes the quantization resolution (e.g., 4 or 3 bits), and 𝑐 is a clipping threshold that restricts activations to the range [−𝑐, 𝑐] prior to scaling. The operator 𝑄 is defined as:
𝑞 round clip(𝑥, −𝑐, 𝑐) max 𝑐 , 𝑞 max = 2 𝑛−1 − 1. 𝑄 (𝑥; 𝑛, 𝑐) = 𝑞 max 𝑐

Thus, activations outside [−𝑐, 𝑐] are clipped, while values inside the range are linearly scaled to the integer grid {−𝑞 max, . . . , +𝑞 max }, rounded to the nearest integer, and rescaled back to real values. Smaller 𝑛 bits and smaller clipping thresholds 𝑐 produce stronger distortions of the internal representation, while leaving the underlying MDP and observations unchanged. Consequently, this shift modifies only the effective policy mapping 𝜋 (𝑎 | 𝑜). Illustration. We illustrate how limited-precision integer inference can alter the action selected by a DQN policy. Consider a single-state input 𝑥 = 0.3478 and two actions 𝑎 1, 𝑎 2 with scalar Q-values parameterized as 𝑄 (𝑎𝑖 ) = 𝑥𝑤𝑖 + 𝑏𝑖 . We use 𝑤 1 = −0.7486,

𝑏 1 = 0.1359,

𝑤 2 = 0.2063,

𝑏 2 = −0.4605.

Using these initial weight values, we evaluate the policy under two settings: • FP32 policy (PC simulation). Using full-precision floating-point arithmetic, the Q-values are: 𝑄 FP32 (𝑎 1 ) = 0.3478 · (−0.7486) + 0.1359 ≈ −0.1245, 𝑄 FP32 (𝑎 2 ) = 0.3478 · 0.2063 − 0.4605 ≈ −0.3888. Hence 𝑄 FP32 (𝑎 1 ) > 𝑄 FP32 (𝑎 2 ), and the FP32 policy selects: 𝜋FP32 (𝑥) = arg max 𝑄 FP32 (𝑎) = 𝑎 1 . 𝑎

• INT8 quantized inference (embedded). Assume a symmetric INT8 quantization scheme with scale factor 127: 𝑣𝑞 = ⌊𝑣 · 127⌉. Quantizing the inputs yields: 𝑥𝑞 = ⌊0.3478 · 127⌉ = 44,

𝑤 1𝑞 = ⌊−0.7486 · 127⌉ = −95,

𝑏 1𝑞 = ⌊0.1359 · 127⌉ = 17,

𝑤 2𝑞 = ⌊0.2063 · 127⌉ = 26,

𝑏 2𝑞 = ⌊−0.4605 · 127⌉ = −58.

All internal computations are performed using integer arithmetic: 𝑧 1𝑞 = 𝑥𝑞 𝑤 1𝑞 + 𝑏 1𝑞 = 44 · (−95) + 17 = −4163,

𝑧 2𝑞 = 𝑥𝑞 𝑤 2𝑞 + 𝑏 2𝑞 = 44 · 26 − 58 = 1086.

Dequantizing (by dividing by 1272 ) yields: 𝑧 1𝑞 𝑧 2𝑞 ≈ −0.2581, 𝑄 INT8 (𝑎 2 ) = ≈ 0.0673. 2 127 1272 Here 𝑄 INT8 (𝑎 2 ) > 𝑄 INT8 (𝑎 1 ), so the quantized policy selects: 𝑄 INT8 (𝑎 1 ) =

𝜋INT8 (𝑥) = arg max 𝑄 INT8 (𝑎) = 𝑎 2 . 𝑎

This example demonstrates that, even with a fixed environment and fixed network parameters, transitioning from full-precision FP32 inference (typical on PCs) to low-precision INT8 inference (typical on embedded hardware) can induce a precision-based internal shift at evaluation, altering the selected action for the same observation. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:23

(a) Training performance.

(b) No-shift evaluation.

(c) State-distribution shift 𝑝 (𝑠) evaluation.

(d) Transition-dynamics shift 𝑝 (𝑠 ′ | 𝑠, 𝑎) evaluation.

(e) Reward-mechanism shift 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) evaluation.

(f) Observation-mapping shift 𝑝 (𝑜 | 𝑠) evaluation.

(g) Policy-induced shift 𝜋 (𝑎 | 𝑜) evaluation. Fig. 10. Explicit-boundary performance across training, no-shift evaluation, and causal-origin shift conditions. In each subfigure, the left plot reports mean reward performance and the right plot reports mean successful episode performance.

5.5

Performance Degradation Results

This section reports the effect of each causal-origin shift on agent performance. We present the results separately for the explicit-boundary setting, where the policy is frozen after training, and the implicit-boundary setting, where the agent continues learning after the shift occurs. 5.5.1 Explicit-Boundary Setting. This subsection reports evaluation curves for the explicit-boundary setting (see Subsection 5.2). For each condition, we report two metrics: mean reward performance and mean successful episode performance. Training is conducted over 150 episodes, while evaluation is performed over 10 episodes. Each experimental result is computed from a single independent run. Since this paper proposes a problem formulation rather than a new learning method, the experiments are intended as a minimum empirical demonstration. Figure 10 summarizes the explicit-boundary results. During training on the base environment, the DQN agent learns a reliable policy: the reward stabilizes after the early learning episodes, and the success rate approaches 1.0. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:24

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

(a) No-shift performance.

(b) State-distribution shift 𝑝 (𝑠) performance.

(c) Transition-dynamics shift 𝑝 (𝑠 ′ | 𝑠, 𝑎) performance.

(d) Reward-mechanism shift 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ) performance.

(e) Observation-mapping shift 𝑝 (𝑜 | 𝑠) performance.

(f) Policy-induced shift 𝜋 (𝑎 | 𝑜) performance.

Fig. 11. Implicit-boundary performance across no-shift and causal-origin shift conditions. In each subfigure, the left plot reports mean reward performance and the right plot reports mean successful episode performance.

Under the no-shift evaluation condition, the frozen policy maintains stable reward and 100% success, showing that the learned policy performs reliably when the evaluation environment matches the training environment. In contrast, all shifted conditions lead to performance degradation. Under the state-distribution shift 𝑝 (𝑠), the initial-state distribution moves to an unseen region, causing the reward to collapse and the success rate to fall to 0%. Under the transition-dynamics shift 𝑝 (𝑠 ′ | 𝑠, 𝑎), the environment evolution differs from training, and the frozen policy fails to reach the target. Under the reward-mechanism shift 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), the altered reward structure also produces a collapse in both reward and success performance. Similarly, the observation-mapping shift 𝑝 (𝑜 | 𝑠) breaks the learned perception–action mapping, while the policy-induced shift 𝜋 (𝑎 | 𝑜) introduces policy-side distortion that prevents the agent from executing the learned behavior successfully. The explicit-boundary results show that a policy that performs well under the base generative process fails to generalize when a single causal-origin component is changed. 5.5.2 Implicit-Boundary Setting. We now report evaluation curves for the implicit-boundary, or continual-learning, setting. For each condition, we report two metrics: mean reward performance and mean successful episode performance. All conditions are evaluated over 450 episodes, with shifts introduced every 150 episodes. As in the previous explicit-boundary setting, each experimental result is computed from a single independent run. Figure 11 summarizes the implicit-boundary results. In the no-shift setting, the agent continues learning on the same base MDP for all 450 episodes. The reward converges to a high value and the success rate remains consistently strong, showing that the policy remains effective when the generative process does not change. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:25

Under shifted conditions, however, the agent exhibits different degradation and recovery patterns depending on which causal-origin component is modified. Under the state-distribution shift 𝑝 (𝑠), performance drops after the shift points, with partial recovery after the first shift. Under the transition-dynamics shift 𝑝 (𝑠 ′ | 𝑠, 𝑎), the reward and success rate fluctuate sharply after each shift, indicating unstable adaptation when the environment evolution differs from the previous regime. Under the reward-mechanism shift 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), the agent initially performs well but collapses after the reward structure changes, showing that altering the reward mechanism alone can disrupt continual learning even when other aspects of the environment remain unchanged. The internal shifts also produce severe degradation. Under the observation-mapping shift 𝑝 (𝑜 | 𝑠), the modified observation process breaks the learned perception–action mapping, causing reward and success rate to fall to very low levels with little recovery. Under the policy-induced shift 𝜋 (𝑎 | 𝑜), quantization-based policy-side distortion similarly causes an immediate collapse in both metrics and prevents the agent from regaining its previous competence. This implicit-boundary results show that every causal-origin shift, whether external or internal, produces a noticeable degradation in reward and success-rate performance. The distinct degradation and recovery profiles across shift types support the proposed taxonomy by showing that different generative components lead to different forms of adaptation failure.

6

Evaluation Framework for Agent Adaptation under Distributional Shift

In reinforcement learning, agent performance is typically evaluated using aggregate metrics such as cumulative return or success rate over entire episodes or training horizons (Papadopoulos et al. 2025; Papoudakis and Christianos 2021). While these metrics capture overall performance, they fail to characterize how agents respond to distributional shift during interaction. In particular, they do not quantify the direct impact of the shift, nor the agent’s ability to recover from the resulting performance degradation. To address this limitation, we complement our taxonomy with an evaluation framework that explicitly quantifies agent response to changes in the generative interaction process. Before defining the metrics, we first specify the user-defined evaluation parameters used by the framework.

6.1

User-defined Variables

The user defines the pre-shift window Wpre , the post-shift window Wpost , and the recovery threshold 𝜂 ∈ [0, 1]. The pre-shift window Wpre contains performance values, such as return or success rate, observed before a shift event, while the post-shift window Wpost contains performance values observed after that event. These two windows are illustrated in Fig. 12a. The recovery threshold 𝜂 specifies the fraction of pre-shift performance that must be regained for the agent to be considered recovered, as illustrated in Fig. 12d Since these windows must be anchored around one or more shift events, the identification of shift events becomes part of the evaluation procedure. We therefore distinguish between controlled and uncontrolled shift settings as follows: • Controlled shift setting. In controlled experiments, the shift event is specified by the experimenter at a known interaction time. In this case, the shift time 𝑡 shift is given by the experimental intervention. The pre-shift window Wpre is defined immediately before 𝑡 shift , and the post-shift window Wpost is defined immediately after 𝑡 shift . • Uncontrolled shift setting. In uncontrolled or real-world settings, the exact shift events may not be directly observable, because changes in the interaction process may occur gradually, unexpectedly, repeatedly, or without explicit annotation. In this case, shift events are estimated from the observed performance trajectory. In addition to the pre-shift window Wpre , the post-shift window Wpost , and the recovery threshold 𝜂, the user defines a degradation-detection threshold 𝛿 detect ∈ [0, 1], which specifies the minimum relative Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:26

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

performance drop required to identify a shift event. For example, if 𝛿 detect = 0.2, then a shift event is detected whenever the current performance falls below 80% of the preceding pre-shift mean. The set of detected shift events is defined as the set of all candidate times 𝑡 for which the current performance falls below the degradation threshold computed from the preceding pre-shift window:  Tshift = 𝑡 Δcurrent (𝑡) < (1 − 𝛿 detect ) · Mean(Wpre (𝑡)) . (13) Here, Tshift denotes the set of detected shift times, Δcurrent (𝑡) denotes the observed performance value at candidate time 𝑡, and Wpre (𝑡) denotes the preceding pre-shift window used to compute the baseline before (𝑘 ) 𝑡. Each element 𝑡 ∈ Tshift is treated as one detected shift event and may be indexed as 𝑡 shift for the 𝑘-th (𝑘 ) detected event. For each 𝑡 shift , the corresponding pre-shift and post-shift windows are then used to compute the evaluation metrics. After identifying 𝑡 shift , either from a controlled intervention or from performance-based detection, the following metrics are computed to quantify the immediate effect of the shift, the severity of the resulting degradation, and the agent’s recovery behavior.

6.2

Evaluation Metrics

After the shift event has been identified, either from a controlled intervention or from performance-based detection, the corresponding pre-shift and post-shift windows are used to compute the evaluation metrics. These metrics are designed to capture complementary aspects of the agent’s response to distributional shift: the immediate performance change after the shift, the worst degradation observed during the post-shift phase, the speed of recovery, the quality of post-shift performance, and the overall efficiency of adaptation. (1) Immediate shift impact. We quantify the direct effect of the shift as: Δshift = Mean(Wpre ) − Wpost (𝑡 shift + 1),

(14)

which captures the instantaneous performance change immediately after the shift, before adaptation dynamics take effect. As illustrated in Fig. 12b, this reflects the inherent difficulty introduced by the new regime. (2) Worst-case degradation. We measure the largest observed degradation after the shift: Δdrop = Mean(Wpre ) − min(Wpost ).

(15)

As shown in Fig. 12c, this captures the worst performance reached during the post-shift phase, reflecting both the shift effect and the agent’s adaptation behavior. (3) Recovery speed. We measure how quickly the agent recovers after the shift via: 𝑡 min = arg min(Wpost ), 

(16)

𝑇rec = min min 𝑡 − 𝑡 shift Wpost (𝑡) ≥ 𝜂 · Mean(Wpre ), 𝑡 ≥ 𝑡 min , |Wpost | , (17) where 𝜂 defines the fraction of pre-shift performance required to consider recovery, and |Wpost | denotes the length of the post-shift window. As illustrated in Fig. 12d, this represents the number of steps from the shift point until performance first reaches the recovery threshold after passing through min(Wpost ). Lower values indicate faster recovery. (4) Recovery quality. To normalize post-shift performance relative to the pre-shift regime, we define: Γpost =

Mean(Wpost ) . Mean(Wpre )

(18)

As shown in Fig. 12e, this measures the average post-shift performance as a fraction of the pre-shift baseline. Values close to 1 indicate that the agent maintains or recovers most of its pre-shift capability, while lower values indicate sustained degradation in the post-shift phase. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

(a) Pre- and post-shift window construction (Wpre and Wpost ).

(b) Immediate shift impact (Δshift ).

(c) Worst-case degradation (Δdrop ).

(d) Recovery speed (𝑇rec ).

(e) Recovery quality (Γpost ).

(f) Adaptation efficiency (𝐴adapt ).

•

00:27

Fig. 12. Evaluation framework for agent adaptation under distributional shift.

(5) Adaptation efficiency. We jointly capture recovery quality and recovery speed via: 𝐴adapt =

Γpost , 1 + 𝑇rec /|Wpost |

(19)

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:28

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

where |Wpost | denotes the length of the post-shift window. As illustrated in Fig. 12f, this reflects how effectively the agent balances retaining a high fraction of its pre-shift performance while recovering quickly after the shift. Higher values indicate more efficient adaptation. Example Application of the Metrics To illustrate how the proposed metrics are computed, we apply them to the controlled shift setting discussed in Sec. 5. In this setting, the shift events are introduced by experimental interventions, so the corresponding shift times are known and can be used directly to define the pre-shift and post-shift windows. Building on the result in Fig. 11c, we further compute the agent adaptation metrics, with the results reported in Table 2. In this example, the evaluation variables are defined as follows: the pre-shift window Wpre covers 10% of the shift period, the post-shift window Wpost covers 70% of the shift period, and the recovery threshold is set to 𝜂 = 0.9, corresponding to 90% of the pre-shift performance. Table 2. Evaluation results of the proposed adaptation metrics using average reward as the performance signal.

Method DQN

Shift 150 300 Avg.

Δshift 0.120 0.193 0.157

Δdrop 0.920 0.993 0.957

𝑇rec Γpost 𝐴adapt 76.0 0.391 0.227 105.0 0.073 0.037 90.5 0.232 0.132

The results show that the second shift (𝑡 = 300) has a stronger negative effect than the first shift (𝑡 = 150). The immediate impact increases from Δshift = 0.120 to 0.193, and the worst-case degradation also increases from Δdrop = 0.920 to 0.993, indicating a deeper post-shift performance collapse. Recovery is also weaker after the second shift, with a longer recovery time (𝑇rec = 105 compared to 76) and a much lower recovery quality (Γpost = 0.073 compared to 0.391). This is reflected in the adaptation efficiency, which decreases from 𝐴adapt = 0.227 to 0.037. Overall, the averaged results indicate that the DQN agent experiences substantial degradation under the controlled shifts and shows limited adaptation, especially after the second shift. This example demonstrates that shifts with similar structural characteristics can lead to different adaptation outcomes. By reporting both per-shift and aggregated metrics, the proposed framework enables a more nuanced comparison of methods, distinguishing between robustness to shift, recovery dynamics, and sustained post-shift performance.

7

Mapping Existing Methods into the Proposed Taxonomy

Our taxonomy provides a basis for categorizing prior reinforcement learning methods according to which component of the generative interaction process they affect. Rather than grouping methods by research domain, we reinterpret them in terms of the causal origin of the shift they address. This mapping serves as an analytical lens and does not imply that the original works adopt this formulation. Table 3 summarizes representative contributions and classifies them based on (i) whether the shift is environmentdriven (external) or agent-driven (internal), (ii) which probabilistic component of the generative process is affected (e.g., 𝑝 (𝑠), 𝑝 (𝑠 ′ | 𝑠, 𝑎), 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), 𝑝 (𝑜 | 𝑠), or 𝜋 (𝑎 | 𝑜)), and (iii) whether the mismatch arises across an explicit training–evaluation boundary or implicitly during continual interaction.

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:29

Table 3. Mapping existing related methods into the proposed taxonomy. Method

Work domain

Short description

Shift type

Target factors

Boundary type

(Chen 2024)

Distributional shift in off-policy RL

Internal

𝜋 (𝑎 | 𝑜 )

Implicit

(Luo et al. 2025)

Distributional shift in offline RL

External, Internal

𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝜋 (𝑎 | 𝑜 )

Explicit

(Herremans et al. 2024)

Robust model-based RL under distributional shift

External

𝑝 (𝑠 ′ | 𝑠, 𝑎)

Explicit

(Altmann et al. 2023)

Distributional shift in fully observable RL

External

𝑝 (𝑠 ′ | 𝑠, 𝑎)

Explicit

(Wang et al. 2025)

Distributional shift in offline RL

Internal

𝜋 (𝑎 | 𝑜 )

Implicit

(Hickman et al. 2025)

Safe RL under distributional shift

Policy updates continuously shift the agent’s visitation distribution during off-policy replaybased learning, causing the replay-buffer training distribution to differ from the distribution visited by the post-update policy. The method predicts the post-update visitation distribution and reweights replay samples accordingly to improve Q-training efficiency. Divides distribution shift in offline learning into model bias and policy mismatch. Model bias reflects the mismatch between the learned dynamics model and the true environment, which fits our external shift category. Policy mismatch reflects the mismatch between the datasetgenerating policy and the learned policy, and can fit our internal shift category under the assumption that both policies belong to the same agent across different learning/deployment stages. The proposed solution introduces a shift-aware reward that corrects both mismatches using modelbias and policy-mismatch adjustments during policy optimization. A model-based RL policy trained with a learned dynamics model in a reference/training MDP often performs poorly when evaluated in nearly identical MDPs with small environment changes, such as different mass, friction, or action noise; the paper improves robustness by adding an adversarial auxiliary model that learns pessimistic worst-case transitions. An RL policy trained with full observations may overfit to environment-specific layout details, causing poor zero-shot generalization in unseen shifted layouts. The paper proposes CROP, a compact observation reshaping method that keeps only task-relevant, agent-centered information to improve robustness. The learned policy may induce a state-action distribution different from the fixed behavior dataset, causing extrapolation error for poorly supported actions. The paper models this via latent dataset distributions from different policies or training phases and learns invariant representations across their gaps; assuming both policies belong to the same agent, this corresponds to policy shift across learning/deployment stages. External covariate shift in safe RL occurs when the unsafe-state distribution changes between offline training and online deployment, causing the cost/safety function learned from offline data to become unreliable under shifted unsafe regions and outliers. The solution combines offline initialization with online adaptation and replaces Gaussian Processes with Student-t Processes to obtain more robust cost and reward prediction with uncertainty estimation during deployment.

External

𝑝 (𝑠 )

Hybrid

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:30

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Method

Work domain

Short description

Shift type

Target factors

Boundary type

(Lee et al. 2020)

Distributional shift in offline-online RL

Internal

𝜋 (𝑎 | 𝑜 )

Hybrid

(W. Jiang et al. 2024)

Distributional shift in RL-based portfolio management

External

𝑝 (𝑠 ) , 𝑝 (𝑠 ′ | 𝑠, 𝑎)

Hybrid

(Slaoui et al. 2020)

Visual domain randomization

External

𝑝 (𝑠 )

Explicit

(Muratore et al. 2021)

Domain randomization

External

𝑝 (𝑠 ′ | 𝑠, 𝑎)

Hybrid

(Tiboni et al. 2023)

Domain randomization

Offline-to-online RL fine-tuning setting, where an agent first trained from a fixed offline dataset becomes unstable during online interaction because the online policy induces state-action samples different from the offline behavior-data distribution. BRED mitigates this shift using separate offline and online replay buffers with gradually adjusted balanced sampling, plus ensemble distillation to stabilize policy updates using ensemble Q-value estimates. RL portfolio policies trained on historical data may degrade in future markets because market states and dynamics naturally evolve over time, independent of the agent’s portfolio actions under the zero-market-impact assumption. The paper addresses this external market shift by pretraining a contrastive representation extractor on historical financial sequences, then training the RL policy directly in this representation space to improve deployment-time generalization without online updating. Environment visual variations such as color or texture changes alter the visual state distribution while leaving the underlying MDP dynamics and rewards unchanged; feature-level regularization is used during training to learn domain-invariant state representations and improve generalization to visually different domains. Sim-to-real policies can fail when the randomized simulator used during training does not match the real robot dynamics, such as differences in mass, friction coefficients, restitution coefficients, or time delays. The paper proposes NPDR, which uses a few real-world rollouts to infer a posterior distribution over these simulator parameters and then trains the policy using this more targeted domain-randomization distribution. Sim-to-real RL policies can fail because manually chosen or uniform domain randomization may not match the real robot dynamics, especially when physical parameters such as mass, friction, contact properties, or controller gains are uncertain. DROPO solves this by using offline realworld trajectories to estimate a likelihood-based dynamics randomization distribution, then trains the policy in simulation using this optimized distribution for zero-shot real-world transfer. RL policies trained in simulation often perform poorly in the real world because of the reality gap, where the simulator and real system have mismatching dynamics parameters such as mass, leg length, or added weight. The paper proposes combining domain randomization and domain adaptation: first train the agent in simulation with randomized uncertain parameters, then finetune the trained policy using a small number of real-world training episodes to improve realworld performance..

External

𝑝 (𝑠 ′ | 𝑠, 𝑎)

Hybrid

External

𝑝 (𝑠 ′ | 𝑠, 𝑎)

Hybrid

(Shakerimov Domain randomizaet al. 2023) tion

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:31

Method

Work domain

Short description

Shift type

Target factors

Boundary type

(Padakandla et al. 2020)

Non-stationary

External

𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Implicit

(Z. Liu et al. 2024)

Non-stationary

External

𝑝 (𝑠 ) , 𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Implicit

(Ajay et al. 2022)

Meta-RL

External

𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Hybrid

(T. Xu et al. 2024)

Meta-RL

External

𝑝 (𝑠 ) , 𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Explicit

(Finn et al. 2017)

Meta-RL

Non-stationary RL environments can switch between different environment models during ongoing interaction, causing the transition dynamics and/or reward function to change over time. This makes standard Q-learning suboptimal because it keeps updating a single Q-table even when samples come from different contexts. The proposed solution, Context Q-learning, detects changepoints from observed experience tuples (𝑠𝑡 , 𝑟𝑡 , 𝑠𝑡 +1 ) using ODCP, then maintains and updates separate Q-tables for different environment contexts so that previously learned policies can be reused when a context reappears. The paper tackles deep reinforcement learning in non-stationary environments with unknown change points. It proposes BADA, which detects changes by comparing behavior embeddings from trajectory data using Wasserstein distance and a permutation test, then adapts by regularizing the policy to move away from previous optimal behavior. The experiments simulate changes in lighting/wall texture, medikit texture, enemy number, and defended map/objective. Meta-RL agents trained on a task distribution can adapt quickly to new tasks from the same distribution, but their exploration and adaptation strategy may fail when meta-test tasks come from shifted reward-function or transition-dynamics distributions. The paper proposes DiAMetR, which trains a population of meta-policies with different distributional-robustness levels using imagined shifted task distributions, then selects the most suitable meta-policy at test time to balance fast adaptation and robustness under taskdistribution shift. Meta-RL agents often fail to generalize when meta-test tasks come from shifted/OOD task distributions rather than the training task distribution; PSBL addresses this by training a transformer-based network, LILTrans, to infer an approximate posterior predictive distribution of the optimal policy from interaction history, so that during meta-testing the network parameters remain frozen while the agent adapts through lifelong in-context learning by conditioning on a rolling window of recent experience and sampling actions from the inferred posterior policy. Fast adaptation to new held-out RL tasks sampled from a task distribution, where the policy must adapt under limited task-specific experience because a new task may not provide enough trajectories for standard training from scratch. The solution, MAML-RL, meta-learns an initial policy parameterization such that one or a few policy-gradient updates using the available trajectories from the new task produce an adapted policy with high return; in the experiments, the task variation mainly corresponds to different goals, target velocities, or movement directions.

External

𝑝 (𝑠 ) , 𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Explicit

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:32

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Method

Work domain

Short description

Shift type

Target factors

Boundary type

(Duan et al. 2016)

Meta-RL

RL2 tackles the high sample complexity of deep RL when facing a new but related MDP, where an agent must adapt using only limited interaction experience. It proposes training a recurrent policy across a distribution of MDPs, where the network parameters are updated only across tasks/MDPs during meta-training, while within one MDP the parameters remain fixed and fast adaptation occurs through the RNN hidden state using past observations/states, actions, rewards, and termination flags.

External

𝑝 (𝑠 ) , 𝑝 (𝑠 ′ | 𝑠, 𝑎) , 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ )

Explicit

This mapping reveals several structural patterns. Most existing approaches primarily address external shifts, especially changes in the environment-side generative process. Domain randomization methods mainly broaden, infer, or adapt the training distribution over environment dynamics, most notably 𝑝 (𝑠 ′ | 𝑠, 𝑎), and in some visualdomain settings also the state distribution 𝑝 (𝑠). Non-stationary RL methods address temporal changes during ongoing interaction, typically in 𝑝 (𝑠 ′ | 𝑠, 𝑎) and sometimes in 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), with some works also considering changes in state or observation-level environmental conditions. Meta-RL methods generally assume task-level variation across an explicit training–testing boundary, where tasks may differ in initial-state distributions, transition dynamics, and/or reward functions. In contrast, comparatively fewer works directly target internal shifts. These works mainly concern policyinduced changes in the state–action visitation distribution, such as the mismatch between replay-buffer data and the post-update policy, or between an offline behavior policy and the learned online policy. Accordingly, offline-to-online mismatch methods and latent-distribution correction approaches are better understood as addressing shifts associated with 𝜋 (𝑎 | 𝑜) and the induced visitation distribution, rather than changes in the environment itself. This reclassification clarifies that much of the literature implicitly assumes a particular source of shift, even when the source is not explicitly formulated in terms of the RL generative process. The proposed taxonomy therefore provides a unified diagnostic framework for identifying whether a method regularizes, adapts, expands, or corrects specific components such as 𝑝 (𝑠), 𝑝 (𝑠 ′ | 𝑠, 𝑎), 𝑝 (𝑟 | 𝑠, 𝑎, 𝑠 ′ ), or 𝜋 (𝑎 | 𝑜).

8

Discussion

Distributional shift in reinforcement learning cannot be fully characterized only from observable effects, such as performance degradation, trajectory divergence, or changes in state visitation. Similar empirical effects may be produced by different sources of change in the interaction process, while different causal factors may also lead to distinct degradation patterns over time. For example, a decrease in return may be caused by a change in the state distribution, transition dynamics, reward function, observation process, or policy behavior, but each source may affect the agent differently in terms of immediate performance drop, worst-case degradation, and recovery behavior. Since different shift origins may require different adaptation mechanisms, a single method that performs well under one shift type cannot be assumed to be optimal for all types of distributional shift. The proposed taxonomy addresses this issue by separating shifts according to their causal origin in the agent– environment interaction process. This makes the source of change more explicit by distinguishing environmentdriven from agent-driven shifts within this process. The taxonomy also provides a structured way to relate common terms, such as stationary/non-stationary and in-distribution/out-of-distribution settings, to more specific shift origins. This view is complemented by the proposed evaluation metrics, which measure not only whether performance decreases, but also how strongly the agent is affected, how quickly it recovers, and how much post-shift performance is retained. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:33

These contributions also create several opportunities for future extension. First, the taxonomy can be extended beyond direct agent–environment interaction to settings such as offline reinforcement learning, where the agent learns from a fixed dataset that may have been generated by another policy or another agent before the evaluated agent interacts with the environment. Second, future work can further refine the interpretation of mixed cases. For example, a visual color change may correspond to an observation shift if it changes the agent’s input without affecting the environment dynamics or reward. However, if the color is part of the environment state and directly affects reward or transition behavior, then it may also involve state, transition, or reward-related shift. Third, in multi-agent RL settings, the formulation can be extended from decentralized agent–environment interaction toward centralized settings, such as centralized critics, shared observations, joint policies, or centralized training with decentralized execution.

Data and Code Availability The source code, experimental configuration files, and raw experimental logs required to reproduce the computational results will be made publicly available upon publication.

Acknowledgments Co-funded by the European Union under the Marie Skłodowska-Curie Grant Agreement No 101081465 (AUFRANDE). Views and opinions expressed are however, those of the author(s) only and do not necessarily reflect those of the European Union or the Research Executive Agency. Neither the European Union nor the Research Executive Agency can be held responsible for them. The authors would also like to acknowledge the support and collaboration of Naval Group.

References A. Ajay, A. Gupta, D. Ghosh, S. Levine, and P. Agrawal. 2022. “Distributionally Adaptive Meta Reinforcement Learning.” In: Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS). P. Altmann, F. Ritz, L. Feuchtinger, J. Nüßlein, C. Linnhoff-Popien, and T. Phan. 2023. “CROP: Towards Distributional-Shift Robust Reinforcement Learning Using Compact Reshaped Observation Processing.” In: Proceedings of the 32th International Joint Conference on Artificial Intelligence (IJCAI). Macao, SAR China, 3414–3422. isbn: 978-1-956792-03-4. doi:10.24963/ijcai.2023/380. E. Amhraoui and T. Masrour. 2023. “Smooth Q-Learning: An Algorithm for Independent Learners in Stochastic Cooperative Markov Games.” Journal of Intelligent & Robotic Systems, 108, 4, 65. doi:10.1007/s10846-023-01917-z. D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané. 2016. Concrete Problems in AI Safety. arXiv:1606.06565. (2016). doi:10.48550/arXiv.1606.06565. K. Åström. 1965. “Optimal Control of Markov Processes with Incomplete State Information.” Journal of Mathematical Analysis and Applications, 10, 1, 174–205. doi:10.1016/0022-247X(65)90154-X. P. Athanasios and S. U. Pillai. 2002. Probability, Random Variables, and Stochastic Processes. (4th ed.). McGraw-Hill. J. Beck, R. Vuorio, E. Z. Liu, Z. Xiong, L. Zintgraf, C. Finn, and S. Whiteson. 2024. A Survey of Meta-Reinforcement Learning. arXiv:2301.08028. (2024). doi:10.48550/arXiv.2301.08028. R. Chen. 2024. “Foresight Distribution Adjustment for Off-policy Reinforcement Learning.” In: Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS). K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. 2019. “Quantifying Generalization in Reinforcement Learning.” In: Proceedings of the 36th International Conference on Machine Learning (ICML). B. C. Da Silva, E. W. Basso, F. S. Perotto, A. L. C. Bazzan, and P. M. Engel. 2006. “Improving Reinforcement Learning with Context Detection.” In: Proceedings of the 5th International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS). ACM, Hakodate Japan, 810–812. isbn: 978-1-59593-303-4. doi:10.1145/1160633.1160779. L. Devroye, L. Györfi, and G. Lugosi. 2008. A Probabilistic Theory of Pattern Recognition. (3rd ed.). Springer, New York, NY. isbn: 978-0-38794618-4. J. Doob. 1990. Stochastic Process. Wiley, New York. P. Du, F. Li, and J. Shao. 2024. “Multi-Agent Reinforcement Learning Clustering Algorithm based on Silhouette Coefficient.” Neurocomputing, 596, 127901. doi:10.1016/j.neucom.2024.127901.

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:34

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. 2016. RL2: Fast Reinforcement Learning via Slow Reinforcement Learning. arXiv:1611.02779. (2016). doi:10.48550/arXiv.1611.02779. G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester. 2021. “Challenges of Real-world Reinforcement Learning: Definitions, Benchmarks and Analysis.” Machine Learning, 110, 9, 2419–2468. doi:10.1007/s10994-021-05961-4. A. Echchahed and P. S. Castro. 2025. A Survey of State Representation Learning for Deep Reinforcement Learning. en. arXiv:2506.17518. (2025). doi:10.48550/arXiv.2506.17518. P. Emami, X. Zhang, D. Biagioni, and A. S. Zamzam. 2023. “Non-Stationary Policy Learning for Multi-Timescale Multi-Agent Reinforcement Learning.” In: Proceedings of the 62nd of IEEE Conference on Decision and Control (CDC). IEEE, Singapore, 2372–2378. isbn: 979-8-3503-0124-3. doi:10.1109/CDC49753.2023.10384223. S. K. Esser, J. L. McKinstry, D. Bablani, R. Appuswamy, and D. S. Modha. 2020. “Learned Step Size Quantization.” In: Proceedings of the 8th International Conference on Learning Representations (ICLR). C. Finn, P. Abbeel, and S. Levine. 2017. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv:1703.03400. (2017). doi:10.48550/arXiv.1703.03400. T. Fujimoto, J. Suetterlein, S. Chatterjee, and A. Ganguly. 2023. “Assessing the Impact of Distribution Shift on Reinforcement Learning Performance.” In: Workshop on Regulatable Machine Learning at the 37th Conference on Neural Information Processing Systems (RegML @ NeurIPS). arXiv:2402.03590. doi:10.48550/arXiv.2402.03590. S. Gu, L. Shi, M. Wen, M. Jin, E. Mazumdar, Y. Chi, A. Wierman, and C. Spanos. 2025. “Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning.” In: Procedings of the 13th International Conference on Learning Representations (ICLR). T. Haider, F. S. Roza, D. Eilers, and K. Roscher. 2021. “Domain Shifts in Reinforcement Learning: Identifying Disturbances in Environments.” CEUR Workshop Proceedings, 2916. S. Herremans, A. Anwar, and S. Mercelis. 2024. Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model. arXiv:2406.09976. (2024). doi:10.48550/arXiv.2406.09976. X. Hickman, Y. Lu, and D. Prince. 2025. “Hybrid Safe Reinforcement Learning: Tackling Distribution Shift and Outliers with the Student-t’s Process.” Neurocomputing, 634, 129912. doi:10.1016/j.neucom.2025.129912. R. A. Howard. 1960. Dynamic Programming and Markov Processes. The Technology Press of The Massachusetts Institute of Technology and John Wiley & Sons, Inc. B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. 2018. “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference.” In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, Salt Lake City, UT, 2704–2713. isbn: 978-1-5386-6420-9. doi:10.1109/CVPR.2018.00286. J. Jiang and Z. Lu. 2022. “I2Q: A Fully Decentralized Q-Learning Algorithm.” In: Proceedings of the 36th Conference on Neural Information Processing Systems (NeurIPS). W. Jiang, M. Liu, M. Xu, S. Chen, K. Shi, P. Liu, C. Zhang, and F. Zhao. 2024. “New Reinforcement Learning based on Representation Transfer for Portfolio Management.” Knowledge-Based Systems, 293, 111697. doi:10.1016/j.knosys.2024.111697. L. P. Kaelbling, M. L. Littman, and A. R. Cassandra. 1998. “Planning and Acting in Partially Observable Stochastic Domains.” Artificial Intelligence, 101, 1-2, 99–134. doi:10.1016/S0004-3702(98)00023-X. K. Khetarpal, M. Riemer, I. Rish, and D. Precup. 2022. “Towards Continual Reinforcement Learning: A Review and Perspectives.” Journal of Artificial Intelligence Research, 75, 1401–1476. doi:10.1613/jair.1.13673. R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. 2023. “A Survey of Zero-shot Generalisation in Deep Reinforcement Learning.” Journal of Artificial Intelligence Research, 76, 201–264. doi:10.1613/jair.1.14174. E. Korkmaz. 2023. “A Survey on Generalization in Deep Reinforcement Learning.” In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS). S. Lee, Y. Seo, K. Lee, P. Abbeel, and J. Shin. 2020. “Addressing Distribution Shift in Online Reinforcement Learning with Offline Datasets.” In: Offline Reinforcement Learning Workshop at Neural Information Processing Systems. https://neurips.cc/virtual/2020/20101. D. A. Levin and Y. Peres. 2009. Markov Chains and Mixing Times. (2nd ed.). American Mathematical Society. W. Li, X. Wang, B. Jin, J. Sheng, and H. Zha. 2022. “Dealing with Non-Stationarity In MARL via Trust-Region Decomposition.” In: Procedings of the 10th International Conference on Learning Representations (ICLR). J. Liu, Z. Shen, Y. He, X. Zhang, R. Xu, H. Yu, and P. Cui. 2023. Towards Out-Of-Distribution Generalization: A Survey. arXiv:2108.13624. (2023). doi:10.48550/arXiv.2108.13624. Q. Liu, C. Szepesvári, and C. Jin. 2022. Sample-Efficient Reinforcement Learning of Partially Observable Markov Games. arXiv:2206.01315. (2022). Z. Liu, J. Lu, G. Zhang, and J. Xuan. 2024. A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Points. arXiv:2405.14214. (2024). doi:10.48550/arXiv.2405.14214. J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang. 2018. “Learning under Concept Drift: A Review.” en. IEEE Transactions on Knowledge and Data Engineering, 1–1. doi:10.1109/TKDE.2018.2876857. W. Luo, H. Li, Z. Zhang, C. Han, C. Zhou, J. Lv, and T. Guo. 2025. Mitigating Distribution Shift in Model-based Offline RL via Shifts-aware Reward Learning. arXiv:2408.12830. (2025). doi:10.48550/arXiv.2408.12830. Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning

•

00:35

A. Malinin and M. Gales. 2018. “Predictive Uncertainty Estimation via Prior Networks.” In: Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS). V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602. (2013). doi:10.48550/arXiv.1312.5602. S. Mohseni, H. Wang, C. Xiao, Z. Yu, Z. Wang, and J. Yadawa. 2023. “Taxonomy of Machine Learning Safety: A Survey and Primer.” ACM Computing Surveys, 55, 8, 1–38. doi:10.1145/3551385. F. Muratore, T. Gruner, F. Wiese, B. Belousov, M. Gienger, and J. Peters. 2021. “Neural Posterior Domain Randomization.” In: Procedings of the 5th Conference on Robot Learning (CoRL). N. S. Neggatu, J. Houssineau, and G. Montana. 2025. Evaluation-Time Policy Switching for Offline Reinforcement Learning. arXiv:2503.12222. (2025). doi:10.48550/arXiv.2503.12222. J. Norris. 1997. Markov Chain. Cambridge University Press. Y. Ovadia, E. Fertig, B. Lakshminarayanan, S. Nowozin, D. Sculley, J. Dillon, J. Ren, Z. Nado, and J. Snoek. 2019. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty under Dataset Shift.” In: Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS). C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song. 2019. Assessing Generalization in Deep Reinforcement Learning. arXiv:1810.12282. (2019). doi:10.48550/arXiv.1810.12282. S. Padakandla. 2022. “A Survey of Reinforcement Learning Algorithms for Dynamically Varying Environments.” ACM Computing Surveys, 54, 6, 1–25. doi:10.1145/3459991. S. Padakandla, P. K. J., and S. Bhatnagar. 2020. “Reinforcement Learning Algorithm for Non-Stationary Environments.” Applied Intelligence, 50, 11, 3590–3606. doi:10.1007/s10489-020-01758-5. G. Papadopoulos, A. Kontogiannis, F. Papadopoulou, C. Poulianou, I. Koumentis, and G. Vouros. 2025. An Extended Benchmarking of Multi-Agent Reinforcement Learning Algorithms in Complex Fully Cooperative Tasks. arXiv:2502.04773. (2025). doi:10.48550/arXiv.2502.04773. G. Papoudakis and F. Christianos. 2021. “Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks.” In: Proceedings of the 35th Conference on Neural Information Processing Systems (NeurIPS). G. Papoudakis, F. Christianos, A. Rahman, and S. V. Albrecht. 2019. Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning. arXiv:1906.04737. (2019). M. L. Puterman. 2005. Markov Decision Process. Wiley-Interscience. J. Quiñonero-Candela, M. Sugiyama, A. Schwaighofer, and N. D. Lawrence, (Eds.) . 2010. Dataset shift in machine learning. Neural information processing series. MIT Press, Cambridge, Mass. isbn: 978-0-262-17005-5 978-0-262-25510-3. A. Shakerimov, T. Alizadeh, and H. A. Varol. 2023. “Efficient Sim-to-Real Transfer in Reinforcement Learning Through Domain Randomization and Domain Adaptation.” IEEE Access, 11, 136809–136824. doi:10.1109/ACCESS.2023.3339568. S. Shalev-Shwartz and S. Ben-David. 2014. Understanding Machine Learning: From Theory to Algorithms. (1st ed.). Cambridge University Press. isbn: 978-1-107-05713-5 978-1-107-29801-9. doi:10.1017/CBO9781107298019. M. Al-Shedivat, T. Bansal, Y. Burda, I. Sutskever, I. Mordatch, and P. Abbeel. 2018. Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments. arXiv:1710.03641. (2018). doi:10.48550/arXiv.1710.03641. R. B. Slaoui, W. R. Clements, J. N. Foerster, and S. Toth. 2020. Robust Visual Domain Randomization for Reinforcement Learning. arXiv:1910.10537. (2020). doi:10.48550/arXiv.1910.10537. M. Sugiyama, C. T. A. Jp, M. Krauledat, and M. Krauledat. 2007. “Covariate Shift Adaptation by Importance Weighted Cross Validation.” Journal of Machine Learning Research. R. S. Sutton and A. G. Barto. 2018. Reinforcement Learning: an Introduction. (Second edition ed.). Adaptive Computation and Machine Learning Series. The MIT Press, Cambridge, Massachusetts. isbn: 978-0-262-03924-6. L. Tamang, M. R. Bouadjenek, R. Dazeley, and S. Aryal. 2025. “Handling Out-of-Distribution Data: A Survey.” IEEE Transactions on Knowledge and Data Engineering, 37, 10, 5948–5966. doi:10.1109/TKDE.2025.3592614. G. Tiboni, K. Arndt, and V. Kyrki. 2023. “DROPO: Sim-to-Real Transfer with Offline Domain Randomization.” Robotics and Autonomous Systems, 166, 104432. doi:10.1016/j.robot.2023.104432. J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. 2017. “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.” en. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, Vancouver, BC, 23–30. isbn: 978-1-5386-2682-5. doi:10.1109/IROS.2017.8202133. V. Vapnik. 1999. “An Overview of Statistical Learning Theory.” IEEE Transactions on Neural Networks, 10, 5, 988–999. doi:10.1109/72.788640. D. Wang, L. Li, W. Wei, Q. Yu, J. Hao, and J. Liang. 2025. “Improving Generalization in Offline Reinforcement Learning via Latent Distribution Representation Learning.” In: Proceedings of the 39th AAAI Conference on Artificial Intelligence. C. Xu, J. Wang, X. Zhu, Y. Yue, W. Zhou, Z. Liang, and D. Wojtczak. 2024. “Decentralized Multi-Agent Cooperation via Adaptive Partner Modeling.” Complex & Intelligent Systems, 10, 4, 4989–5004. doi:10.1007/s40747-024-01421-3. T. Xu, Z. Li, and Q. Ren. 2024. “Meta-Reinforcement Learning Robust to Distributional Shift Via Performing Lifelong In-Context Learning.” In: Proceedings of the 41st International Conference on Machine Learning (ICML). Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.

00:36

•

Wibowo, Santos, Baghdadi, Stephenson, Sammut, & Diguet

Z. Yuan, S. Yang, P. Hua, C. Chang, K. Hu, and H. Xu. 2023. “RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization.” In: Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS). Y. Zhai, P. Peng, C. Su, and Y. Tian. 2023. “Dynamic Belief for Decentralized Multi-Agent Cooperative Learning.” In: Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI). Macau, SAR China, 344–352. isbn: 978-1-956792-03-4. doi:10.24963/ijcai.202 3/39. C. Zhang, O. Vinyals, R. Munos, and S. Bengio. 2018. A Study on Overfitting in Deep Reinforcement Learning. arXiv:1804.06893. (2018). doi:10.48550/arXiv.1804.06893. W. Zhao, J. P. Queralta, and T. Westerlund. 2020. “Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey.” In: Proceedings of the IEEE Symposium Series on Computational Intelligence (SSCI). arXiv:2009.13303, 737–744. doi:10.1109/SSCI47803.2020.9308468..

Journal of Artificial Intelligence Research, Vol. 00, Article 00. Publication date: 0000.