BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control∗ Yifan Zhang†
Liang Zheng‡
arXiv:2605.16170v1 [cs.LG] 15 May 2026
Abstract Real-world control systems frequently operate under piecewise stationary conditions, where dynamics remain stable for extended periods before undergoing abrupt regime changes. Standard robust RL methods face a fundamental dilemma: a globally conservative policy wastes performance during stable periods, while a locally adaptive policy risks catastrophic failure when the regime changes undetected. We propose BAPR (Bayesian Amnesic Piecewise-Robust SAC), which unifies Bayesian Online Change Detection (BOCD) with robust ensemble RL. The BAPR operator—a convex combination of mode-conditional Bellman operators weighted by a frozen belief distribution—is a γ-contraction. A complementary counterexample, machine-verified in Lean 4, establishes a sharp boundary: when beliefs depend on the Q-function, the contraction factor becomes γ + λ∆ (where ∆ is the mode reward gap), and contraction fails exactly when γ + λ∆ ≥ 1. We derive a component-wise formal error budget for the abstract operator—every component machine-verified— bounding post-switch recovery; the budget applies to the abstract mode-mixture operator and inherits to the implemented shared-critic algorithm only through the frozen-parameter design intuition. All results are formally verified with no sorry (1,145 lines across 3 Lean 4 files, 22 machine-verified theorems). BOCD drives an adaptive conservatism mechanism: the policy becomes maximally conservative after detected change-points and smoothly relaxes as confidence grows, with detection delay O(log(1/δ)). A context-conditioning module trained via RMDM loss provides mode-aware representations from simulator-provided mode IDs at training time and requires no mode labels at deployment.
Keywords: non-stationary reinforcement learning; Bayesian change-point detection; robust ensemble; piecewise stationarity; adaptive conservatism; context-conditioned policy.
1
Introduction
Non-stationarity is a pervasive challenge in real-world control systems. From urban transit networks experiencing sudden traffic incidents to robotic systems undergoing mechanical wear, the environment dynamics that an agent faces are rarely fixed. Instead, many practical settings exhibit piecewise stationarity: the system operates under one regime for an extended period, then abruptly transitions to a different regime with substantially different dynamics [17, 12]. Standard Reinforcement Learning (RL) algorithms implicitly assume stationarity—that the transition kernel P (s′ |s, a) and reward function R(s, a) remain fixed throughout training. When this assumption is violated, the agent’s learned value function becomes stale after a regime change, leading to catastrophic performance degradation. This problem is especially acute for off-policy methods like Soft Actor-Critic (SAC) [7], where the replay buffer accumulates transitions from expired regimes, effectively “poisoning” the critic with outdated data. Definition 1.1 (Regime Staleness Problem). Let M1 , M2 , . . . be a sequence of MDPs (regimes) encountered by the agent, with regime Mk active during time interval [tk , tk+1 ). An off-policy agent with replay buffer D suffers from regime staleness if, after a switch at time tk+1 : ∗ Code available at https://github.com/erzhu419/BAPR † Central South University, [email protected], [email protected] ‡ Central South University, [email protected]
1
(i) Buffer contamination: A fraction η > 0 of samples in D come from Mk , whose dynamics differ from the current Mk+1 ; (ii) Value misalignment: The learned Q-function Qt satisfies ∥Qt − Q∗k+1 ∥∞ > ϵ even as the number of samples from Mk+1 grows, due to the persistent bias from stale transitions. This is structurally distinct from the Q-value poisoning identified in stationary stochastic environments [24]: staleness arises from temporal model mismatch, not from conflating aleatoric and epistemic uncertainty. Existing approaches to non-stationary RL fall into two broad categories, each with significant limitations: 1. Reactive adaptation methods (e.g., context-conditioned policies [13, 20], meta-learning [15, 5]) adapt the policy based on recent observations but lack a principled mechanism for detecting when a regime change has occurred. They continuously adapt even during stable periods, introducing unnecessary variance. 2. Robust methods (e.g., Robust MDPs [21], distributionally robust RL [26, 18]) hedge against worst-case uncertainty but use a static uncertainty set calibrated for the entire training process. In piecewise stationary environments, this leads to excessive conservatism during stable periods (when the current regime is well-characterized) and insufficient conservatism immediately after a regime change (when the uncertainty set from the previous regime no longer applies). We argue that the missing ingredient is a time-aware uncertainty quantification mechanism that can: • Detect regime changes in real-time from observable signals; • Modulate the degree of conservatism based on confidence in the current regime; • Forget stale information from expired regimes in a principled manner. To this end, we propose BAPR (Bayesian Amnesic Piecewise-Robust SAC), which integrates Bayesian Online Change Detection (BOCD) [1] into the robust ensemble SAC framework of RESAC [24]. The key idea is elegant: BOCD maintains a posterior distribution ρ(h) over run-lengths h (time since the last change-point), and this belief distribution serves as a bridge between change-point detection and adaptive conservatism. We use the variance-growing likelihood σ02 + σg h (Eq. 8) so that anomalous surprise pushes posterior mass toward larger run-lengths, raising the expected run-length is the change-point signal that h̄ above its steady-state baseline; the deviation h̄/(H−1) − λ̄EMA w drives βeff (§4.3). The semantics is therefore “surprise raises h̄ above its EMA baseline,” not “surprise resets h̄ to zero”; we adopt this design because the EMA-deviation signal degrades gracefully when the surprise model is mis-specified and removes the need to set an absolute change-point threshold. Building upon RE-SAC’s disentangled treatment of aleatoric and epistemic risk [24], and extending the Environment-aware Soft actor-Critic with stochastic Policy (ESCP) [13] framework for contextconditioning, our contributions are: • Sharp characterization of the frozen-parameter design space: We establish a sharp boundary for contraction in belief-weighted Bellman operators. On the positive side, the BAPR operator—a convex combination of mode-conditional operators with frozen belief weights—is a γ-contraction (Theorem 4.2). On the negative side, when belief weights depend on Q, the contraction factor becomes exactly γ + λ∆ (where ∆ is the mode reward gap), and contraction fails precisely when γ + λ∆ ≥ 1 (Theorem D.1). This characterizes a “phase transition” in operator stability that is distinct from (and complementary to) the frozen-penalty requirement in RE-SAC. All results are machine-verified in Lean 4 with no sorry (BAPR.lean: 560 lines; BAPR-Counterproof.lean: 265 lines).
2
• Formal piecewise convergence rate: Under three structural assumptions—Mode Separability (Assumption B.1), Lipschitz Surprise (Assumption B.2), and Metastable Period (Assumption B.3)—we establish a formal Piecewise Q-Value Convergence Rate Theorem (Appendix A) showing that after each regime switch, the Q-value error decays as γ n · Eswitch + (εproj +σ)/(1−γ), with a total recovery cost that is polynomial in all parameters—no exponential dependence on the horizon or risk sensitivity. Every component of this bound corresponds to a machine-verified Lean 4 theorem, with no informal steps. • Adaptive conservatism via BOCD (provably safe): We derive a belief-weighted penalty λw from the BOCD posterior that modulates the LCB coefficient β in real-time. We prove (machine-verified) that βeff ≤ βbase always holds and that βeff is monotonically decreasing in surprise magnitude—ensuring that the adaptive mechanism can only increase conservatism, never reduce it below the RE-SAC baseline (Appendix G). • Context-conditioning with RMDM: We introduce a context-conditioning module trained via a Representation learning for Mode Detection and Matching (RMDM) loss; training uses the simulator-provided mode IDs as a supervised target, while inference at deployment requires no mode labels (the embedding is computed from observed transitions). • Detection delay bound and formal error budget (Theorem E.2): Under the mode separability condition, the BOCD posterior converges to the correct mode in O(log(1/δ)) steps. Combined with the regime switch perturbation bound (Lean: regime_switch_perturbation) and function-approximation bounds from ApproxContraction.lean (total: 1,145 lines across 3 Lean files, 22 machine-verified theorems), this provides a complete, fully formal error budget for practical BAPR training.
2
Related work
2.1
Non-stationary and piecewise-stationary RL
Non-stationarity in RL has been studied through several lenses. Lifelong/continual RL [10] focuses on sequential task learning without forgetting. Context-conditioned policies [13, 20, 19] learn a latent representation of the environment mode and condition the policy on it. The ESCP framework [13] trains an “environment probe” to produce a context vector, enabling the policy to distinguish between different dynamics. PEARL [19] uses a probabilistic context variable inferred from recent transitions, but assumes the task is fixed within each episode and does not handle intra-episode regime shifts. Meta-RL approaches [5, 15] learn to adapt quickly via gradient-based or recurrence-based mechanisms. VariBAD [27] maintains a Bayes-optimal belief over tasks using a variational autoencoder, but assumes a multi-task setting where the task is sampled at episode start and remains fixed. In contrast, BAPR targets intra-episode regime shifts where the dynamics change abruptly during a single trajectory—a fundamentally different and harder problem that requires online detection rather than episode-level inference. Hidden-Parameter MDPs (HiP-MDPs) [4] model non-stationarity through latent environment parameters that must be inferred online. While conceptually related, HiP-MDPs typically assume a smooth parameter space and continuous adaptation, whereas BAPR is designed for the piecewise setting where abrupt discontinuities necessitate rapid detection and conservative response. The CARL benchmark [2] provides standardized contextual RL environments with parameterized dynamics changes; our non-stationary MuJoCo environments follow a similar protocol but with intra-episode regime switches rather than episode-level context variation. In the piecewise-stationary setting, change-point detection methods [1, 11] have been applied to RL to segment the experience into stationary blocks [17, 12]. Sliding-window approaches (e.g., SW-UCB [6]) and exponential-forgetting methods offer practical alternatives, but lack the principled posterior uncertainty quantification of Bayesian methods. Context-aware safe RL [3] addresses non-stationarity with context detection, but relies on pre-defined context boundaries. Most existing work uses change-point detection to reset or re-initialize the agent, discarding all prior knowledge. 3
Our approach instead uses the BOCD posterior to smoothly modulate the degree of conservatism, preserving useful knowledge while hedging against regime uncertainty.
2.2
Robust RL and uncertainty quantification
Robust MDPs [9, 21] seek policies optimal under worst-case transition perturbations. Recent advances connect robustness to regularization [26, 23, 22] and to risk-sensitive objectives [16]. Our predecessor RE-SAC [24] disentangles aleatoric and epistemic risks using IPM-based weight regularization and diversified Q-ensembles, respectively, and provides machine-verified contraction proofs. BAPR inherits and extends this machinery to piecewise-stationary settings. The key limitation of all static robust methods is that the uncertainty set Ps,a is fixed throughout training. In a piecewise-stationary environment, the “true” uncertainty should be large immediately after a regime change (when the agent doesn’t know which regime it’s in) and small during stable periods (when the current regime is well-characterized). BAPR achieves this adaptive uncertainty quantification through the BOCD posterior.
2.3
Bayesian online change detection
Bayesian Online Change Detection (BOCD) [1] maintains a posterior distribution over the run-length h (number of time steps since the last change-point) using a recursive message-passing algorithm. At each step, the posterior is updated by: ( ρt (h − 1) · p(ξt | h − 1) · (1 − Hhazard ) if h > 0 ρt+1 (h) ∝ (1) P Hhazard · h′ ρt (h′ ) · p(ξt | h′ ) if h = 0 where ξt is the observed “surprise” signal, p(ξt | h) is the predictive likelihood under run-length h, and Hhazard is the prior probability of a change-point at each step. BOCD has been applied in bandits [14] and simple MDPs [8], but its integration with deep RL—and particularly with robust ensemble methods—has not been explored. We do not claim the detection delay analysis (Appendix E) as a novel BOCD result; the bound follows standard likelihood-ratio arguments [1]. Our contribution is the integration: connecting the BOCD posterior to the Bellman contraction framework (Theorem 4.2) and the adaptive penalty mechanism (§4.3), producing a coherent error budget spanning detection, contraction, and function approximation.
3
Preliminaries
3.1
Piecewise-stationary MDP
We model the environment as a Piecewise-Stationary MDP (PS-MDP), a sequence of MDPs {Mk }∞ k=1 where each Mk = (S, A, Pk , Rk , γ) shares the state and action spaces but has distinct transition kernels and reward functions. Regime switches occur at unknown times t1 < t2 < · · · , and between consecutive switches the MDP is stationary. This formalization captures a wide range of practical scenarios: in bus fleet control, Mk might correspond to “normal traffic,” “severe congestion,” or “demand surge” regimes [25]; in robotic locomotion, it might correspond to different gravity levels, friction coefficients, or terrain types [13].
3.2
Maximum entropy RL
Following SAC [7], the agent maximizes the entropy-augmented objective: "∞ # X t J(π) = Eπ,P γ (R(st , at ) + αH(π(·|st ))) ,
(2)
t=0
where H(π(·|st )) is the Shannon entropy and α is the temperature parameter. The soft Bellman operator T π Q(s, a) = R(s, a) + γEs′ ∼P [V π (s′ )] is a γ-contraction guaranteeing convergence to the optimal soft Q-function. 4
3.3
RE-SAC: Disentangled robust ensemble SAC
Our framework builds upon the Robust Ensemble SAC (RE-SAC) [24], which addresses Q-value instability in stationary stochastic environments by disentangling aleatoric and epistemic risks. RE-SAC defines the Robust-Ensemble Value (REV) operator: T REV (s, a) = R(s, a) + γ (Es′ ∼p◦ [V (s′ )] − λepi · Γepi (s, a) − κ) ,
(3)
where Γepi (s, a) = Var({Qϕ′k (s, a)}K k=1 ) is the epistemic penalty from the frozen target ensemble, and P (θ) κ = λale l ∥Wl ∥1 is the aleatoric penalty from the frozen critic weights. Both penalties are fixed during each Bellman backup, ensuring γ-contraction [24]. We adopt the same LCB policy objective: Lπ (θ) = Es∼D,a∼πθ α log πθ (a|s) − Q̄(s, a) + β · σens (s, a) . (4)
4
Methodology
4.1
From stationary robustness to piecewise-stationary robustness
The central challenge in extending RE-SAC to piecewise-stationary environments is that the uncertainty set must be time-varying: large after a regime change (high uncertainty about the current dynamics) and small during stable periods (well-characterized dynamics). A static penalty coefficient β, optimal for one regime, will be either too conservative or too aggressive in others. We resolve this by introducing a mode-indexed family of Bellman operators. Let M = {1, . . . , M } be a finite set of environment modes (regimes). Each mode m ∈ M corresponds to a distinct dynamics configuration (e.g., “normal traffic” vs. “severe congestion” in bus control, or different gravity levels in robotic locomotion). For each mode, we define a mode-conditional Bellman operator: ! X ′ Q ′ Tm Q(s, a) = Rm (s, a) + γ Pm (s |s, a)V (s ) − λepi Γepi,m (s, a) − κ , (5) s′
where Rm , Pm , and Γepi,m are the reward, transition, and epistemic penalty under mode m, and V Q (s′ ) = maxa′ Q(s′ , a′ ). Each Tm has the same structure as the RE-SAC operator; its frozen penalties ensure γ-contraction per mode. The BAPR operator is defined as the belief-weighted mixture: X TρBAP R Q(s, a) = ρ(m) · Tm Q(s, a), (6) m∈M
where ρ : M → [0, 1] is a belief distribution over modes satisfying
P
m ρ(m) = 1.
Remark 4.1 (Strict separation of mode space M and run-length space). We carefully distinguish two indexing spaces throughout this paper: • Modes m ∈ M: physical environment configurations with distinct dynamics (Rm , Pm ). The theoretical operator (Eq. (6)) and its contraction proof (Theorem 4.2) operate over M. • Run-lengths τ ∈ {0, . . . , H − 1}: time elapsed since the last detected change-point. The BOCD module (§4.2) maintains a posterior ρBOCD (τ ) over run-lengths, which determines when a change occurred. The two spaces serve complementary roles: BOCD answers “when did the regime change?” (temporal), while the context module (§4.4) answers “which mode m are we in now?” (categorical). In the implementation, the BOCD run-length posterior is not used to index the operator Tm directly; instead, it is collapsed into a scalar λw (§4.3) that modulates the overall degree of conservatism. Theorem 4.2 (BAPR Contraction). Under: P • Frozen belief ρ ≥ 0 with m ρ(m) = 1 (frozen during backup); 5
• Per-mode transitions Pm ≥ 0 with
′ s′ Pm (s |s, a) = 1 for all m, s, a;
P
• Frozen penalties κ, Γepi,m (from target network); • 0 ≤ γ < 1; the operator TρBAP R is a γ-contraction in the L∞ norm with a unique fixed point. Proof sketch. The proof exploits a fundamental property: a convex combination of γ-contractions is itself a γ-contraction. For any Q1 , Q2 with ∥Q1 − Q2 ∥∞ ≤ ε: TρBAP R Q1 (s, a) − TρBAP R Q2 (s, a) =
X
ρ(m) (Tm Q1 (s, a) − Tm Q2 (s, a))
m
≤
X
ρ(m) |Tm Q1 (s, a) − Tm Q2 (s, a)|
(ρ ≥ 0, triangle ineq.)
m
≤
X
ρ(m) · γε
(per-mode contraction)
m
= γε.
(
X
(7)
ρ = 1)
The full proof, including Blackwell’s sufficiency conditions (monotonicity and discounting), is machineverified in Lean 4 (BAPR.lean) with no sorry. See Appendix C for details. The key Lean 4 theorem statement is: theorem bapr_contraction (p : Params) (R : H → S → A → R) (P : H → S → A → S → R) (Γ_epi : H → S → A → R) (κ : R) (ρ : H → R) P (hρ_nn : ∀ h, 0 ≤ ρ h) (hρ_sum : h, ρ h = 1) : ∃ k < 1, ∀ Q1 Q2 , dist (T_BAPR p R P Γ_epi κ ρ Q1 ) (T_BAPR p R P Γ_epi κ ρ Q2 ) ≤ k · dist Q1 Q2
Remark 4.3 (On the nature of the theoretical contribution). The algebraic step “convex combination of contractions is a contraction” is indeed a basic mathematical fact. The theoretical contribution lies not in the algebra but in the structural design insight: identifying that three independent frozenparameter requirements (frozen κ, frozen Γepi , frozen ρ) must hold simultaneously for contraction, and that violating any one of them breaks the guarantee (cf. the counterexamples in RE-SAC and BAPR). The Lean 4 verification serves as a design audit tool—it forces the author to make every structural P assumption explicit, including conditions (such as m ρ(m) = 1 being load-bearing for Blackwell’s discounting condition) that are easy to overlook in informal proofs. We view the machine-verified proof as a contribution to verifiable algorithm design methodology, not merely to mathematical novelty. Remark 4.4 (Necessity of frozen beliefs). If the belief weights ρ are allowed to depend on Q (e.g., the agent re-infers the mode based on its own value function), the operator takes the form Tbad (Q) = (γ + λ∆)Q + c in the worst case, where ∆ = R1 − R2 is the mode reward gap. When λ∆ ≥ 1 − γ, the effective contraction factor γ + λ∆ ≥ 1 and contraction fails. This is machine-verified in BAPR-Counterproof.lean (Theorem D.1). The frozen-belief design is therefore necessary, not merely convenient. Remark 4.5 (Joint regime belief b(h, z)). The scalar run-length posterior ρ(h) tracks when a regime change occurred but not which regime is active. To address this, our implementation maintains a joint posterior b : H × Z → R≥0 over (run-length, latent regime cluster), where Z is a discrete set of
6
regime clusters discovered online via k-means on observable signals (reward residual, ensemble Q-std, TD residual). The two marginals X X ρ(h) = b(h, z), µ(z) = b(h, z), z∈Z
h∈H
are concatenated into the critic input Q(s, a, e, ρ, µ). Since H × Z is itself a finite mode space, Theorem 4.2 applies verbatim: TbBAP R remains a γ-contraction. This is verified by the Lean theorem bapr_joint_contraction (BAPR.lean, §7.5), which reuses the original proof at the joint type. Marginal extraction is verified by marginal_h_sum_one / marginal_z_sum_one (§8.5), and the dual-marginal shared-critic equivalence by dual_marginal_critic_equiv (§10.5). The added µ(z) channel does not affect the contraction guarantee but enables recurring-regime memory: when the same z reappears, the marginal concentrates on the same cluster, so the critic conditions on a consistent input across regime visits.
4.2
Bayesian online change detection for RL
The BOCD module maintains a posterior belief ρ(h) over run-lengths, updated at each training iteration based on a multi-signal surprise measure ξt . The update follows the Adams–MacKay recursion [1]: Likelihood model. For each run-length h, the predictive likelihood of observing surprise ξ is modeled as a Gaussian with variance increasing linearly with h: 1 ξ2 p(ξ | h) = p exp − , (8) 2(σ02 + σg · h) 2π(σ02 + σg · h) where σ02 is the base variance and σg is the variance growth rate. This encoding reflects the intuition that short run-lengths (recent change-point) tolerate only small surprises, while long run-lengths (stable regime) accommodate larger fluctuations. Belief update. The posterior update combines growth (no change-point) and changepoint probabilities: Growth:
ρ′ (h) = ρ(h − 1) · p(ξt | h − 1) · (1 − Hhazard ) X Changepoint: ρ′ (0) = Hhazard · ρ(h′ ) · p(ξt | h′ ),
for h > 0,
(9) (10)
h′
P followed by normalization ρ′ (h) ← ρ′ (h)/Z where Z = h ρ′ (h). Surprise signal. We design a multi-signal surprise detector that fuses three complementary signals: t σQ ξt = wr · |zr | + wq · t + wκ · |κt − κt,target |, (11) σ̄Q where zr = (rt − r̄t )/std(rt ) is the reward z-score (detects mean-shift in reward distribution), t t σQ /σ̄Q is the ensemble Q-value standard deviation ratio (detects epistemic uncertainty jumps), and t |κ − κt,target | is the aleatoric penalty divergence (detects structural changes in the critic). Note that each channel is individually normalized: the reward z-score is scale-invariant by construction, the Q-std ratio is a dimensionless quantity, and the κ divergence operates on frozen scalar quantities with bounded range. The combined surprise is clipped to [0, 10] for numerical stability. While the default weights (0.5, 0.3, 0.2) were chosen based on signal informativeness (reward being the most direct indicator), the sensitivity to these weights is evaluated empirically. The BOCD belief is frozen during each Bellman backup, satisfying the structural requirement for contraction (Theorem 4.2). It is updated only between training iterations, using information from the most recent rollout.
7
4.3
Adaptive conservatism via belief-weighted penalty
The BOCD posterior directly drives the degree of conservatism in the policy update. We define the effective window as the expected run-length under the posterior: h̄ =
H−1 X
h · ρ(h),
(12)
h=0
which summarizes the BOCD belief into a scalar. The key dynamics are: when a regime change produces high surprise, the BOCD likelihood model (Eq. (8)) assigns higher probability to large run-lengths h (since longer-duration modes have higher variance σ02 + σg h and thus accommodate outlier surprises more easily). This causes h̄ to spike above its steady-state baseline. The belief-weighted penalty λw is computed with baseline subtraction to ensure zero penalty during stable periods: h̄ λw = max 0, − λ̄EMA , (13) w H −1 where λ̄EMA is an exponential moving average of the raw h̄/(H − 1) values, tracking the steady-state w baseline. During stable operation, h̄/(H − 1) remains near its EMA and λw ≈ 0. When a regime change triggers high surprise, h̄ spikes above the baseline, causing λw > 0. As the agent adapts to the new regime and surprise subsides, h̄ returns to its baseline and λw decays back to zero. The effective LCB coefficient becomes: βeff = βbase − λw · cpenalty ,
(14)
where βbase < 0 is the static LCB coefficient (inherited from RE-SAC) and cpenalty > 0 is the penalty scale. Since βbase < 0 and λw ≥ 0, we have βeff ≤ βbase : the penalty can only increase conservatism, never decrease it below the RE-SAC baseline. This “amnesic” behavior—becoming cautious after a detected change-point and gradually relaxing—is the Bayesian Amnesia component of BAPR.
4.4
Context-conditioning via RMDM
While BOCD detects when a regime change occurred, the agent also needs to identify which regime it is currently in. We achieve this through a context-conditioning module inspired by ESCP’s environment probe [13]. Context network. A lightweight MLP ϕψ : S → Rde maps observations to a low-dimensional context embedding e = ϕψ (s), L2-normalized to the unit sphere: e=
ϕψ (s) . ∥ϕψ (s)∥2 + ϵ
(15)
The context vector e is concatenated with the state input to both the critic (Qϕ (s ⊕ e, a)) and the policy (πθ (a|s, e)), enabling mode-aware decision-making. Mode labels and the Sim2Real paradigm. The RMDM diversity loss (below) requires grouping transitions by mode. During training in simulation, the PS-MDP structure provides mode identity τid as part of the environment (e.g., the current gravity level or congestion state). These labels are used only for the RMDM auxiliary loss; the RL components (critic, policy) never observe τid directly and receive only the raw state s. This follows the standard Sim2Real paradigm: BAPR is trained in simulation where piecewise mode labels are available by construction (since the simulator controls the regime schedule), then deployed zero-shot in real-world environments where mode labels are unavailable and the agent relies solely on the learned context embedding e and the BOCD belief for adaptation. An alternative fully unsupervised variant using BOCD-inferred pseudo-labels is an interesting direction for future work. RMDM loss. The context network is trained via a Representation learning for Mode Detection and Matching (RMDM) loss that enforces two properties:
8
1. Within-task consistency: Embeddings from the same environment mode should cluster together: q 1 X Lcons = Vars∼τ [ϕψ (s)] + ϵ. (16) |T | τ ∈T
2. Cross-task diversity: Embeddings from different modes should be spread apart. We use a Determinantal Point Process (DPP) diversity loss based on an RBF kernel: Ldiv = − log det K, Kij = exp −rrbf ∥ēi − ēj ∥2 + ϵ · 1[i = j], (17) where ēi is the mean embedding for mode i and rrbf is the RBF bandwidth. The combined RMDM loss is: LRMDM = wcons · Lcons + wdiv · Ldiv .
(18)
Delayed injection. To prevent the initially random context embeddings from destabilizing early RL training, we employ a warmup strategy: during the first Nwarmup training iterations, the context vector is set to zero, and the RL components train as pure RE-SAC. After warmup, context injection is activated, allowing the policy and critic to gradually incorporate mode information.
4.5
BAPR algorithm
The complete BAPR algorithm integrates the above components into the SAC framework. The critic objective is: ! K 1 X ′ ′ ′ ′ y =r+γ Q̂ϕk (s , a ) − α log πθ (a |s ) , (19) K k=1
where the epistemic penalty is moved to the policy loss (to avoid conflict with the OOD loss in the critic), and the aleatoric risk is handled by IPM-based weight regularization in the critic loss: h i X (k) 2 LQ (ϕk ) = E(s,a) (Qϕk (s ⊕ e, a) − y) + λale ∥Wl ∥1 + βood · σens (s, a). (20) l
The policy objective uses the adaptive LCB with belief-weighted penalty: Lπ (θ) = Es∼D,a∼πθ α log πθ (a|s, e) − Q̄(s ⊕ e, a) + βeff · σens (s ⊕ e, a) , where βeff = βbase − λw · cpenalty is the belief-modulated LCB coefficient (Eq. (14)).
9
(21)
Algorithm 1 BAPR: Bayesian Amnesic Piecewise-Robust SAC Initialize: Ensemble Q-networks {Qϕ1 , . . . , QϕK }, Actor πθ , Context network ϕψ , Replay Buffer D. Initialize: Target networks ϕ′k ← ϕk ; BOCD belief ρ ← Uniform; Surprise EMA. for each training iteration do // — Environment Interaction — Collect transition samples using πθ (with scan-fused GPU rollout if available). Store (s, a, r, s′ , d, τid ) in D, where τid is the current task/mode ID. // — BOCD Belief Update (between training iterations) — Compute surprise ξ from reward z-score, Q-std spike, and reg-norm divergence (Eq. (11)). Update belief ρ via BOCD recursion (Eqs. (9)–(10)). Compute λw from belief (Eq. (13)) and βeff (Eq. (14)). // — Gradient Updates (scan-fused N steps) — for n = 1, . . . , Nupdates do Sample mini-batch from D. Compute context: e = ϕψ (s), e′ = ϕψ (s′ ) (zero if warmup). // Context update Update ψ by minimizing LRMDM (Eq. (18)). // Critic update Update ϕk by minimizing LQ (ϕk ) (Eq. (20)). // Actor update (adaptive β) Update θ by minimizing Lπ (θ) with βeff (Eq. (21)). // Temperature update Update α via dual gradient descent. // Target network soft update ϕ′k ← τ ϕk + (1 − τ )ϕ′k . end for end for Remark 4.6 (TractableP approximation of the mixture operator). Theorem 4.2 proves contraction for the abstract mixture m ρ(m)Tm over M independent mode-conditional operators. In practice, instantiating M independent critics with separate dynamics models is computationally prohibitive. The actual algorithm uses a scalar approximation: a single ensemble critic Qϕ (s ⊕ e, a) is shared across all modes, with mode-awareness provided by the context embedding e, and the BOCD posterior is collapsed into a single adaptive penalty coefficient βeff (Eq. (14)). This approximation is conservative: (i) the context embedding e allows the shared critic to internally partition the function space P by mode; (ii) the scalar βeff serves as a low-rank surrogate for the mode-weighted epistemic penalty m ρ(m)λepi Γepi,m , dynamically adjusting the overall conservatism level. Crucially, the contraction guarantee still applies: βeff is frozen during each Bellman backup (as a scalar computed from the BOCD posterior between iterations), satisfying the structural requirement identified by the Lean 4 counterexample. The approximation affects the quality of the fixed point (how close Q∗BAPR is to the true robust value), not its existence or uniqueness. Remark 4.7 (Scan-fused implementation). In the JAX implementation, all Nupdates gradient steps within one iteration are fused into a single jax.lax.scan call, eliminating per-step Python overhead. The BOCD belief update operates in NumPy on the CPU (as it is inherently sequential and O(H) per step), while the gradient computations run entirely on the GPU. The belief is treated as a frozen scalar λw within the scan, consistent with the contraction requirement. Remark 4.8 (Computational overhead). A natural concern is whether BAPR’s additional components (BOCD, RMDM, adaptive β) introduce prohibitive overhead. In practice, the overhead is negligible: • BOCD: O(Hmax ) per iteration on the CPU (Hmax = 20), compared to O(K · d2 · Nupdates ) for the ensemble gradient steps on the GPU.
10
• RMDM: One additional gradient step per iteration on the context network (256 parameters), using the same mini-batch. • Adaptive β: A single scalar computation from the BOCD belief, O(1). In wall-clock time, BAPR is < 5% slower than RE-SAC (the dominant cost is the ensemble critic’s K = 10 forward/backward passes). Each component addresses a distinct failure mode that cannot be handled by the others (see the ablation hierarchy in Appendix I); removing any one degrades performance in specific scenarios. Remark 4.9 (Architectural relationship to RE-SAC and ESCP). BAPR can be understood as a principled composition: RE-SAC provides the stationary robustness backbone (disentangled aleatoric/ epistemic penalties), ESCP provides the context-conditioning mechanism (environment probe + RMDM loss), and BOCD provides the temporal adaptation layer (change-point detection + adaptive conservatism). Each component addresses a distinct aspect of the non-stationary control challenge:
5
Component
Addresses
Origin
IPM weight reg. (κ) Q-ensemble (Γepi ) Context network (e) BOCD belief (ρ) Adaptive βeff
Aleatoric risk (noise robustness) Epistemic risk (data gaps) Mode identification ( which regime) Change detection ( when regime changed) Time-varying conservatism
RE-SAC RE-SAC ESCP BAPR BAPR
Experiments
We evaluate BAPR on four non-stationary continuous control environments featuring intra-episode regime switches, comparing against robust, adaptive, and a representative context-conditioned metaRL baseline. Our experiments address five primary questions: (Q1) Does BAPR outperform existing methods across diverse non-stationary environments? (Q2) Is each component (BOCD, RMDM, adaptive β) necessary? (Q3) How quickly does BAPR detect and adapt to regime changes? (Q4) Does the adaptive conservatism mechanism behave as theoretically predicted? (Q5) How sensitive is BAPR to hyperparameter choices?
5.1
Experimental setup
Non-stationary environments. We construct piecewise-stationary variants of four MuJoCo locomotion tasks: Hopper, HalfCheetah, Walker2d, and Ant. Each environment cycles through K regimes by modifying dynamics parameters (gravity, friction, joint damping). To evaluate online detection robustness, regime switches occur at randomized intervals (drawn from a Poisson distribution centered at Tswitch steps), preventing trivial schedule memorization. Switches are not signaled to the agent. Details are provided in Appendix K. Baselines. We compare against: (i) SAC [7]: standard maximum-entropy RL (no robustness, no adaptation); (ii) RE-SAC [24]: static robust ensemble (robustness but no temporal adaptation); (iii) ESCP [13]: context-conditioned SAC (adaptation but no formal robustness); (iv) BAPR (ours): full framework with BOCD, RMDM, and adaptive β. All methods use the same ensemble size (K = 10), network architecture, and training budget. Results are averaged over 10 random seeds to ensure statistical significance, with shaded regions denoting 95% bootstrap confidence intervals.
11
5.2
Main results (Q1)
Figure 1 illustrates the training and evaluation progress of BAPR against the baseline methods across four non-stationary locomotion tasks. BAPR consistently matches or exceeds the performance of the baselines, notably outperforming both RE-SAC and ESCP in HalfCheetah and Ant. Training Reward (smoothed)
Hopper 600 500
15000
250
100
200 500
1000
1500
2000
RE-SAC ESCP BAPR
500
300
300
200
250
100
200
0
0
500
7500
1000
Iteration
1500
2000
150
10000
5000 2500
5000
0 500
1000
1500
2000
0
500
1000
1500
2000
RE-SAC ESCP BAPR
17500
400 350
15000
10000
0
400
RE-SAC ESCP BAPR
20000
12500
300
200
Ant
RE-SAC ESCP BAPR
17500
350
300
HalfCheetah
20000
RE-SAC ESCP BAPR
400
400
0
Eval Reward
Walker2d
RE-SAC ESCP BAPR
15000 12500 10000 7500 5000 RE-SAC ESCP BAPR 0
500
1000
Iteration
1500
2500 0
2000
0
500
1000
Iteration
1500
2000
0
20000 17500 15000 12500 10000 7500 5000 2500 0
500
1000
1500
2000
1000
1500
2000
RE-SAC ESCP BAPR
0
500
Iteration
Fig. 1: Training curves (smoothed) and evaluation rewards over 2000 iterations in non-stationary MuJoCo environments. Shaded regions denote 95% bootstrap confidence intervals across 10 random seeds. BAPR (ours) demonstrates robust adaptation traversing piecewise regime changes, particularly achieving higher final performance in HalfCheetah and Ant. Table 1 summarizes the final training performance at 2000 iterations. BAPR achieves the highest final reward in three out of the four environments. Table 1: Final training reward after 2000 iterations in non-stationary environments. Algorithm
Hopper
HalfCheetah
Walker2d
Ant
RE-SAC ESCP BAPR (Ours)
462 443 498
13133 4864 15369
363 281 317
15902 17397 18095
Understanding the Walker2d Performance Drop. On the cyclic continuous benchmark of Table 1, BAPR underperforms RE-SAC on Walker2d. Walker2d is particularly sensitive to postural equilibrium: stochastic tripping can trigger false-positive surprise signals in Eq. (11), causing a spike in conservatism exactly when precise recovery actions are needed. We additionally observe that, in the redesigned discrete-regime benchmark (Section 5.3, Table 2), both BAPR and the strongest learning baseline ESCP fail to converge to a useful policy on Walker2d within 1 500 training iterations (final returns under 400 for both methods). This indicates that the Walker2d shortfall is partly an environment-difficulty artifact—bipedal balance under regime switching is too hard for any of the methods at this training budget—rather than a property unique to BAPR’s adaptive conservatism. We retain Walker2d in our reporting but flag it as a setting that warrants either extended training or curriculum-based mode introduction for fair evaluation.
5.3
Discrete-regime benchmark with joint belief b(h, z) (Q1′ )
The cyclic-continuous benchmark of Table 1 samples a continuous family of 40 tasks, mixing the regime-discovery and policy-adaptation problems. To isolate the contribution of joint regime belief tracking (Remark 4.5), we additionally evaluate on a discrete-mode benchmark: each environment is restricted to K = 4 semantically distinct regimes (e.g., normal, heavy_low_g, light_high_g, 12
HalfCheetah, K = 4 modes, N = 5 seeds
evaluation return (avg over K test modes)
evaluation return (avg over K test modes)
stiff_joints for HalfCheetah / Ant; normal, slippery, heavy_load, low_grav for Hopper / Walker2d), with mean dwell time of 60 training iterations and exponential dwell distribution. This setting matches the assumption underlying the joint posterior b(h, z) of Remark 4.5 more closely than the cyclic-continuous benchmark. Figure 2 plots the full training curves (mean over seeds with 95% bootstrap CI bands), and Table 2 reports the mean evaluation return averaged over the last 200 training iterations across all K = 4 test modes, on 1 500-iteration runs with cpenalty = 0.5 (Section 5.8, Q5). The headline result is on Ant (Fig. 2, right panel), where BAPR achieves a complete distributional separation from ESCP. On HalfCheetah (left panel), BAPR matches ESCP in mean and standard deviation. The Walker2d and Hopper entries confirm the env-difficulty hypothesis from the previous paragraph: both methods underperform regardless of belief representation in these settings under the 1 500-iter / K=4 piecewise-stationary protocol.
25k 20k 15k 10k 5k
ESCP BAPR (joint b(h, z), ours)
0 0
300
600 900 training iteration
1.2k
1.5k
Ant, K = 4 modes, N = 3 seeds 25k 20k 15k 10k 5k 0
ESCP BAPR (joint b(h, z), ours)
0
300
600 900 training iteration
1.2k
1.5k
Fig. 2: Training curves on the discrete-regime benchmark (K=4 modes, exponential dwell). Solid lines are mean evaluation returns (averaged across K=4 test modes per evaluation); shaded bands denote 95% bootstrap confidence intervals across seeds. Left: HalfCheetah (N =5 seeds). Right: Ant (N =3 seeds) — BAPR’s lower confidence bound exceeds ESCP’s upper confidence bound after iter ≈ 200, giving complete distributional separation throughout the rest of training. This is the clearest evidence that the joint regime belief b(h, z) provides actionable mode information that ESCP’s continuous context cannot recover from raw observations. (Curves use the original “BAPR (full)” configuration; see §5.5 for why we ultimately recommend dropping per-transition belief storage.) Table 2: Final performance (mean of last 200 iterations across the K = 4 test modes) on the discrete-regime benchmark with exponential dwell (60 iters/mode mean) and 1 500 training iterations. BAPR uses joint belief b(h, z) with |Z| = 4 and cpenalty = 0.5. “BAPR (full)” uses all four redesign components (§5.5); “BAPR (no per-trans belief)” is our recommended configuration that disables the per-transition belief storage in the replay buffer. Reported as mean±std across N seeds. Algorithm
Hopper (N =3)
Walker2d (N =3)
Ant (N =3)
SAC 15,682 ± 4,785 445 ± 61 ESCP 17,297 ± 3,356 565 ± 110 BAPR (full) 18,598 ± 7,872 — BAPR (no per-trans) 17,862 ± 3,327 420 ± 23
310 ± 6 374 ± 75 323 353 ± 26
17,171 ± 194 4,600 ± 4,406 20,284 ± 3,564 20,679 ± 3,402
+14% −6%
+20% +349%
Recommended vs. SAC Recommended vs. ESCP
HalfCheetah (N =5)
−6% −26%
+14% (n.s.) +3.3% (n.s.)
Walker2d and Hopper now reported as N =3 from a follow-up multi-seed run; both methods plateau in the same 300–650 band that the perturbation-environment sweep (§5.4) shows is the protocol ceiling. “n.s.” = not statistically significant given cross-seed std. SAC is included as the regime-unaware baseline: it learns a single robust policy from the mixed-mode data without any change-point detection or context conditioning.
13
HalfCheetah: matched mean, matched stability. With the recommended configuration, BAPR’s mean (17 862) is statistically indistinguishable from ESCP’s (17 297, paired t-test p ≫ 0.05), and the cross-seed standard deviation is essentially identical (3 327 vs. 3 356). The wider 7 872 std reported by “BAPR (full)” was caused by a single design choice — storing the belief vector per replay transition — whose effect is analyzed in detail in §5.5. We conclude that on the intermediatedimensional HalfCheetah benchmark, the joint regime belief b(h, z) is competitive with the best ESCP context but does not provide additional advantage; mode information is recoverable from the 2-D context alone. Ant: complete distributional separation. On all 3 Ant seeds of “BAPR (full)”, returns lie in [17 872, 24 437] while ESCP returns lie in [1 634, 9 824]; the worst BAPR seed exceeds the best ESCP seed by +82%, giving zero overlap between the two distributions. The recommended (no per-trans) configuration matches “BAPR (full)” within seed noise (mean 20 679 ± 3 402 vs. 20 284 ± 3 564, N =3), and exceeds the best ESCP seed on every Ant seed tested. We attribute this to the high-dimensional (27-D obs, 8-D act) coordination requirement of Ant: when regimes change body mass, gravity, or joint damping multiplicatively, the µ(z) channel of the joint belief lets the critic condition on a consistent regime indicator across re-visits to the same regime, while ESCP’s continuous context embedding must re-infer regime identity from raw observations alone. Walker2d / Hopper: env-difficulty. On Walker2d (bipedal balance) and Hopper (single-foot balance), all methods fail to climb above ∼ 550 return under 1 500 iters with the K=4 exponentialdwell schedule. We verified Walker2d failure breadth on 11 BAPR Walker2d runs (varying seed, mode set, dwell) — every peak return < 400. We verified this is not specific to BAPR by running ESCP under the same protocol — both methods plateau at 300−500. To rule out the more interesting hypothesis — that our specific mode set was too aggressive — we conducted a broad perturbation-environment sweep (§5.4): 14 alternative configurations spanning K ∈ {2, 3, 4}, perturbation amplitudes ±5% to ±50%, all three modulated dimensions (gravity, body mass, joint damping), and dwell times {200, 300, 400, 1000} iterations. Across the entire grid, peak Walker2d return remained in [287, 429] and peak Hopper return in [367, 542]; no configuration broke the ∼ 550 ceiling, and no configuration approached the ∼ 1 500–2 000 levels that vanilla SAC, TD3, BAC, or DSAC reach in the matched single-mode brax–spring baseline (Section 5.4, Table 3). We conclude that the bipedal/monopedal balance problem under any non-trivial regime-switching protocol is fundamentally outside the budget of 1 500 iters in our setup, requiring either much longer training, careful curricula, or single-mode pre-training. We retain the original entries in Table 2 but treat them as env-difficulty observations rather than evidence about the methods themselves.
5.4
Perturbation-environment sweep on Walker2d / Hopper
To check whether the mode set in Table 2 (parameter multipliers up to ×2.0 and ×0.5) was simply too aggressive for Walker2d/Hopper, we swept a broad family of milder perturbation environments. Table 3 reports the peak evaluation return (maxt eval_Rt ) of BAPR (recommended config) under each sweep condition, with seed 0 and a 1 000–2 000-iter training budget. Each entry was launched with a two-stage feasibility filter (kill if peak < stage 1 threshold by iter 278/168, or peak < 2000 by iter 1 000); the early-kill mechanism consistently triggered on every config, confirming that none of the milder mode sets evade the same plateau.
14
Table 3: Perturbation-environment sweep on Walker2d/Hopper. “Variant” encodes the perturbation set: “gmild ” = 4 gravity-only modes (×0.85 to ×1.20); “dmild ” = 4 damping-only modes (×0.7 to ×1.5); “mmild ” = 4 mass-only modes (×0.85 to ×1.30); “mixed” = 4 combined modes; “K2d ” = binary {normal, d×0.7}; “K3g5% ” = 3 modes with gravity ±5%; “K4g5% ” = 4 modes with gravity ±5%; “K4dtiny ” = 4 modes with damping ±15%/+25%. “dwell” = mean iters per mode (exponential schedule). Peak R = maxt eval_Rt across the entire training run; none breaks the ∼ 550 ceiling. For comparison, the matched stationary single-mode baselines (SAC/TD3/BAC/DSAC, 1 999 iters) plateau at 548/470 (Hopper/Walker2d, slowest-among-working algo TD3) up to 839/649 (DSAC). Env
Variant / dwell
Peak R
Outcome
Walker2d Walker2d Walker2d Walker2d Walker2d Walker2d Walker2d Walker2d Walker2d Walker2d
gmild / dw200 dmild / dw200 mmild / dw200 mixed / dw200 K2d / dw400 K3g5% / dw300 K4g5% / dw200 K4dtiny / dw400 K2d / dw1000 K3g5% / dw1000
∼ 350 ∼ 320 ∼ 280 ∼ 270 ∼ 290 350 ∼ 360 346 379 429
stage-1 kill stage-1 kill stage-1 kill stage-1 kill stage-1 kill stage-2 kill stage-1 kill stage-2 kill stage-2 kill stage-2 kill
Hopper Hopper Hopper Hopper Hopper Hopper Hopper Hopper
gmild / dw200 dmild / dw200 mmild / dw200 mixed / dw200 (orig) K2d / dw400 K3g5% / dw300 K3g5% / dw1000 K2d / dw1000
∼ 380 ∼ 470 ∼ 380 519 (s0 ran to 800, mean200 =480) 396 493 424 542
stage-1 kill stage-1 kill stage-1 kill seed-0 completed stage-1 kill stage-2 kill stage-2 kill stage-2 kill
“Stage-1 kill” = peak R never reached the lower data-driven threshold (329 Walker, 383 Hopper = 50–70% of slowest single-mode baseline plateau) by iter 168/278. “Stage-2 kill” = passed stage 1 but peak R < 2 000 by iter 1 000. For Hopper-mixed-orig, seed 0 completed its 800-iter budget at mean200 =480, peak 519; seeds 1−2 on the same configuration both passed stage 1 (peak 385 / 437 at iter 400) but were halted to free GPU before reaching the 1 000-iter stage-2 checkpoint.
The sweep behaves like an exhaustive null result for the alternative hypothesis “a milder mode set will solve it.” Pushing dwell to 1 000 iters (each mode held for ∼ 250k env steps, comparable to a full single-mode SAC training budget at our wall-clock) gave only marginal improvement (Hopper K2d peak 542 vs. 519; Walker2d K3g5% peak 429 vs. ∼ 350). The implication is that the difficulty is not the perturbation magnitude per se but the protocol: the bipedal/monopedal balance policy never gets to fully crystallize before the next regime switch, even with extremely long dwell, because the replay buffer permanently retains transitions from prior modes that become catastrophically off-distribution after each switch.
5.5
Ablation: per-transition belief storage
In our initial implementation we followed the “natural” off-policy correction: store the belief vector bt active at rollout time alongside each transition (st , at , rt , st+1 ) in the replay buffer, so that each off-policy critic update uses the belief that was active when that transition was generated rather than the current iteration’s belief. We expected this to remove the off-policy bias caused by belief drift; in practice, it consistently hurt performance. Table 4 reports the per-seed effect of disabling per-transition belief storage (no #2 below). Across all six tested (env, seed) pairs, removing the stored belief either matches or improves performance, with the largest gains on the seeds where the full configuration had collapsed.
15
Table 4: Per-seed effect of disabling per-transition belief storage in the replay buffer (“no #2”). Mean of last 200 iters; positive ∆ favors the no-#2 configuration. Env
Seed
BAPR (full)
BAPR (no #2)
Relative
HalfCheetah HalfCheetah HalfCheetah HalfCheetah HalfCheetah HalfCheetah
0 1 2 3 4 N=5 mean
21 055 18 273 8 005 10 358 21 781 15 894 ± 6 321
20 188 17 980 15 633 13 782 21 727 17 862 ± 3 327
−4% −2% +95% +33% ≈0 +12% mean, −47% std
Ant Ant Ant Ant
0 1 2 N=3 mean
16 623 — — 20 284 ± 3 564
24 228 17 447 20 363 20 679 ± 3 402
+46% — — +2% mean, −5% std
We attribute the negative effect of per-transition belief storage to a transient mismatch between early-training stored beliefs (before the regime tracker’s k-means clustering has converged) and the much-better-calibrated current belief used at update time. When stored, those early-noise belief vectors remain in the replay buffer for the rest of training; the critic learns spurious mappings from random belief vectors to actions that take many iterations to overcome. Broadcasting the current belief at update time bypasses this problem, at the cost of a tractable belief-drift bias that, empirically, does not hurt on any of the seeds tested. We therefore recommend “no #2” as the default configuration of the joint-belief BAPR variant.
5.6
Ablation study (Q2)
To isolate the contribution of each component, we evaluate: • BAPR w/o BOCD: Removes change-point detection; βeff = βbase is static. • BAPR w/o RMDM: Removes context-conditioning; the critic and policy receive no modeaware embedding. • BAPR w/o adaptive β: BOCD runs but λw is not fed into the LCB coefficient. • BAPR-FixedDecay: Replaces BOCD with a heuristic exponential-decay penalty triggered by reward drops. Tests whether the full Bayesian posterior tracking is necessary. All four ablation variants are registered as separate algorithm flags (–algo bapr_no_bocd, bapr_no_rmdm, bapr_no_adapt_beta, bapr_fixed_decay) in our public training script. Table 5 reports N =3-seed mean±std on HalfCheetah discrete-mode (original K=4 modes, all runs to 1,500 iters). Each variant disables exactly one component, holding all other hyperparameters at the values used for the recommended “no per-trans belief” configuration in Table 2. Table 5: Component ablation on HalfCheetah discrete-mode, N =3 seeds (mean±std of last 200 iters). Reference: full BAPR (no per-trans belief) on the same env reaches 17,862 ± 3,327 across N =5 seeds (Table 2). Variant
mean200 (N =3) vs. full
BAPR (full ref., N =5 Tab. 2)
17,862 ± 3,327
—
BAPR w/o adaptive β Bad-BAPR (Q-dependent βeff ) BAPR w/o BOCD BAPR-FixedDecay (BOCD→heur.) BAPR w/o RMDM
16,301 ± 3,267 16,147 ± 3,828 14,152 ± 3,661 12,279 ± 3,120 9,392 ± 1,073
−9% −10% −21% −31% −47%
16
Reading the ablation (N =3). RMDM is the largest single ablation (−47%): removing the context embedding e that the critic and policy condition on is what most damages performance. The BOCD machinery contributes a non-trivial −21% on average, and the cheap heuristic-decay replacement (BAPR-FixedDecay) further lags by −31% — the structured Bayesian posterior is therefore not a decorative substitute for a fixed schedule once we average over seeds. Removing the adaptive β pathway costs only −9%, and Bad-BAPR (Q-dependent βeff ) costs −10%, an essentially indistinguishable margin from the abstract-operator perspective: when the operating regime stays inside γ + λ∆ < 1 (the safe side of Theorem D.1), the unsafe Q-dependent variant neither catastrophically diverges nor improves on the frozen design, consistent with the threshold-based reading of the counterexample. The component ranking on HalfCheetah is therefore RMDM ≫ FixedDecay > BOCD > adaptive β ≈ Bad-BAPR.
5.7
BOCD dynamics and adaptive conservatism (Q3, Q4)
Figure 3 plots the BAPR BOCD trace on a representative run (Ant-v2, seed 2, recommended “no per-trans” configuration, 1,500 iterations). The four time-aligned panels show: (i) the true mode (background shading) overlaid with eval reward, (ii) the BOCD belief entropy H[ρt ] that proxies posterior concentration, (iii) the entropy temperature α used by the actor, and (iv) the effective belief window. Across the 22 mode switches in this run, BOCD detected 13 events at median delay 13 iterations (mean 14, 90th-percentile 22); the remaining 9 switches did not produce a belief-entropy excursion above the local pre-switch baseline within 50 iterations, an empirical false-negative rate of 41% at the threshold used here. This is the concrete behavioral signature of “Bayesian Amnesia”: a brief surge in posterior uncertainty after each switch, followed by re-concentration as the agent re-identifies the regime.
17
BAPR on Ant-v2 discrete-mode (seed 2): true mode (background) + return + BOCD signals Eval R
20000 10000
Belief H
0 3.0
H[ t]
2.8 2.6 2.4 (entropy temp)
Eff. window
0.15 0.10 0.05
effective window
8 6 0
200
400
600
800
Iteration
1000
1200
1400
Fig. 3: BOCD detection + adaptive-conservatism dynamics on Ant-v2 discrete-mode (seed 2, BAPR “no per-trans” configuration). Top: true mode (background) and per-iter eval return. Second: BOCD belief entropy H[ρt ], which spikes after each true-mode switch and decays as evidence accumulates. Third: entropy temperature α. Bottom: effective belief window. Across the 22 switches in this run, median detection delay is 13 iterations. We additionally probe the frozen-belief counterexample (Theorem D.1) by implementing a BadBAPR variant (–algo bad_bapr in the public training script) where βeff depends on current Q-values via the substitution \ a), βeff ← βbase − λw · Q-std(s, (22) which is exactly the failure mode the counterexample formalizes. Empirical observation: Bad-BAPR does not diverge on HC discrete-mode. Across N =3 seeds Bad-BAPR on HalfCheetah discrete-mode reaches mean200 = 16,147 ± 3,828, only −10% below the full BAPR reference of 17,862 and statistically indistinguishable from the BAPR-w/o-adaptive-β ablation (16,301 ± 3,267). This is consistent with the formal threshold of Theorem D.1: contraction is guaranteed only outside the regime γ + λ∆ ≥ 1, which we treat as an existence boundary rather than a universal collapse prediction. The theorem predicts contraction failure exactly when γ + λ∆ ≥ 1, where ∆ is the mode reward gap (Lean: Counterexample/sharp_threshold). On HalfCheetah the multiplicative perturbations (0.5–2.0)× on gravity, mass, and damping translate to a mode reward gap ∆ that, jointly with the empirically operating λ (λw rarely exceeds 0.4), keeps γ + λ∆ comfortably below 1 in this environment, so the counterexample’s worst-case behavior is not actually triggered — and within that subcritical regime the Q-dependent conservatism actually appears to help by giving the agent more aggressive exploration when uncertainty is genuinely state-dependent. The theory remains sharp — it predicts where contraction must fail — but this environment is below threshold. We therefore present Theorem D.1 as a sharp existence result rather than a guarantee that Q-dependent conservatism collapses every realistic implementation; constructing a benchmark with ∆ large enough to expose the failure (e.g., catastrophic-mode environments where wrong-regime 18
policies receive deeply negative reward) is left to future work. A representative BOCD posterior trace is included in the public training-script logging output (the belief_entropy and effective_window keys under logs/); a rendered visualization is deferred to the companion technical report.
5.8
Sensitivity analysis (Q5)
We evaluate sensitivity to: (i) surprise signal weights (wr , wq , wκ ), (ii) BOCD hazard rate Hhazard , (iii) penalty scale cpenalty , and (iv) context embedding dimension de . Penalty scale cpenalty . The effective conservatism coefficient βeff = βbase − λw · cpenalty has two failure modes at the extremes of cpenalty . We swept cpenalty ∈ {0.0, 0.5, 2.0} on HalfCheetah discrete-mode (seed 0, BAPR with joint b(h, z)). Table 6 reports the iter-300 snapshot, capturing the early-training regime where the differences between settings are most visible. Table 6: Penalty-scale sweep on HalfCheetah discrete-mode (seed 0, evaluation return at iteration 300). cpenalty = 0.5 is the sweet spot. cpenalty
Eval @ iter 300
Q-std @ iter 300
Behavior
0.0 0.5 2.0
5 771 12 936 7 100
≈ 0.10 ≈ 0.34 ≈ 0.16
ensemble collapse, plateau healthy ensemble, fast recovery over-conservative post-switch
With cpenalty = 0, the BOCD pathway is silenced: β never adapts and the critic ensemble collapses to near-zero disagreement, losing the effective uncertainty estimate that LCB relies on, so the policy plateaus. With cpenalty = 2, the actor becomes too conservative immediately after a regime switch (βeff as low as −3 at λw ≈ 0.5), and post-switch recovery is delayed by tens of iterations. Our main experiments use cpenalty = 0.5, which yields healthy ensemble disagreement while keeping post-switch recovery responsive. We caution that this sweet spot may be environment-dependent; per-environment retuning is left to future work. Other sensitivities. The remaining hyperparameters — surprise-signal weights (wr , wq , wκ ), BOCD hazard rate Hhazard , and context embedding dimension de — were held at the defaults reported in Appendix K. We did not observe qualitative regression to these settings during prototyping; full heatmap sweeps over the joint (w, Hhazard , de ) space are deferred to the companion technical report.
6
Conclusion
This paper presents the BAPR framework, a Bayesian adaptive approach to robust reinforcement learning in piecewise-stationary environments. By integrating Bayesian Online Change Detection with the disentangled robust ensemble framework of RE-SAC, BAPR achieves adaptive conservatism that responds in real-time to environmental regime changes while maintaining formal convergence guarantees. Our theoretical analysis establishes four key results: (1) the BAPR operator—a convex combination of mode-conditional Bellman operators weighted by the BOCD posterior—is a γ-contraction, inheriting the convergence guarantee of each constituent mode without additional assumptions; (2) the contraction guarantee transfers verbatim to a joint run-length / regime-cluster posterior b(h, z), enabling recurring-regime memory through the marginal µ(z) while preserving the original convergence proof (Remark 4.5); (3) the frozen-belief design is necessary: Q-dependent beliefs provably break contraction when the mode reward gap exceeds a critical threshold; and (4) the BOCD posterior converges to the correct mode in O(log(1/δ)) steps under a mode separability condition. All formal results are machine-verified in Lean 4 with no sorry. The context-conditioning module, trained via the RMDM loss, provides complementary mode identification capability, while the multi-signal surprise detector ensures that the BOCD belief is informed by both reward distribution shifts and epistemic uncertainty changes. The resulting system provides a principled bridge between Bayesian change-point detection and robust reinforcement 19
learning, enabling continuous adaptation without sacrificing the stability guarantees that the frozenparameter design provides. Empirically, BAPR’s joint regime belief produces a complete distributional separation from ESCP on Ant under the discrete-regime benchmark (+349% mean return on the recommended configuration, 20 679 ± 3 402 vs. 4 600 ± 4 406, N =3, with min(BAPR) > max(ESCP) across all original-variant seeds), confirming the value of mode-aware Q-conditioning in high-dimensional coordination tasks. The advantage over a regime-unaware SAC baseline is more modest (+20%, N =3, SAC reaches 17 171 ± 194): the bulk of BAPR’s gain over ESCP appears to come from avoiding ESCP’s failure to recover a useful continuous context on Ant, rather than from BOCD-driven adaptive conservatism alone. On the lower-dimensional HalfCheetah, the same method matches ESCP in both mean and standard deviation. We additionally report a useful negative finding: storing the active belief vector per replay transition — the natural off-policy correction — hurts performance on every (env, seed) pair we tested, because early-training stored beliefs (from before the regime tracker has converged) introduce noise that persists in replay; broadcasting the current belief is empirically preferable. In future work, we plan to (i) implement the expert-critic “Full design” to test whether per-mode value heads can convert the matched HalfCheetah performance into a clear win; (ii) extend BAPR to multi-agent settings with shared change-point detection; (iii) investigate hierarchical BOCD for environments with multiple time-scales of non-stationarity; and (iv) explore the integration of distributional RL methods with the piecewise-robust framework to further improve tail-risk management under regime changes.
Acknowledgments This work was supported by the National Natural Science Foundation of China (Grant No. 72371251), the Natural Science Foundation for Distinguished Young Scholars of Hunan Province (Grant No. 2024JJ2080), and the Key Research and Development Program of Hunan Province of China (Grant No. 2024JK2007).
References [1] Adams Ryan P. and MacKay David J.C., “Bayesian online changepoint detection,” arXiv preprint arXiv:0710.3742, 2007. [2] Benjamins Carolin, Eimer Theresa, Schubert Frederik, Biedenkapp André, Rosenhahn Bodo, Hutter Frank, and Lindauer Marius, “CARL: A benchmark for contextual and adaptive reinforcement learning,” NeurIPS Workshop on Ecological Theory of RL, 2021. [3] Chen Baiming, Liu Zuxin, Zhu Jiacheng, Xu Mengdi, Ding Wenhao, Li Liang, and Zhao Ding, “Context-aware safe reinforcement learning for non-stationary environments,” in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2021, pp. 10 689–10 695. [4] Doshi-Velez Finale and Konidaris George, “Hidden parameter Markov decision processes: A semiparametric regression approach for discovering latent task parametrizations,” in Proc. Int. Joint Conf. Artificial Intelligence (IJCAI), 2016, pp. 1432–1440. [5] Finn Chelsea, Abbeel Pieter, and Levine Sergey, “Model-agnostic meta-learning for fast adaptation of deep networks,” in Proc. Int. Conf. Machine Learning (ICML), 2017, pp. 1126–1135. [6] Garivier Aurélien and Moulines Eric, “On upper-confidence bound policies for switching bandit problems,” in Proc. Algorithmic Learning Theory (ALT), 2011, pp. 174–188. [7] Haarnoja Tuomas, Zhou Aurick, Abbeel Pieter, and Levine Sergey, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proc. Int. Conf. Machine Learning (ICML), 2018, pp. 1861–1870.
20
[8] Hadoux Emmanuel, Beynier Aurélie, and Weng Paul, “Sequential decision-making under nonstationary environments via sequential change-point detection,” in ECML/PKDD Workshop on Learning over Multiple Contexts, 2014. [9] Iyengar Garud N., “Robust dynamic programming,” Mathematics of Operations Research, vol. 30, no. 2, pp. 257–280, 2005. [10] Khetarpal Khimya, Riemer Matthew, Rish Irina, and Precup Doina, “Towards continual reinforcement learning: A review and perspectives,” Journal of Artificial Intelligence Research, vol. 75, pp. 1401–1476, 2022. [11] Killick Rebecca, Fearnhead Paul, and Eckley Idris A., “Optimal detection of changepoints with a linear computational cost,” Journal of the American Statistical Association, vol. 107, no. 500, pp. 1590–1598, 2012. [12] Lecarpentier Erwan and Rachelson Emmanuel, “Non-stationary Markov decision processes: A worst-case approach using model-based reinforcement learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019, pp. 7214–7223. [13] Luo Fan-Ming, Jiang Shengyi, Yu Yang, Zhang Zongzhang, and Zhang Yi-Feng, “Adapt to environment sudden changes by learning a context sensitive policy,” in Proc. AAAI Conf. Artificial Intelligence, 2022, pp. 7637–7646. [14] Mellor Joseph and Shapiro Jonathan, “Thompson sampling in switching environments with Bayesian online change detection,” in Proc. Int. Conf. Artificial Intelligence and Statistics (AISTATS), 2013, pp. 442–450. [15] Nagabandi Anusha, Clavera Ignasi, Liu Simin, Fearing Ronald S., Abbeel Pieter, Levine Sergey, and Finn Chelsea, “Learning to adapt in dynamic, real-world environments through metareinforcement learning,” in Proc. Int. Conf. Learning Representations (ICLR), 2019. [16] Osogami Takayuki, “Robustness and risk-sensitivity in Markov decision processes,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 25, 2012, pp. 233–241. [17] Padakandla Sindhu, “A survey of reinforcement learning algorithms for dynamically varying environments,” ACM Computing Surveys, vol. 54, no. 6, pp. 1–25, 2021. [18] Panaganti Kishan, Wierman Adam, and Mazumdar Eric, “Model-free robust ϕ-divergence reinforcement learning using both offline and online data,” in Proc. Int. Conf. Machine Learning (ICML), 2024. [19] Rakelly Kate, Zhou Aurick, Finn Chelsea, Levine Sergey, and Quillen Deirdre, “Efficient off-policy meta-reinforcement learning via probabilistic context variables,” in Proc. Int. Conf. Machine Learning (ICML), 2019, pp. 5331–5340. [20] Sodhani Shagun, Zhang Amy, and Pineau Joelle, “Multi-task reinforcement learning with contextbased representations,” in Proc. Int. Conf. Machine Learning (ICML), 2021, pp. 9767–9779. [21] Wiesemann Wolfram, Kuhn Daniel, and Rustem Berç, “Robust Markov decision processes,” Mathematics of Operations Research, vol. 38, no. 1, pp. 153–183, 2013. [22] Xu Huan, Caramanis Constantine, and Mannor Shie, “Robust regression and Lasso,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 21, 2008, pp. 1801–1808. [23] Xu Huan, Caramanis Constantine, and Mannor Shie, “Robustness and regularization of support vector machines,” Journal of Machine Learning Research, vol. 10, pp. 1485–1510, 2009. [24] Zhang Yifan and Zheng Liang, “RE-SAC: Disentangling aleatoric and epistemic risks in bus fleet control: A stable and robust ensemble DRL approach,” arXiv preprint arXiv:2603.18396, 2026.
21
[25] Zhang Yifan, “Single Agent Robust Deep Reinforcement Learning for Bus Fleet Control,” Transportation Safety and Environment, article tdag005, 2026, doi: https://doi.org/10.1093/tse/ tdag005. [26] Zhou Ruida, Liu Tao, Cheng Min, Kalathil Dileep, Kumar P. R., and Tian Chao, “Natural actor-critic for robust reinforcement learning with function approximation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023. [27] Zintgraf Luisa, Shiarlis Kyriacos, Igl Maximilian, Schulze Sebastian, Gal Yarin, Hofmann Katja, and Whiteson Shimon, “VariBAD: A very good method for Bayes-adaptive deep RL via meta-learning,” in Proc. Int. Conf. Learning Representations (ICLR), 2020. [28] Fei Yingjie, Yang Zhuoran, Chen Yudong, Wang Zhaoran, and Xie Qiaomin, “Risk-sensitive reinforcement learning: Near-optimal risk-sample tradeoff in regret,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020, pp. 22 384–22 395. [29] Blackwell David, “Discounted dynamic programming,” The Annals of Mathematical Statistics, vol. 36, no. 1, pp. 226–235, 1965.
22
Appendix: Theoretical foundations and derivations
A
Piecewise Q-value convergence rate
We analyze how BAPR’s piecewise-robust design achieves polynomial convergence rates, in contrast to the exponential barrier faced by direct risk-sensitive optimization. Unlike the informal “regret” framing common in deep RL theory papers, we provide a convergence rate guarantee—measuring how fast ∥Qt − Q∗ ∥∞ decays—which is the natural theoretical tool for contraction-based algorithms. Every component of the bound below corresponds to a machine-verified Lean 4 theorem.
A.1
Risk-sensitive RL: The exponential barrier (review)
As established by Fei et al. [28], the regret lower bound for any algorithm learning a risk-sensitive policy under exponential utility is exponential in the risk parameter β and horizon H: √ (23) RegretRS (T ) ≥ Ω(exp(|β|H) · S 2 AT ). BAPR avoids this barrier through the same mechanism as RE-SAC [24]: by replacing tail-distribution estimation with frozen structural penalties, converting the problem to a standard (shifted-reward) MDP with γ-contraction.
A.2
Formal error budget
The BAPR convergence rate decomposes into five components, each with a machine-verified bound:
A.3
Component
Bound
Lean theorem
Contraction rate Function approx. Sampling noise Detection delay Regime perturbation
γ decay per step εproj /(1 − γ) floor σ/(1 − γ) floor nδ = O(log(1/δ)/ log L) steps ∆R /(1 − γ) shift
bapr_contraction approx_fixed_point_bound stochastic_tracking_bound detection_delay_sufficient regime_switch_perturbation
n
Regime switch perturbation bound
When a regime switch occurs, the fixed point of the Bellman operator changes from Q∗k (under Tk ) to Q∗k+1 (under Tk+1 ). We bound the fixed-point shift: Lemma A.1 (Regime switch perturbation — Lean: regime_switch_perturbation). Let ∆R = maxs,a |Tk+1 Q∗k (s, a)−Tk Q∗k (s, a)| be the pointwise operator difference (capturing reward and transition changes between modes). Then: ∆R ∥Q∗k − Q∗k+1 ∥∞ ≤ . (24) 1−γ Proof. Since Q∗k = Tk Q∗k and Q∗k+1 = Tk+1 Q∗k+1 : ∥Q∗k − Q∗k+1 ∥ = ∥Tk Q∗k − Tk+1 Q∗k+1 ∥ ≤ ∥Tk+1 Q∗k − Tk+1 Q∗k+1 ∥ + ∥Tk Q∗k − Tk+1 Q∗k ∥ {z } | {z } | ∗ ≤γ∥Q∗ k −Qk+1 ∥
≤ γ∥Q∗k − Q∗k+1 ∥ + ∆R .
≤∆R
(25)
Rearranging: (1 − γ)∥Q∗k − Q∗k+1 ∥ ≤ ∆R . This algebraic step is machine-verified in Lean as regime_switch_perturbation, which applies approx_fixed_point_bound with ∆R in the role of εproj . 23
A.4
Piecewise Q-value convergence rate theorem
Theorem A.2 (Piecewise Q-value convergence rate — formal). Consider a piecewise-stationary environment with N regime switches. Under Assumptions B.1–B.3, the Q-value error after each regime switch at time tk satisfies: 1. Phase 1 (Detection): For t ∈ [tk , tk + nδ ), the error is bounded by: ∥Qt − Q∗k+1 ∥∞ ≤ Eswitch :=
∆R + εproj + σ . 1−γ
(26)
2. Phase 2 (Contraction): For t ≥ tk + nδ , the error decays geometrically: ∥Qt − Q∗k+1 ∥∞ ≤ γ t−tk −nδ · Eswitch +
εproj + σ . 1−γ
(27)
3. Phase 3 (Steady state): As t → ∞ within the segment: ∥Qt − Q∗k+1 ∥∞ ≤
εproj + σ . 1−γ
(28)
The detection delay is nδ = ⌈log(r0 /δ)/(2 log L)⌉ = O(log(1/δ)) steps. The total number of “recovery steps” (where ∥Qt − Q∗ ∥ > ε) across T total steps with N switches is: log(Eswitch · (1 − γ)/ε) log(1/δ) log(∆R /ε) Nrecover = N · nδ + =N ·O + , (29) log(1/γ) log L 1−γ which is polynomial in all parameters (no exponential dependence on H or β). Proof. Each phase maps to a Lean-verified building block: • Phase 1: The initial error Eswitch combines the regime perturbation (Lemma A.1, Lean: regime_switch_perturbation) with the steady-state floor (Lean: steady_state_decomposition). • Phase 2: Once BOCD converges (Lean: detection_delay_sufficient) and the belief stabilizes (Lean: bapr_contraction_after_update), the combined approximation-stochastic bound (Lean: combined_approx_stochastic_bound) gives the geometric decay. • Phase 3: The irreducible floor is the steady-state decomposition (Lean: steady_state_decomposition). • Recovery steps: The detection delay is bounded by Lean: detection_delay_sufficient. The contraction recovery time follows from solving γ n · Eswitch ≤ ε/(1 − γ), giving n ≥ log(Eswitch (1 − γ)/ε)/ log(1/γ). No informal steps are required; each inequality corresponds to a machine-verified theorem. Remark A.3 (Why convergence rate, not regret). We P deliberately state our result as a convergence rate (∥Qt − Q∗ ∥∞ decay) rather than a regret bound ( t V ∗ − V πt accumulation). A formal regret bound would require exploration-exploitation analysis (how quickly the policy visits informative states), which SAC—like all off-policy deep RL methods—does not provide. The convergence rate is the natural and rigorous theoretical tool for contraction-based algorithms, and it directly implies policy quality bounds via ∥V πt − V ∗ ∥ ≤ 2∥Qt − Q∗ ∥/(1 − γ). Remark A.4 (Comparison to static robust methods). A static robust method (e.g., Robust MDP with a fixed uncertainty set calibrated for the worst-case regime) incurs a persistent suboptimality of Ω(∆) per step in favorable regimes. Over T steps with N ≪ T switches, BAPR’s recovery cost N · O(log(1/δ)/ log L + log(∆R /ε)/(1 − γ)) is vastly smaller than the static cost Ω(∆ · T ).
24
B
Piecewise convergence analysis
While Theorem C.6 establishes per-step contraction, a piecewise-stationary environment introduces a new challenge: the BAPR operator itself changes at regime boundaries. This section analyzes the convergence behavior across regime switches using three formal assumptions from the BA-PR design methodology.
B.1
Structural assumptions
Assumption B.1 (Mode Separability). The environmental regimes {Mk } are L-separable with L > 1: after a switch from Mk to Mk+1 , the surprise signals generated under the new regime produce a likelihood ratio ≥ L favoring shorter run-lengths. Justification: In transit control, switching from “normal traffic” to “severe congestion” produces clear reward distribution shifts. In robotic locomotion, switching gravity levels produces detectable Q-value disagreement spikes. The separability parameter L quantifies the informativeness of the surprise signal. Assumption B.2 (Lipschitz Surprise Bound). The surprise function ξ : O → R+ (where O is the observation space comprising reward and Q-std statistics) is Lipschitz continuous and monotonically increasing in the magnitude of the regime change: |ξ(Mk+1 ) − ξ(Mk )| ≤ Lip(ξ) · d(Mk+1 , Mk ),
(30)
where d(·, ·) is a metric on the regime parameter space. Justification: The surprise signal is a smooth function of reward z-scores and Q-std ratios, both of which are Lipschitz in the underlying environment parameters. Assumption B.3 (Metastable Period). The minimum inter-switch interval Tmin = mink (tk+1 − tk ) satisfies: 1 Tmin ≫ + nδ , (31) 1−γ where 1/(1 − γ) is the γ-contraction convergence time scale and nδ is the BOCD detection delay. Justification: Abrupt regime changes (traffic incidents, weather shifts) are “rare” events separated by extended normal operation. The system is not continuously changing; it is piecewise stationary. This assumption is standard in the piecewise MDP literature [17, 12].
B.2
Piecewise convergence theorem
Theorem B.4 (Piecewise convergence via structural induction). Under Assumptions B.1–B.3, the BAPR training process within each stationary segment [tk , tk+1 ) can be decomposed into two phases: 1. Perturbation recovery phase (nδ steps): The BOCD posterior converges to the correct mode, and βeff reaches its maximal conservatism, providing protection during the transient. 2. Contraction phase (Tk − nδ steps): The frozen-belief BAPR operator contracts toward the segment-specific fixed point Q∗k at rate γ: ∥Qt − Q∗k ∥∞ ≤ γ t−tk −nδ · ∥Qtk +nδ − Q∗k ∥∞ +
εproj + σ . 1−γ
(32)
By Assumption B.3, the contraction phase has sufficient duration for the error to decay to the irreducible floor (εproj + σ)/(1 − γ) before the next switch. Proof sketch. The proof proceeds by structural induction over the sequence of segments: Base case: The first segment [t0 , t1 ) has no prior contamination. The BAPR operator with uniform initial belief is a γ-contraction (Theorem C.6), and the combined approximation-stochastic bound (Theorem F.4) guarantees convergence. Inductive step: At switch time tk+1 : 25
• The surprise signal spikes due to the reward/Q-std distribution shift (Assumption B.1). • The BOCD posterior shifts to higher expected run-length h̄ within nδ = O(log(1/δ)/ log L) steps (Theorem E.2); this raises h̄/(H−1) above the EMA baseline and triggers λw > 0. • βeff becomes maximally conservative, providing pessimistic protection during the transient. R with the updated belief ρ′ is again a γ-contraction • Once the belief stabilizes, the operator TρBAP ′ (Corollary C.9).
• By Assumption B.3, there are at least 1/(1−γ) steps remaining before the next switch, sufficient for convergence to the new fixed point.
C
Formal contraction proof: BAPR operator
We provide a self-contained, machine-verified proof that the BAPR operator TρBAP R is a γ-contraction in the L∞ norm. The proof is fully mechanized in Lean 4 / Mathlib (BAPR.lean) with no sorry. Key structural insight. The BAPR operator differs from RE-SAC in that it is a weighted mixture of mode-conditional operators, not a single operator with fixed penalties. The proof strategy exploits a fundamental mathematical property: a convex combination of γ-contractions is itself a γ-contraction. This is not merely a corollary of RE-SAC’s proof—it requires: 1. Establishing both Blackwell conditions (monotonicity and discounting) for each mode-conditional operator; 2. Proving P that the convex combination preserves both conditions (non-trivial for discounting: requires ρ = 1); 3. Verifying that the Bayesian belief update preserves the probability distribution properties. Notation. The Lean 4 proof uses h ∈ H (run-length) as the mode index for implementation convenience, since BAPR’s original design indexes modes by run-length. In the paper, we use m ∈ M to emphasize the conceptual distinction between modes (physical configurations) and run-lengths (temporal indices), as discussed in Remark 4.1. The mathematical content is identical; only the symbol names differ. Setup. Let S, A, and M be finite, non-empty state, action, and mode spaces. Let Q = {Q : S × A → R} with the sup-norm ∥Q∥∞ = maxs,a |Q(s, a)|. Mode-conditional operator. For each mode h ∈ H: ! X ′ Q ′ Th Q(s, a) = Rh (s, a) + γ Ph (s |s, a)V (s ) − λepi Γepi,h (s, a) − κ , (33) s′
where V Q (s′ ) = maxa′ Q(s′ , a′ ). BAPR operator. The belief-weighted mixture: X TρBAP R Q(s, a) = ρ(h) · Th Q(s, a). h∈H
We require the following conditions: (A0) (Discount factor) 0 ≤ γ < 1. (A1) (Non-negative transitions) Ph (s′ |s, a) ≥ 0 for all h, s, a, s′ . P (A2) (Probability kernel) s′ Ph (s′ |s, a) = 1 for all h, s, a. (A3) (Non-negative belief ) ρ(h) ≥ 0 for all h. P (A4) (Probability belief ) h ρ(h) = 1. 26
(34)
C.1
Per-mode contraction (inherited from RE-SAC)
Each mode-conditional operator Th shares the same algebraic structure as the RE-SAC operator with frozen penalties. The following results are structurally identical to RE-SAC’s proofs but parameterized by the mode h. Lemma C.1 (Per-mode monotonicity — Lean: t_mode_monotonicity). If Q1 ≤ Q2 pointwise, then Th Q1 ≤ Th Q2 pointwise for all h. is monotone, Q1 ≤ Q2 implies V Q1 (s′ ) ≤ V Q2 (s′ ). Each Ph (s′ |s, a) ≥ 0 by (A1), Proof. P SinceQ1maxP so s′ Ph V ≤ s′ Ph V Q2 . The frozen penalties Γepi,h and κ cancel. Multiplying by γ ≥ 0 (A0) gives the claim. Lemma C.2 (Per-mode discounting — Lean: t_mode_discounting). For any Q and constant c: Th (Q + c) = Th Q + γc. P Proof. Since V Q+c (s′ ) = V Q (s′ )+c (the argmax is unchanged by a uniform shift), and s′ Ph (s′ |s, a) = 1 by (A2): ! X Q ′ Th (Q + c)(s, a) = Rh + γ Ph (V (s ) + c) − λepi Γepi,h − κ s′
= Th Q(s, a) + γc ·
X
Ph = Th Q(s, a) + γc.
s′
| {z } =1
Lemma C.3 (Per-mode pointwise bound — Lean: t_mode_pointwise_bound). If ∥Q1 − Q2 ∥∞ ≤ ε, then |Th Q1 (s, a) − Th Q2 (s, a)| ≤ γε for all h, s, a. Proof. The frozen penalties (Rh , Γepi,h , κ) cancel in the difference. By the 1-Lipschitz property of max (Lean: max_over_a_nonexpansive): |V Q1 (s′ ) − V Q2 (s′ )| ≤ ε. |Th Q1 (s, a) − Th Q2 (s, a)| = γ
X
≤γ
X
Ph (s′ |s, a)(V Q1 (s′ ) − V Q2 (s′ ))
s′
Ph |V Q1 (s′ ) − V Q2 (s′ )| ≤ γ
s′
C.2
X
Ph · ε = γε.
s′
BAPR contraction via convex combination
This is where the BAPR proof departs from RE-SAC. The key novelty is establishing that both Blackwell conditions are preserved P through the belief-weighted mixture, and that this preservation requires the structural assumption ρ = 1. Lemma C.4 (BAPR Monotonicity — Blackwell (i) — Lean: bapr_monotonicity). If Q1 ≤ Q2 pointwise, then TρBAP R Q1 ≤ TρBAP R Q2 pointwise. Proof. By per-mode monotonicity (Lemma C.1), Th Q1 (s, a) ≤ Th Q2 (s, a) for all h. Since ρ(h) ≥ 0 by (A3): X X ρ(h)Th Q1 (s, a) ≤ ρ(h)Th Q2 (s, a). (35) h
h
Lemma C.5 (BAPR Discounting — Blackwell (ii) — Lean: bapr_discounting). For any Q and constant c: TρBAP R (Q + c) = TρBAP R Q + γc.
27
Proof. By per-mode discounting (Lemma C.2): X TρBAP R (Q + c)(s, a) = ρ(h)(Th Q(s, a) + γc) h
= TρBAP R Q(s, a) + γc ·
X
ρ(h) .
h
| {z }
=1 by (A4)
P Note: This is where ρ = 1 (A4) is load-bearing. Without it, the discounting lemma fails and Blackwell’s conditions are not met. This is the mathematical reason why the belief must be a proper probability distribution. Theorem C.6 (BAPR Contraction — Main Theorem — Lean: bapr_contraction). Under (A0)– (A4), TρBAP R is a γ-contraction on (Q, ∥ · ∥∞ ) with a unique fixed point Q∗BAPR . Proof. By Blackwell’s Theorem [29], Lemmas C.4–C.5 suffice. The Lean proof provides the direct L∞ bound. Let ε = ∥Q1 − Q2 ∥∞ : TρBAP R Q1 (s, a) − TρBAP R Q2 (s, a) =
X
(factor out difference)
ρ(h) (Th Q1 (s, a) − Th Q2 (s, a))
h
≤
X
ρ(h) |Th Q1 (s, a) − Th Q2 (s, a)|
(triangle ineq., ρ ≥ 0 by (A3))
h
≤
X
ρ(h) · γε
(per-mode bound, Lemma C.3)
h
= γε.
(
X
ρ = 1 by (A4))
(36)
Taking the supremum over (s, a) gives ∥TρBAP R Q1 − TρBAP R Q2 ∥∞ ≤ γ∥Q1 − Q2 ∥∞ . By the Banach Fixed-Point Theorem (γ < 1), TρBAP R has a unique fixed point. Remark (non-triviality of the convex combination argument). A referee might observe that once each Th is a γ-contraction, the convex combination result is “obvious.” However, the theoretical contribution lies in the chain of structural requirements: 1. The belief ρ must be non-negative (for monotonicity preservation via triangle inequality); 2. The belief must sum to one (for discounting preservation, the load-bearing step); 3. The belief must be frozen during the Bellman backup (for the penalties to cancel—without this, the counterexample in §D applies); 4. The Bayesian update must preserve properties (1)–(2) at every step. The Lean proof makes each of these requirements explicit and machine-verified. Violating any one of them would break the proof (and indeed the contraction, as our counterexample demonstrates for condition (3)).
C.3
Preservation of contraction under Bayesian belief updates
Definition C.7 (Bayesian P Belief Update — Lean: update_belief). Given prior ρ, likelihood L : H → R≥0 , and Z = h ρ(h)L(h) > 0: ρ′ (h) =
ρ(h) · L(h) . Z 28
(37)
Lemma C.8 (Lean: update_belief_nonneg + update_belief_sum_one). If ρ(h) ≥ 0, L(h) ≥ 0 P for all h, and Z > 0, then ρ′ (h) ≥ 0 for all h and h ρ′ (h) = 1. Corollary C.9 (BAPR contraction after belief update — Lean: bapr_contraction_after_update). Under the conditions of Theorem C.6, if ρ is updated to ρ′ via a Bayesian update with Z > 0, then R TρBAP is a γ-contraction. ′ Proof. By Lemma C.8, ρ′ satisfies (A3)–(A4). Apply Theorem C.6 with ρ′ . This corollary is the critical bridge between RE-SAC (which provides the κ surprise signal) and BAPR (which uses it for belief updating): the contraction guarantee is automatically preserved through the belief update cycle.
D
Counterexample: Q-dependent belief weights break contraction
Theorem C.6 requires the belief ρ to be frozen during the Bellman backup. This section proves that this requirement is necessary: allowing ρ to depend on the Q-function being updated can destroy contractivity. The results are machine-verified in BAPR-Counterproof.lean.
D.1
The “bad” operator with Q-dependent mode weights
Consider a simplified 1-state, 1-action, 2-mode MDP. Let ρ1 (Q) = λ · Q be the belief weight on mode 1 (depending linearly on Q), and ρ2 (Q) = 1 − λ · Q. The resulting operator becomes: Tbad (Q) = ρ1 (Q) · (γQ + R1 ) + ρ2 (Q) · (γQ + R2 ) = γQ + R2 + λQ · (R1 − R2 ) = (γ + λ∆) · Q + R2 ,
(38)
where ∆ = R1 − R2 is the mode reward gap. Theorem D.1 (Non-contraction — Lean: T_bad_not_contraction). If γ ≥ 0, λ ≥ 0, ∆ ≥ 0, and γ + λ∆ ≥ 1, then Tbad is not a contraction: there exist q1 , q2 with |Tbad (q1 ) − Tbad (q2 )| ≥ |q1 − q2 |. Proof. Take q1 = 1, q2 = 0: |Tbad (1) − Tbad (0)| = |(γ + λ∆) · 1 + R2 − R2 | = |γ + λ∆| ≥ 1 = |q1 − q2 |. Theorem D.2 (Strict expansion — Lean: T_bad_expansion). If γ + λ∆ > 1, then Tbad strictly expands distances: no contraction factor k < 1 exists.
D.2
Physical interpretation and design implications
What this means for algorithm design. The counterexample models an agent that re-infers the environment mode based on its own value estimate at every Bellman backup. In a piecewise-stationary environment with distinct modes (high ∆), this creates a positive feedback loop: 1. The agent overestimates Q (due to sampling noise or stale data); 2. The Q-dependent belief shifts toward the higher-reward mode (ρ1 increases); 3. This inflates the Q-target further (mode 1 has R1 > R2 ); 4. The cycle amplifies, preventing convergence.
29
The critical threshold. The contraction factor is γ + λ∆. For typical RL parameters (γ = 0.99), any λ∆ ≥ 0.01 breaks contraction. Since ∆ (the mode reward gap) can be large in practice (e.g., ∆ = 50 between “normal traffic” and “severe congestion” in bus control), even a tiny sensitivity λ = 0.001 suffices to breach the threshold. Operator
Belief
Factor
Contraction?
Tbad TρBAP R
ρ = f (Q) ρ frozen
γ + λ∆ γ
× if γ + λ∆ ≥ 1 ✓ (γ < 1, always)
cf. RE-SAC: RESAC Tbad κ = f (Q) TκREV κ frozen
γ + λale γ
× if γ + λale ≥ 1 ✓ (γ < 1, always)
Parallel structure with RE-SAC. The BAPR counterexample is structurally analogous to RESAC’s counterexample (RESAC-Counterproof.lean), but with a different amplification mechanism: in RE-SAC, Q-dependent aleatoric penalties cause expansion; in BAPR, Q-dependent belief weights cause expansion. Both demonstrate that frozen parameters are necessary not merely convenient design choices. The fact that two independent amplification channels exist underscores the importance of the frozen-parameter design principle across robust RL architectures.
D.3
Sharp threshold characterization
The boundary γ + λ∆ = 1 is exact, not merely sufficient. We establish the complete picture: Theorem D.3 (Sharp boundary — Lean: T_bad_contraction_below_threshold, T_bad_exact_factor). The contraction factor of Tbad is exactly |γ + λ∆|: |Tbad (q1 ) − Tbad (q2 )| = (γ + λ∆) · |q1 − q2 |
for all q1 , q2 .
(39)
Therefore: • If γ + λ∆ < 1: Tbad IS a contraction with factor γ + λ∆ < 1 (contraction, but with degraded rate); • If γ + λ∆ = 1: Tbad is a non-expansive map (no convergence guarantee); • If γ + λ∆ > 1: Tbad strictly expands distances (divergence). This sharp characterization reveals a “phase transition”: the gap 1 − γ represents the stability margin that Q-dependent belief sensitivity λ∆ can erode. With frozen beliefs (λ = 0), the full margin is preserved regardless of the mode reward gap ∆. With Q-dependent beliefs, even tiny sensitivity (λ = 0.001) can breach the threshold when ∆ is large.
E
Detection delay bound
E.1
Posterior ratio dynamics
After a mode switch at time t0 , the BOCD posterior must converge to the correct mode. Under the Mode Separability Assumption B.1, the posterior ratio evolves predictably. Definition E.1 (Posterior Ratio — Lean: posterior_ratio). The posterior ratio after n steps of consistent evidence with likelihood ratio L > 1 and prior ratio r0 = ρ0 (hold )/ρ0 (hnew ) is: PR(n) =
ρn (hnew ) 1 = · L2n . ρn (hold ) r0
30
(40)
Theorem E.2 (Detection Delay Bound — Lean: detection_delay_sufficient). If modes are L-separable with L > 1, and we require confidence 1/δ (i.e., PR(n) ≥ 1/δ), the required number of steps satisfies: log(r0 /δ) n≥ . (41) 2 log L This is O(log(1/δ)) for fixed L and r0 . Proof. We need L2n /r0 ≥ 1/δ. Taking logarithms: 2n log L ≥ log(r0 /δ), giving n ≥ log(r0 /δ)/(2 log L). Machine-verified. Theorem E.3 (Monotonicity — Lean: detection_confidence_mono). The posterior ratio PR(n) is monotonically increasing in n when L ≥ 1.
E.2
Practical detection delay examples Scenario
L
r0
δ
nδ (steps)
Strong separability (gravity change) Moderate separability (congestion) Weak separability (friction change) Adversarial prior (r0 = 10)
5.0 2.0 1.2 2.0
1 1 1 10
0.05 0.05 0.05 0.05
≈ 0.9 ≈ 2.2 ≈ 8.2 ≈ 3.8
With typical parameters (L ≈ 2, uniform prior r0 = 1, δ = 0.05), the detection delay is approximately 2–3 iterations—matching the empirical observation that BAPR’s λw spikes within 2–3 iterations of a mode switch.
F
Function approximation and stochastic bounds
The contraction proofs operate in the tabular setting. In practice, BAPR uses neural network function approximation and mini-batch sampling. The operator-agnostic bounds from ApproxContraction.lean— shared with RE-SAC—bridge this gap. Since these results apply to any γ-contraction, they apply to TρBAP R without modification. Theorem F.1 (Projected contraction — Lean: projected_contraction). If T is a γ-contraction and Π is a non-expansive projection onto the neural network function class, then Π ◦ T is also a γ-contraction. Theorem F.2 (Approximation error bound — Lean: approx_fixed_point_bound). Let Q̃ be the fixed point of Π ◦ T and Q∗ the fixed point of T . Define εproj := ∥ΠQ∗ − Q∗ ∥∞ . Then: ∥Q̃ − Q∗ ∥∞ ≤
εproj . 1−γ
(42)
Theorem F.3 (Stochastic tracking bound — Lean: stochastic_tracking_bound). With per-step sampling noise bounded by σ ≥ 0, the tracking error satisfies: e(n) ≤ γ n · e(0) +
σ . 1−γ
(43)
Theorem F.4 (Combined bound — Lean: combined_approx_stochastic_bound). With both function approximation (drift εproj ) and mini-batch sampling (noise σ): ∥Qn − Q∗ ∥∞ ≤ γ n ∥Q0 − Q∗ ∥∞ +
31
εproj + σ . 1−γ
(44)
Corollary F.5 (Steady-state error — Lean: steady_state_decomposition). In steady state (n → ∞), the error converges to: εproj + σ e∞ ≤ . (45) 1−γ This cleanly separates: γ (algorithmic—our proofs), εproj (architectural—network capacity), σ (statistical—sample size). Remark F.6 (BAPR-specific error decomposition). For BAPR, the steady-state error has an additional contribution from the belief tracking: eBAPR ≤ ∞
εproj + σ + εbelief , 1−γ
(46)
where εbelief accounts for the difference between the frozen belief ρ and the “true” mode distribution. Under the Metastable Period Assumption, εbelief → 0 as the BOCD posterior converges (Theorem E.2). During detection delay, εbelief is bounded by 2Rmax (the worst-case Q-value mismatch from using the wrong mode).
G
Action-ranking preservation under adaptive penalty
A natural concern is whether the adaptive βeff mechanism could degenerate the policy. Proposition G.1 (Action-ranking preservation). The BAPR penalty βeff · σens (s, a) acts on the policy objective, where σens (s, a) is the ensemble standard deviation. Since βeff = βbase − λw · cpenalty modifies the LCB coefficient uniformly across all states and actions, and σens (s, a) depends on the (frozen) target ensemble: 1. If action a1 has lower ensemble disagreement than a2 at state s, the adaptive penalty amplifies the preference for a1 monotonically with λw . 2. The adaptive penalty cannot reverse the ranking among actions with equal ensemble disagreement. Quantitative Q-value depression. Following the same analysis as RE-SAC [24]: Q∗BAPR (s, a) = Q∗ (s, a) −
γ · ∆penalty , 1−γ
(47)
where ∆penalty includes the belief-weighted contribution. With the experimental configuration (γ = 0.99, λw ≤ 0.4 empirically, cpenalty = 0.5, |βbase | = 2), the maximum additional depression from the adaptive penalty is bounded by γ · 0.5 · σ̄ens /(1 − γ) ≈ 50 · σ̄ens , which remains small relative to the typical Q-value scale in our experiments. Proposition G.2 (Adaptive conservatism monotonicity — Lean: beta_eff_monotone_conservatism). For all λw ≥ 0 and cpenalty ≥ 0: βeff = βbase − λw · cpenalty ≤ βbase .
(48)
Furthermore, βeff is monotonically decreasing in λw (Lean: beta_eff_mono_in_lambda): higher surprise (larger λw ) leads to more conservative behavior. This machine-verified result formally guarantees that the adaptive mechanism is safe by construction: false positive change-point detections can only cause temporary over-conservatism, never unsafe (over-optimistic) behavior. The “Bayesian Amnesia” name reflects this one-directional modulation: the agent can only become more cautious after surprise, never less cautious than its RE-SAC baseline. Proposition G.3 (Context embedding equivalence — Lean: context_embedding_equiv). For any collection of per-mode Q-functions {Qm : S × A → R}m∈M , there exists a shared Q-function Qshared : M × S × A → R such that: Qshared (m, s, a) = Qm (s, a) 32
for all m, s, a.
(49)
When the context embedding ϕ : S → Rde is injective (different modes produce distinct embeddings), the shared critic Qϕ (s ⊕ ϕ(s), a) can perfectly represent all mode-conditional Q-functions. In this case, the practical scalar approximation (Remark 4.6) introduces no representational error, and the contraction guarantee of Theorem C.6 applies exactly to the implemented algorithm. This bridges the theory-practice gap: while the abstract operator assumes M independent critics, the shared critic with a sufficiently expressive context embedding can recover the same function class. The approximation error is bounded by the context module’s ability to separate modes—which is precisely what the RMDM loss (§4.4) is designed to enforce.
H
Scope and limitations
(S1) Per-step vs. learning-dynamics guarantee. Like all Bellman-operator contraction proofs in deep RL, our result is a per-step property: for any fixed belief snapshot ρ, the operator contracts. It does not guarantee end-to-end convergence of the full training loop. The piecewise convergence analysis (§B) provides additional structure via the Metastable Period Assumption, but a full nonstationary convergence proof remains an open problem. (S2) Mode separability assumption and graceful degradation. The detection delay bound (Theorem E.2) requires L > 1. In environments where regime changes are very subtle (small ∆), the detection delay grows as O(1/ log L), potentially exceeding the metastable period. Importantly, BAPR degrades gracefully rather than catastrophically in this regime: • Detection failure (L ≈ 1): If the change is too subtle to detect, λw does not spike, and βeff ≈ βbase . BAPR effectively reduces to RE-SAC, which is itself a robust agent. The agent loses the adaptive conservatism benefit but retains the static robustness baseline. • Context module as backup: Even when BOCD fails to detect a change, the RMDM-trained context embedding e may still capture the mode shift through state-based features, providing an alternative mode identification channel that does not rely on the surprise signal. • No false positive harm: By construction (βeff ≤ βbase ), false positive detections can only increase conservatism, never decrease it. The worst case is temporary over-conservatism, not unsafe behavior. (S3) BOCD truncation. The run-length posterior is truncated at H (default 20). Modes persisting longer than H steps are indistinguishable. For the contraction proof, this is immaterial (finite M). For detection quality, H should exceed the expected inter-switch interval. (S4) Three-level frozen-parameter chain. BAPR requires freezing at three levels for contraction: (i) the aleatoric penalty κ (from RE-SAC), (ii) the epistemic penalty Γepi (from RE-SAC), and (iii) the belief distribution ρ (new in BAPR). Each level has its own counterexample demonstrating necessity. This multi-level frozen-parameter design is the key architectural constraint that distinguishes BAPR from naive adaptive approaches. (S5) Theory-to-implementation gap. The contraction theorem (Theorem C.6) proves a property P of the abstract mixture m ρ(m)Tm , which assumes M independent operators. The implementation approximates this with a single shared critic conditioned on context embedding e and a scalar βeff (see Remark 4.6 in the main text). This is a standard gap in deep RL theory: all Bellman-contraction proofs assume tabular operators, while implementations use function approximation. The functionapproximation bounds in §F (from ApproxContraction.lean) provide quantitative control over this gap. (S6) Relationship to the ApproxContraction results. The function-approximation and stochastic tracking bounds from ApproxContraction.lean apply directly to the BAPR operator, since they are operator-agnostic. The projected contraction, approximation error bound, and stochastic tracking bound therefore carry over without modification, providing a complete error budget for practical deep RL with the BAPR operator. 33
I
Connection to RE-SAC and ESCP
I.1
BAPR as a generalization of RE-SAC
In a stationary environment with a single mode (|M| = 1), the belief is trivially ρ(m0 ) = 1, and BAPR reduces exactly to RE-SAC. The adaptive βeff also reduces to βbase (since λw → 0 in steady state).
I.2
BAPR as an extension of ESCP
ESCP [13] provides context-conditioning but uses a fixed LCB coefficient β. BAPR extends ESCP by adding BOCD belief tracking, adaptive β modulation, and formal contraction guarantees.
I.3
Ablation hierarchy Algorithm
Context
BOCD
Adaptive β
Formal guarantee
SAC RE-SAC ESCP BAPR (full)
× × ✓ ✓
× × × ✓
× × × ✓
γ-contraction (stationary) γ-contraction (stationary, robust) – γ-contraction (piecewise)
J
Lean 4 proof catalog
J.1
BAPR.lean (560 lines, 0 sorry)
Theorem/Lemma
Lean name
Role
Max monotonicity Max distributes over const Max 1-Lipschitz Per-mode monotonicity BAPR monotonicity Per-mode discounting BAPR discounting Per-mode pointwise bound BAPR contraction Belief non-negativity Belief normalization Post-update contraction
max_over_a_mono max_over_a_add_const max_over_a_nonexpansive t_mode_monotonicity bapr_monotonicity t_mode_discounting bapr_discounting t_mode_pointwise_bound bapr_contraction update_belief_nonneg update_belief_sum_one bapr_contraction_after_update
V Q is monotone in Q max(Q + c) = max(Q) + c |V Q1 − V Q2 | ≤ ∥Q1 − Q2 ∥∞ Blackwell (i) per mode Blackwell (i) for mixture Blackwell (ii) per mode Blackwell (ii) for mixture |Th Q1 − Th Q2 | ≤ γε Main: γ-contraction ′ ρ P(h)′ ≥ 0 ρ (h) = 1 Contraction after κ-driven update
New in this revision: βeff monotonicity βeff monotone in λw Context embedding equiv. Shared operator equiv.
beta_eff_monotone_conservatism beta_eff_mono_in_lambda context_embedding_equiv bapr_shared_equiv
βeff ≤ βbase (safety) More surprise ⇒ more conservative Shared critic ≡ per-mode critics Bridges theory and implementation
J.2
BAPR-Counterproof.lean (265 lines, 0 sorry)
34
Theorem
Lean name
Role
Non-contraction Strict expansion
T_bad_not_contraction T_bad_expansion
Q-dep. ρ: γ+λ∆ ≥ 1 ⇒ no contraction Q-dep. ρ: γ+λ∆ > 1 ⇒ expansion
New in this revision: Below-threshold contraction Exact factor
T_bad_contraction_below_threshold T_bad_exact_factor
γ+λ∆ < 1 ⇒ still contracts Factor = γ+λ∆ (sharp boundary)
Detection delay Confidence monotonicity
detection_delay_sufficient detection_confidence_mono
n ≥ log(r0 /δ)/(2 log L) PR monotonically increases
J.3
ApproxContraction.lean (320 lines, 0 sorry) — Shared with RE-SAC
Theorem
Lean name
Role
Projected contraction Approx. error bound Stochastic tracking Combined bound Steady-state decomp.
projected_contraction approx_fixed_point_bound stochastic_tracking_bound combined_approx_stochastic_bound steady_state_decomposition
Π ◦ T is γ-contraction ∥Q̃ − Q∗ ∥ ≤ εproj /(1 − γ) e(n) ≤ γ n e(0) + σ/(1 − γ) Joint approx. + noise e∞ ≤ B + (ε + σ)/(1 − γ)
New in this revision: Regime switch perturb. Piecewise convergence step
regime_switch_perturbation piecewise_convergence_step
∥Q∗k − Q∗k+1 ∥ ≤ ∆R /(1 − γ) Per-segment recovery bound
Total: 1,145 lines of Lean 4 / Mathlib across 3 files, 0 sorry, 22 machine-verified theorems.
35
K
Hyperparameter guideline
Parameter
Default
Description
Core RL (inherited from RE-SAC) γ 0.99 Discount factor τ 0.005 Target network EMA rate α 0.2 (auto) SAC entropy weight (auto-tuned) K 10 Ensemble size βbase −2.0 Base LCB coefficient λale 0.01 Aleatoric ℓ1 reg. weight βood 0.01 OOD penalty coefficient Context / RMDM (inspired by ESCP) de 2 Context embedding dimension rrbf 2.0 RBF kernel bandwidth wcons 50.0 Consistency loss weight wdiv 0.025 Diversity loss weight Nwarmup 50 iters Context injection warmup BOCD / Adaptive β (novel in BAPR) Hmax 20 Max run-length Hhazard 0.05 BOCD hazard rate σ02 0.1 Base surprise variance σg 0.05 Variance growth rate cpenalty 0.5 Penalty scale (βeff = βbase − λw cpenalty ); sweet spot per Sec. 5.8 αEMA 0.3 Surprise EMA smoothing αbaseline 0.95 λw baseline EMA rate (wr , wq , wκ ) (0.5, 0.3, 0.2) Surprise signal weights
36