RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms

arXiv:2610.32971v1 [cs.LG] 23 Oct 2026

Xi Chen† JHU

Fengqi Hu∗† JHU

Zeyu Shen∗‡ JHU

Jiaying Xu¶ UCSD

Shangding Gu‖ UC Berkeley

Anqi Liu† JHU

Laixi Shi§ JHU

Pan Xu∗∗ Duke

October 26, 2026

Abstract Robust reinforcement learning (RL) is critical for reliable real-world deployment, where policies may fail under diverse shifts and disruptions. Existing robust RL algorithms offer a rich set of mechanisms and principles, yet two major barriers limit their broader potential: limited algorithm reusability and an unclear empirical capability boundary. We introduce RobustRLlib, an algorithm-centric library and benchmark that addresses both challenges. It integrates 17 representative robust online, offline, and safe RL algorithms under a unified interface for substantially improved reusability, together with an enhanced evaluation platform spanning diverse real-world shifts and advanced humanoid and vision-language-action tasks. Systematic evaluation over 273 perturbation configurations provides an up-to-date capability map of current robust RL algorithms, revealing where they remain effective and where substantial robustness gaps persist. As an open-source, user-friendly platform, RobustRLlib offers a reproducible and extensible codebase for applying existing methods to new problems and developing new robust RL algorithms. The code and tutorial are available at https://robustrl-lib.github.io.

Keywords: Robust RL, benchmark, algorithm library, distribution shift, sim-to-real transfer Algorithm Library

Robust Online

Robust Offline Learner-centric

Data-centric

RFQI

RAMBO

ATLA

IQL

Generative

MOPO

Environment-centric

RSC-SAC

DR-PPO

ROMB

ATLA-PPO

DR-SAC

PLR-PVL

ATLA-SA-PPO

FWM

RSC-IQL

RORL

Learner-centric

Agent RAMU-PPO

SPiDR-PPO

RARL-PPO

Robust MDP

RARL-TRPO

SynthER

TD3+BC

Standard RL base

Disruptor toolbox

MDP Elements

Sources

Dynamic / Observation / Action / Reward

Modes

Robust Safe

Adversarial

Stochastic

SAC

Latency Parametric

Non-stationary

Sim. PPO

Environment

Real

Semantic Composition

Disrupt

Figure 1: The overview of RobustRLlib. ∗ The second and third authors contributed equally. † Department of Computer Science, ‡ Department of Applied Mathematics and Statistics, § Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA. ¶ University of California San Diego, La Jolla, CA 92093, USA. ‖ Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, Berkeley, CA 94720, USA. ∗∗ Department of Biostatistics and Bioinformatics, Duke University, Durham, NC 27705, USA.

1

Contents 1 Introduction

2

2 Related Work

3

3 RobustRLlib: An Algorithm-Centric Robust RL Benchmark 3.1 Robust RL algorithm library construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Expandable disruptor toolbox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5 5 7

4 Experimental Evaluation 4.1 Evaluation of the algorithm library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Evaluation under diverse deployment shifts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Extension to advanced robotic tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

9 10 11 12

5 Conclusion

13

A Benchmark construction 19 A.1 Benchmark Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 A.2 Disruptor toolbox: shift modes, operators and implementations . . . . . . . . . . . . . . . . . 19 B Evaluation protocol 21 B.1 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 B.2 Evaluation grids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 C Benchmark tasks and datasets

25

D Algorithms 27 D.1 Offline algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 D.2 Online algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 E Training Settings 30 E.1 Training protocol . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 E.2 Training budget alignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 F Hyperparameter settings 32 F.1 Online algorithm hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 F.2 Offline algorithm hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 F.3 Robust safe algorithm hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 G Additional results and analysis 39 G.1 Analysis of robust methods’ gains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 G.2 Analysis of task capability: On-policy vs Off-policy . . . . . . . . . . . . . . . . . . . . . . . . 40 G.3 Results of advanced robotic tasks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 G.4 Supplementary results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

1

Introduction

Despite remarkable advances in reinforcement learning (RL), its real-world deployment remains limited by robustness and safety concerns, particularly in high-stakes domains such as autonomous driving (Ding et al., 2023b), clinical trials (Liu et al., 2015), robotics (Li et al., 2021), and semiconductor manufacturing (Kozak et al., 2023). Policies trained in idealized environments can easily fail under distribution shifts or disruptions affecting various stages of the interaction, such as observations (Han et al., 2022; Sun et al., 2021; Xiong et al., 2022; Zhang et al., 2021; 2020), actions (Huang et al., 2017), or environment dynamics (Iyengar, 2005; Pinto et al., 2017), induced by diverse factors such as sim-to-real gaps (Pinto et al., 2017), model uncertainty

2

(Bertsimas et al., 2019), noise, or adversarial attacks (Klopp et al., 2017; Mahmood et al., 2018; Zhang et al., 2020). Robust RL therefore aims to develop policies that remain reliable under diverse types of disruptions and shifts, or their combinations in real-world scenarios (Arthur, 1991; Bertsimas et al., 2019; Tversky and Kahneman, 1974). A broad range of robust RL algorithms has shown strong performance in their specific type of disrupted application scenarios. More importantly, existing methods provide a rich set of robustness mechanisms and principles, including adversarial training (Pinto et al., 2017; Rigter et al., 2022; Zhang et al., 2021), dynamics randomization (Peng et al., 2018), robust value updates (Panaganti et al., 2022; Yang et al., 2022), and data augmentation (Ding et al., 2024; Lu et al., 2023; Yu et al., 2020b), offering substantial promise across a far broader range of applications and shift scenarios. However, realizing this broader potential faces two fundamental barriers: algorithm reusability and systematic characterization of the capability boundary. First, existing robust RL algorithms are fragmented across separate implementations and training pipelines, lacking a unified library or interface through which practitioners can readily try and compare a broad collection of robust RL methods on a new application. Second, the empirical capability boundary of current robust RL remains unclear: for a given application and distribution shift, we lack a systematic understanding of how well existing methods perform, when they remain effective, and where they fail. This is largely because the methods are typically evaluated in distinct environments and shift scenarios, making cross-method comparison difficult and performance beyond their original settings poorly understood. To address these two gaps, we introduce RobustRLlib, an algorithm-centric robust RL benchmark that brings together (1) a robust RL method library with reusable implementations to substantially lower the barrier to applying existing methods to new problems, and (2) a comprehensive evaluation pipeline across tasks and diverse real-world shift scenarios. Complementing existing robust RL benchmarks that primarily focus on developing evaluation tasks and shift scenarios (Dulac-Arnold et al., 2020; Gu et al., 2025; Zouitine et al., 2024), RobustRLlib focuses on making robust RL algorithms systematically accessible, comparable, and extensible, while providing a broader evaluation framework (Comparison in Appendix A.1). Our main contributions are summarized as follows: • A robust RL algorithm library. We integrate existing 17 representative robust online, offline, and safe RL algorithms, understanding and systematically organizing them by their learning methods, intended target shifts, and base algorithms for applicators’ easy reference. It provides a reusable foundation for applying existing methods to new applications and developing new robust RL algorithms. • An enhanced evaluation platform. Our extensible disruptor toolbox covers a broad range of distribution shifts, including MDP-element shifts in observations, actions, dynamics, and reward/cost, as well as control-loop latency and task-semantic shifts. It supports diverse shift modes, including stochastic, adversarial, and parametric perturbations with non-stationary schedules, enabling both isolated and compound-shift evaluation. The same disruptor interface further extends to massively parallel training in Isaac Lab (Mittal et al., 2025) and advanced vision–language–action (VLA) robotic tasks (Intelligence et al., 2025; Zhang et al., 2026). • An up-to-date capability map of robust RL algorithms. Evaluation over 273 perturbation configurations reveals that robust methods outperform standard references on average, but both groups deteriorate markedly as shifts intensify. A method’s claimed target shift does not reliably predict its empirical strengths, gains vary across tasks and shift sources, with online methods showing larger incremental effects than offline ones. By connecting diverse algorithm features to their measured strengths, this map provides guidance for designing robust RL methods for deployment shifts.

2

Related Work

Robust RL problems under diverse deployment shifts. Robust RL addresses deployment shifts affecting different components and stages of the agent–environment interaction, and prior work can be categorized accordingly. Oikarinen et al. (2021); Zhang et al. (2021; 2020) address observation shift, which perturbs policy inputs through sensing errors or external attacks. Tan et al. (2020); Tessler et al. (2019) address action shift, which perturbs or replaces executed actions. Dynamics shift, involving uncertainty in 3

or changes to the transition kernel, is addressed either by optimizing worst-case performance over ambiguity sets (Goyal and Grand-Clement, 2023; Iyengar, 2005; Nilim and El Ghaoui, 2005; Shi and Chi, 2024; Shi et al., 2026; Wang et al., 2024; Wiesemann et al., 2013; Xu and Mannor, 2010) or by introducing random or adversarial perturbations to physical parameters and external forces (Mankowitz et al., 2019; Peng et al., 2018; Pinto et al., 2017; Rajeswaran et al., 2016). Ma et al. (2019); Queeney and Benosman (2023) address reward and cost shifts through constraint-aware learning, and reward-poisoning analysis. Beyond these MDP components, Bouteiller et al. (2021); Peng et al. (2018); Tan et al. (2018) address latency shift, involving delays or irregularities in observation and action timing. Ding et al. (2023a); Tobin et al. (2017) address semantic shift, involving changes in appearance or task-relevant relationships among visual cues, objects, and goals. In practice, real-world scenarios may involve multiple shifts, such as changes in dynamics parameters, process noise, and observation noise (Mandlekar et al., 2017), or non-stationary shifts that evolve during deployment (Lecarpentier and Rachelson, 2019). Since robustness to one shift does not necessarily extend to others or their combinations, this motivates developing algorithms for co-occurring shifts and evaluating them under a unified framework. Robust RL algorithm taxonomy. Robust RL algorithms promote robustness through two main approaches: modifying the learning objective or algorithm (learner-centric) and modifying the training data distribution (data-centric). For robust online RL, learner-centric methods seek robust policies through optimization against an adversary (Zhang et al., 2021; 2020), worst-case robust value objectives (Li et al., 2024; Liang et al., 2022), smoothing the policy over perturbed observations (Sun et al., 2024), or causalinspired data augmentation to handle spurious correlated shifted scenarios (Ding et al., 2023a). Data-centric methods can collect trajectories by interacting with shifted environments through physical parameter randomization (Peng et al., 2018), model ensembles (Mankowitz et al., 2019; Rajeswaran et al., 2016; Tanabe et al., 2022), or by applying adversarial disturbances (Pinto et al., 2017). For robust offline RL, learnercentric methods address model uncertainty and limited data coverage through robust Bellman updates over an uncertainty set of transitions (Blanchet et al., 2023; Panaganti et al., 2022; 2023), and mitigate value overestimation by smoothness regularization over perturbed states (Yang et al., 2022). Data-centric methods instead augment or reweight fixed datasets by generating samples from adversarial environments (Bhardwaj et al., 2023; Rigter et al., 2022), mixing diffusion-generated synthetic data with real data (Lu et al., 2023), or downweighting corrupted samples (Yang et al., 2024). In particular, generative-model methods, as a data-centric subfamily, learn dynamics models from offline data and generates plausible future transitions or multi-step trajectories for robust policy learning (Berdica et al., 2024; Ding et al., 2024; Foffano et al., 2026). Furthermore, robust safe RL extends robust policy learning to jointly maintain task performance and satisfy cost constraints under uncertainty (Liu et al., 2022; Queeney and Benosman, 2023; Wang et al., 2022). In this work, RobustRLlib integrates 17 representative algorithms spanning robust online, offline, and safe RL, systematically organizes them by algorithm features, and evaluates them across diverse shift sources and modes under a unified protocol. It provides a reusable foundation for applying existing methods to new applications and developing new robust RL algorithms. Related RL benchmarks. Existing RL benchmarks can be broadly categorized into data-oriented benchmarks, which primarily provide fixed datasets for algorithm development (Fu et al., 2020; Gulcehre et al., 2020; Qin et al., 2022); task-oriented benchmarks, which offer standardized tasks and environments for developing and evaluating RL algorithms, ranging from continuous control to advanced robotics (Duan et al., 2016; Ji et al., 2023; Pumacay et al., 2024; Sferrazza et al., 2024; Yu et al., 2020a); and algorithm-oriented libraries and benchmarks, which provide reproducible implementations, training infrastructure, and pipelines for existing RL algorithms, serving as a foundation for downstream applications, fair comparisons, and further algorithm development (Hu et al., 2023; Huang et al., 2022; Liang et al., 2018; Raffin et al., 2021; Weng et al., 2022). In particular, for robust RL, closely related task-oriented benchmarks include RRLS, which focuses on continuous control (Zouitine et al., 2024), and Robust-Gymnasium, which covers a broader range of control and robotics tasks (Gu et al., 2025). Complementing these efforts, this work introduces RobustRLlib, the first unified algorithm-oriented library and benchmark for robust RL. It provides a reusable foundation for applying existing methods to new applications and developing new robust RL algorithms, while enabling easy comparisons with existing methods. In addition, RobustRLlib extends the considered deployment shifts to include latency and semantic shifts, and expands evaluation to advanced high-dimensional robotic tasks. 4

There exist other benchmarks that address related settings, such as off-dynamics transfer, real-world operating stressors, and adversarial robustness (Dulac-Arnold et al., 2020; Lyu et al., 2024; Mohan et al., 2026). Sim-to-real gap in robotics. The sim-to-real gap is a longstanding challenge in robotics, where mismatches between simulation and real-world hardware can substantially degrade the performance of policies trained in simulation. Prior work addresses this gap through various approaches, including dynamics randomization (Andrychowicz et al., 2020; Peng et al., 2018), actuator and latency modeling (Tan et al., 2018), and online adaptation to changing dynamics (Kumar et al., 2021). These considerations have been explored in whole-body humanoid control (Gu et al., 2024; He et al., 2024), safety-critical robotics (As et al., 2026), real-time VLA policies that overlap execution and inference (Black et al., 2026), and demonstration-driven policies for contact-rich manipulation (Sharma et al., 2026). However, many existing approaches are taskspecific and address particular sources of sim-to-real mismatch. In this work, we provide a unified robust RL algorithm library with 17 representative algorithms covering diverse tasks and sim-to-real gaps. These algorithms and their underlying design principles have the potential to support broader robotics applications while reducing manual iteration before hardware deployment.

3

RobustRLlib: An Algorithm-Centric Robust RL Benchmark

RobustRLlib is organized around algorithm attribution: it records how a robust RL method changes learning, which shift the method claims to address, and which base algorithm realizes the method. The disruptor toolbox then instantiates the claimed target shift independently of the base learner, while the evaluation platform measures both common robustness and method-specific specialization. This separation makes each result traceable to an algorithmic feature and an evaluation condition rather than treating robust RL as a single undifferentiated class. Markov decision processes (MDP). We consider a finite-horizon MDP M0 = (S, A, T, P0 , r0 ), where S and A are the state and action spaces, T is the horizon, and P0 and r0 denote the nominal transition kernel and reward function. At time t, the policy selects at ∼ πt (· | st ), receives reward r0,t (st , at ), and the environment transitions according to st+1 ∼ P0,t (· | st , at ). Classical robust MDPs replace the nominal transition model P0 with a shift set P and seek a policy with high worst-case return (Iyengar, 2005; Nilim and El Ghaoui, 2005): " T # X max inf J(π; P, r0 ), J(π; P, r0 ) = Eπ,P r0,t (st , at ) . (1) π

P ∈P

t=1

For safety-constrained tasks, we augment the MDP with a cost function c = {ct }Tt=1 and a budget d, PT requiring Eπ,P [ t=1 ct (st , at )] ≤ d in addition to maximizing task return. This cost interface is retained when shift is introduced, allowing robustness and safety to be evaluated jointly.

3.1

Robust RL algorithm library construction

The library has three tracks: offline RL learns from fixed data, online RL learns through continued environment interaction, and safe RL adds cost constraints, including robust-safe methods implemented here in the online setting. Within the offline and online tracks, robustness primarily enters through two routes: learnercentric methods change how experience is interpreted or optimized, while data-centric and environmentcentric methods change the distribution available for learning. Table 1 summarizes each method’s feature, target shift, and base algorithm. 3.1.1

Robust offline RL algorithms

Offline RL learns from a fixed dataset without environment interaction, making limited data coverage and optimistic value estimates for unsupported actions central challenges. Robust offline RL must also maintain performance under distribution shift. We therefore group its methods by how they address these challenges:

5

Problem

Algorithm feature

Target

Base

Shifted-env

Adversarial

Learned

shift

algorithm

rollout

network

model

Dynamic

FQI

RORL

Observation

SAC

ATLA

Observation

IQL

RSC-IQL

Semantics

IQL

RAMBO

Dynamic

SAC

✓

ROMB

Dynamic

IQL

✓

FWM

Dynamic

IQL

Method RFQI

Learner-centric Robust Offline Data-centric Generative model

Robust Online

Learner-centric

Environment-centric Robust Safe

Robust safe

✓ ✓ ✓ ✓ ✓ ✓

PLR-PVL

Dynamic

IQL

ATLA

Observation

PPO

✓

ATLA-SA

Observation

PPO

✓

RSC

Semantic

SAC

RARL

Dynamic

PPO / TRPO

✓

DR

Dynamic

SAC / PPO

✓

RAMU

Dynamic

PPO

SPiDR

Dynamic

PPO

✓

✓

✓

✓

Table 1: Structural comparison of the robust methods in the library. Online methods are grouped as learner-centric or environment-centric, offline methods as learner-centric or data-centric; generative model is a data-centric subfamily, and robust safe methods form a separate family. Shifted-env rollout marks rollouts collected from a deliberately perturbed physical environment. Adversarial network marks a learned adversary optimized against the policy. Learned model marks a trained next-state predictor used by the method. Target shift records the intended shift source. A checkmark denotes a present feature; a blank denotes its absence. learner-centric methods modify the objective or update, whereas data-centric methods reshape the training distribution, including through data augmentation or generative models. As standard RL baselines, IQL (Kostrikov et al., 2021) and TD3+BC (Fujimoto and Gu, 2021) learn directly from logged transitions; MOPO (Yu et al., 2020b) adds model rollouts, and SynthER (Lu et al., 2023) adds diffusion-generated datasets. Learner-centric methods. These methods make policy learning less sensitive to uncertain transitions or perturbed observations by changing the value target or policy update. • RFQI: worst-case Bellman update. RFQI (Panaganti et al., 2022) replaces the nominal fitted-Q target with a worst-case backup over transition ambiguity. • RORL: local policy and value smoothing. RORL (Yang et al., 2022) combines pessimistic bootstrapping with smoothing around states to stabilize predictions under observation perturbations. • ATLA: observation adversary. The offline ATLA adaptation (Zhang et al., 2021) trains an observation adversary alongside standard base algorithm, exposing policy updates to adverse inputs. Data-centric methods. These methods expand the experience available to the base learner through counterfactual transitions or model-generated rollouts. • RSC: causal counterfactual rewrite. The offline RSC (Ding et al., 2023a) adaptation rewrites samples using an inferred causal mask to weaken spurious semantic correlations. • RAMBO: adversarial model rollouts. RAMBO (Rigter et al., 2022) branches short synthetic rollouts from dataset states and adjusts its dynamics model adversarially within a data-fit constraint. Generative model methods. These data-centric methods learn conditional models of transition dynamics and use generated next states or multi-step futures in policy learning, which potentially improve robustness under dynamics shifts. SynthER is excluded from this family because it models the distribution of logged transitions to enlarge the dataset rather than modeling dynamics for policy learning.

6

• ROMB: maximin model ensemble. ROMB (Chen et al., 2025) optimizes against adverse transitions represented by an ensemble of learned models. • FWM: flow-generated multi-step targets. FWM uses flow-matching to replace diffusion world model (Ding et al., 2024) and samples future trajectories to construct multi-step critic targets. • PLR-PVL: worst-case world model selection. PLR-PVL (Berdica et al., 2024) searches a flow-model ensemble for adverse plausible futures during policy learning. 3.1.2

Robust online RL algorithms

Online robust RL methods preserve continued interaction with the environment but differ in where robustness enters training. We distinguish two primary routes. Learner-centric methods modify the policy input and learning objective while retaining the nominal physical transition law. Environment-centric methods instead interact with deliberately perturbed environments to collect training rollouts. Whether a method updates onpolicy or off-policy remains an important implementation choice, so we record it through the base algorithm. PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015) and SAC (Haarnoja et al., 2018) serve as the standard base algorithms in our experiments. Learner-centric methods. These methods train the policy to make stable decisions using adversarial observations, local regularization, or causal counterfactual experience. • ATLA: observation adversary. ATLA (Zhang et al., 2021) alternates updates between the policy and a learned adversary that perturbs its observations. • ATLA-SA: state-adversarial regularization. ATLA-SA (Zhang et al., 2021) adds recurrent history and SA-MDP regularization to stabilize decisions under perturbed observations. • RSC: causal counterfactual replay. RSC (Ding et al., 2023a) infers causal dependencies and rewrites experience counterfactually to reduce reliance on spurious semantic features. Environment-centric methods. These methods build robustness through training trajectories generated under varied physical environments or external disturbances. • RARL: external force adversary. RARL (Pinto et al., 2017) perturbs the environment with bounded forces selected by an adaptive adversary. The protagonist and adversary train in a two-player game, concentrating experience on the policy’s current vulnerabilities. • DR: domain randomization. DR (Peng et al., 2018) samples physical parameters across training episodes, exposing the policy to a prescribed envelope of transition dynamics. 3.1.3

Robust safe RL algorithms

Robust safe RL methods seek to maximize worst-case expected return while ensuring cost-constraint satisfaction across a prescribed dynamics uncertainty set. • RAMU: risk-sensitive reward and cost targets. RAMU (Queeney and Benosman, 2023) applies risk measures over transition-model uncertainty to both reward and cost Bellman targets. • SPiDR: robust safety via penalized costs. SPiDR (As et al., 2026) measures disagreement among nextstate predictions from domain-randomized models and penalizes uncertain safety outcomes.

3.2

Expandable disruptor toolbox

Robust RL preserves the underlying task objective of standard RL by maximizing cumulative return but extends the single nominal MDP into a family of robust MDP with different types of shifts. These shifts intervene in different components of the agent–environment interaction, such as observations, actions, transition dynamics, rewards or costs, latency, and task semantics. The resulting robust RL problem therefore seeks a policy that performs reliably across these shifted interaction conditions. 3.2.1

Shift sources

We classify a shift by where it enters the agent–environment interaction. The first four sources act directly on components of the MDP interface (Figure 2a); latency (Figure 2b) and semantics (Figure 2c) then capture 7

Dynamic shift

Observation shift

Shorter leg

Gaussian Noise

Action shift

Reward shift

Perturbed action

(a) MDP-element shifts.

(b) Latency shift.

(c) Semantic shift.

Figure 2: Shift sources covered in RobustRLlib. (a) Shifts of the MDP elements: dynamic, observation, action, and reward. (b) Latency between the policy’s and the executed time stamp. (c) Semantic shift in appearance and goal displacement. deployment factors that are not represented cleanly by a single nominal MDP component. Given xt = (st , ãt ) and yt = (st , ãt , st+1 ), we expose the six shift sources as below: • Dynamic shift. Changes the environment response under the same state and executed action, st+1 ∼ Pθp (· | xt ), where θp specifies physical-parameter changes, external disturbances, and latent process noise. Its operators cover physical changes including internal dynamics such as morphology, friction, gravity and external disturbance such as wind, external pushes, and latent noise. These variations represent physical mismatch and motivate dynamics randomization for sim-to-real control (Peng et al., 2018). • Observation shift. Changes the information available to the policy, õt ∼ Oθo (· | st ) and at ∼ π(· | õt ), where θo specifies the perturbation applied to the policy observation. It represents sensor noise, bias, missing or imperfect feedback, and bounded observation attacks (Dulac-Arnold et al., 2020). • Action shift. Changes the command that reaches the plant, ãt ∼ Aθa (· | st , at ), where θa specifies how the intended action is altered before execution. It captures execution noise, calibration error, saturation, opposition, and other actuator-side distortions (Tessler et al., 2019). • Reward/cost shift. Changes the feedback used for task learning or safety evaluation, r̃t ∼ Rθr (· | yt ) and c̃t ∼ Cθc (· | yt ), where θrc = (θr , θc ) specifies the reward and cost perturbations. It represents noisy or biased supervision while keeping the underlying transition explicit. • Latency shift. Captures deployment-time mismatch in the timing of sensing, action execution, control, and VLA inference delay (Bouteiller et al., 2021; Dulac-Arnold et al., 2020; Tan et al., 2018). We treat latency as a composable deployment-shift source within this unified robust RL evaluation framework. Specifically, we expose observation delay, action delay, and control-period variation through õt = I(Ht , t− (κ ) δo ), ãt = at−δa , st+1 ∼ P0 t (· | st , ãt ), capturing delayed sensing, delayed execution, and variable control rates, respectively. Implementation details are provided in Appendix A.2.1. • Semantic shift. Changes task-relevant scenes or structures rather than merely adding task-agnostic noise. Such shifts can be represented as ot ∼ Oθz (· | st ), st+1 ∼ Pθz (· | xt ), (rt , ct ) ∼ (Rθz , Cθz )(· | yt ), where θz specifies appearance or task-semantic parameter changes. Camera, lighting, color, blur, and distractor shifts act primarily through Oθz , whereas relocating a goal or object may also change the transition dynamics and task feedback. Such semantic shifts are increasingly important for visually grounded policies, where deployment may involve diverse task-specific scenes, including different objects, workspaces, viewpoints, layouts, or instructions (Brohan et al., 2023; Fei et al., 2025; Pumacay et al., 2024), potentially inducing semantic spurious correlations as an important deployment shift (Ding et al., 2023a; Tobin et al., 2017). 3.2.2

Shift modes

A mode says how an intervention is built, and is one field of the specification tuple (source, factor, mode, magnitude, resampling rule, optional schedule, seed): the source says where the intervention enters the

8

agent–environment interaction (Section 3.2.1), the factor says what it perturbs (e.g., gravity or friction), the magnitude says how far it departs from nominal, the resampling rule and optional schedule say how it evolves within an episode, and the seed makes its random draws reproducible. Each instantiated tuple defines one perturbation configuration. We retain the stochastic interface of Robust-Gymnasium (Gu et al., 2025), and extend it with several new modes as below: • Stochastic. Samples signal errors or parameters from specified distributions. • Adversarial. Computes bounded perturbations from the current states to counter the policy. • Parametric. Applies a specified assignment, scale, offset, or translation at a severity. • Non-stationary. Varies a mode’s severity over time or triggers intermittent disturbances. • Composition. Stacks compatible shifts into reproducible multi-shift scenarios. Together, these controls support both individual shift and compound shift scenarios. The appendix A.2 provides the operators, magnitudes, resampling rules, and composition procedure. 3.2.3

Expandable to advanced robotics tasks

Because a shift is fully described by its specification tuple, a new backend needs only one adapter to the simulator’s own primitives. We validate this on two deployment-oriented platforms. For Isaac Lab (Mittal et al., 2025), the adapter realizes transition, action, and latency shifts on the vectorized GPU environment and exposes a joint-limit cost channel for every method’s evaluation, so unconstrained and constrained learners are compared on one safety quantity. For a VLA policy (Intelligence et al., 2025) fine-tuned on the bimanual demonstrations (Zhang et al., 2026), the adapter injects appearance, proprioception, action, latency, and physical shifts into a closed-loop simulator.

4

Experimental Evaluation Online

100

Online

Offline

Navigation: 2 Tasks × 3 Shift Factors × 4 Configurations Online

100

RFQI RORL ATLA RSC-IQL RAMBO ROMB FWM PLR-PVL

Standard RL

ATLA ATLA-SA RSC RARL-P RARL-T DR

Standard RL

RFQI RORL ATLA RSC-IQL RAMBO ROMB FWM PLR-PVL

20

Standard RL

40

ATLA ATLA-SA RSC RARL-P RARL-T DR

60

0

Vehicle control: 2 Tasks × 3 Shift Factors × 6 Configurations

Offline

Online

Offline

Algorithm

Robust RL:

Learner-centric

RFQI RORL ATLA RSC-IQL RAMBO ROMB FWM PLR-PVL

Standard RL

ATLA ATLA-SA RSC RARL-P RARL-T DR

0

Standard RL

20

RFQI RORL ATLA RSC-IQL RAMBO ROMB FWM PLR-PVL

40

Standard RL

60

ATLA ATLA-SA RSC RARL-P RARL-T DR

80

Standard RL

Normalized score

Manipulation: 2 Tasks × 2 Shift Factors × 4 Configurations

Offline

80

Standard RL

Normalized score

Locomotion: 3 Tasks × 3 Shift Factors × 8 Configurations

Algorithm

Environment-centric

Learner-centric

Data-centric

Generative model

Figure 3: Performance evaluation of 20 robust RL algorithms in the RobustRLlib library under unified perturbation configurations. Every colored bar is one of the fourteen robust methods, keyed by algorithm feature and online/offline setting. Scores average the perturbation configurations within each shift factor, factors within each task, and then tasks with equal weight. The aggregation covers 20 algorithms across 220 distinct perturbation configurations.

9

The evaluation proceeds in three blocks over our library. Section 4.1 measures how much robust methods gain across the whole library and how that gain behaves as the shifts intensify. Section 4.2 isolates each shift source and composes them, to ask which shift source a method actually defends. Section 4.3 carries the same interface to deployment-oriented humanoid and vision–language–action (VLA) systems. Experimental setup. The library implements 23 algorithms: 6 standard RL references, 15 robust RL and 2 robust safe RL algorithms. The library-wide grid runs the 20 algorithms across Locomotion, Manipulation, Navigation and Vehicle Control; one robust RL and two robust safe algorithms enter on the advanced robotic tasks. Every algorithm starts from the same nominal task specification, uses its native training procedure and prescribed budget, and is then frozen and evaluated on the same shift grid. The results use five training seeds and 20 paired evaluation episodes per condition. Exact environments, perturbation configurations, and protocols are in Appendix B. Performance metric. For a comparable return across different tasks, we report the unclipped normalized score S(R̄) = 100(R̄ − Rmin )/(Rmax − Rmin ) as our evaluation metric, where Rmin and Rmax are fixed task-level reference returns shared by every method. Appendix B.1 details the metrics.

4.1

Evaluation of the algorithm library

Robustness Decay +10.3

60

Gain

Normalized score

70

Standard RL Robust RL

50 40 30

+5.1 Q1

Q2 Learner-centric

Shift level

Q3

Environment-centric

Q4

12 10 8 6 4 2 0 2 4

Robust RL outperforms Standard RL

Learner-centric

+5.5

+9.8

+9.6

+9.5

+7.6

+6.6 +3.8 +2.9

+2.5 -0.1

Online Data-centric

Offline Generative model

Figure 4: Robust RL gains across shift levels and algorithm features. Left: Aggregated performance of standard and robust RL algorithms across diverse environment shift levels Q1–Q4 (Refer to Appendix B.2 for more details). Right: performance gains over respective base algorithms (dark) and the average of standard RL (light) across diverse algorithm designs. See Appendix G.1 for details. Robust RL methods differ in their training assumptions and access to data and environments, making results from their original settings difficult to compare directly. We therefore evaluate their final policies on representative tasks to identify their capability boundaries. Our first comparison covers 20 online and offline algorithms, including 6 standard and 14 robust RL, over 220 perturbation configurations across four task families. Figure 3 shows each robust algorithm separately, while Figure 4 aggregates the same evaluation by shift levels and algorithm features to show both the robust–standard gap and feature-grouped algorithms’ gain over its base algorithms. Robust RL helps at every shift level, but both degrade as shifts intensify. Across all four shift levels, robust methods achieve a higher pooled normalized score than standard RL. Their advantage remains positive but narrows from +10.3 points in Q1 to +5.1 in Q4, while both groups’ scores decline as shifts intensify.

10

Methods that explicitly model dynamics shifts exhibit greater robustness to such shifts. Two families explicitly target dynamics shifts: environment-centric online methods such as DR-SAC perturb the environment simulator during training, while generative offline methods such as FWM learn a generative dynamics model and train on its trajectories. These families achieve the strongest performance under the controlled dynamic shifts within their respective online and offline settings. Overall, robustness gains reflect both the robust methods and the strength of the base learner.

4.2

Evaluation under diverse deployment shifts

Normalized score

Normalized score

We evaluate method-specific robustness through individual shift source sweeps across 44 perturbation configurations and reference-derived compound shifts. The former cover multiple perturbation modes. For the latter, Hopper adapts the coupled physical, actuation, force, and delay factors of OmniH2O (He et al., 2024), while Pusher adapts the dynamics, observation, and control-rate factors of Peng et al. (2018), with goal displacement labeled separately. Appendix B.2 details the individual and the compound shift scenarios.

100

Observation

Action

87.0

83.3

Dynamics

Reward

Execution latency

101.5

Credit latency

Semantic binding

82.9

68.6

65.8

DR

ATLA-SA

RSC

Execution latency

Credit latency

60.0 50 0

125 100 75 50 25 0

ATLA-SA

ATLA-SA

PPO

Observation

SAC

RARL-P

DR

RARL-T

ATLA-SA

ATLA

Action

Dynamics

38.1

45.8

ATLA-SA

DR

RSC

nominal (row mean)

Reward

108.6

68.0

RFQI

ATLA

PLR-PVL

RSC-IQL

ROMB

FWM

104.2

52.1

PLR-PVL

PLR-PVL

ATLA

RAMBO

RFQI

RORL

RORL ATLA

nominal (row mean)

Figure 5: Evaluation under diverse individual shift scenarios. The figure aggregates 14 robust and 2 standard RL methods across 44 distinct perturbation configurations. Semantic binding is evaluated only online, on the goal-conditioned causal door task. See Appendix G.4 for full results.

Claimed target shifts do not fully predict empirical strengths. Figure 5 shows DR-SAC leads dynamics and execution latency, while ATLA-SA-PPO leads observation and action; offline leaders vary across shift sources. A method’s intended target shift therefore does not uniquely determine where it performs best. The measured ranking is also conditional on the task, shift mode and specific perturbation used to instantiate each shift. Online robust methods generally achieve larger gains over their base algorithms than offline robust methods. Figure 6 shows that online methods can produce large gains on most shift sources, whereas most offline differences remain close to their base algorithms. The smaller offline increments may partly reflect limited coverage of the tested perturbation modes in fixed datasets. Because offline RL relies on fixed data, methods often use conservative updates to limit value overestimation which potentially constrains robustness gains. Dynamic shifts cause the largest performance drops, while the next most impactful shift varies by task. We test our strongest online (ATLA-SA-PPO) and offline robust (RORL) algorithms under compound shifts. Figure 7 shows that dynamic dominate both scenarios. Dynamic shift is the most damaging shift source in the two tasks, while the next most damaging differs by task. Locomotion loses

11

Offline

Figure 6: Gains over base algorithms. Each cell is the score difference between a robust method and its base algorithm, averaged over 44 perturbation configurations of five shift sources.

.

mp

n.

mp

nominal (RORL)

40

Co

0

Dy

0

m.

20

Se

20

s.

40

t.

40

Ob

60

Pusher · online · ATLA-SA-PPO

La

80

60

.

80

Co

20

100

t. n.

+3 +2 +1 +11 +6 +7 -2 -2 -0 +2 -2 +2 +0 -3 +1 -0 -0 -1 +10 +4 +1 +3 +1 +6 +6

120

100

Dy

IQL IQL IQL IQL IQL

0

Hopper · offline · RORL

La

ATLA RSC-IQL ROMB FWM PLR-PVL

+20

120

t.

Online

+40

Ac

PPO +10 +23 +8 +2 +8 PPO +64 +48 +23 +14 +26 PPO +16 +36 +10 -67 -7 SAC +24 +47 +20 +28 +23 SAC +1 +24 +13 -38 +2

Normalized score

obs. act. dyn. rew. lat. ATLA ATLA-SA RARL DR RSC

nominal (ATLA-SA-PPO)

Figure 7: Strongest algorithms performance under compound shifts from multiple sources. Each panel shows the performance of the strongest offline (RORL) and online (ATLASA-PPO) robust algorithms under individual and compound shifts. The horizontal line marks each algorithm’s nominal score. See Appendix G.4 for full results.

about two-thirds of its score to latency, because delayed action on a balancing gait is a fall rather than a slower trajectory, while semantic shift on manipulation costs a quarter, because it moves the goal of the policy. Therefore, a sim-to-real design must identify which shift is core to its task. Composition is worse than any of its parts, making both algorithms drop below half of nominal. This indicates the poor capability of algorithms in complex, multi-shift environments.

4.3

Extension to advanced robotic tasks

We extend the benchmark from standard control to advanced robotics tasks, attaching the same disruptor interface through task-specific adapters, covering a high-dimensional humanoid, a contact-rich manipulator and a VLA control task. Robust safe methods can ensure both performance and constraint satisfaction under shifts. We compare PPO, DR-PPO (Schwarke et al., 2025) and ATLA-PPO against the robust safe methods RAMUPPO and SPiDR-PPO on Unitree G1 locomotion and Franka drawer manipulation in Isaac Lab (Mittal et al., 2025). Figure 8 shows the two highest-scoring policies on G1 survive about a quarter under action delay; while on the drawer DR-PPO matches SPiDR-PPO’s 99.6% performance with 0% of its episodes inside the joint-limit cost budget against SPiDR-PPO’s 84%. The results indicate that only robust safe methods especially SPiDR-PPO, keep both metrics high on both tasks, so a higher performance is not the evidence that a method is ready for sim-to-real deployment. Performance sensitivity differs sharply across shift sources, with semantic shifts more impactful in VLA tasks. We fine-tune π0.5 on the insert-mouse-battery task (Figure 11) and drive it under eight shift factors. At the largest measured level the policy still remains 0.95 of its nominal performance under gravity but only 0.33 under image blur. Blur at its largest level moves only 13% of the image but drops the policy to a third of nominal, while image brightness moves the same frames at seven times and costs almost nothing. Semantic shift is a core shift source which affects more than physical shifts since a clear scene is the base of the action chunk.

12

Humanoid control tasks 100

100

80

80

60 40 20 0

0

25

50

75

PPO

DR-PPO

Success rate (%)

60 40 20 0

100 ATLA-PPO

Retention (1.0 = nominal)

Manipulation · Drawer

Performance

Performance

Locomotion · G1

VLA robotic tasks

0

25

50

75

Success rate (%)

RAMU-PPO

100

Performance degradation under shifts

1.1 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3

Gravity 0.95 Dry friction 0.91 Brightness 0.88 Proprio. noise 0.77 Execution delay 0.70 Command noise 0.65 Joint damping 0.53 Blur 0.33

Q1 dynamics action

SPiDR-PPO

Q2

Q3

Shift levels of 8 factors latency observation

Q4

semantic real robot's measured operating point

Figure 8: Performance evaluation on Isaac Lab and VLA tasks. Left: each point is one algorithm, plotting task performance against success rate on two tasks (G1: survival under action delay; Drawer: episodes within the joint-limit cost budget). See Appendices B.2 and G.3 for the detailed setting and results. Right: performance retention of a VLA policy as eight shift factors moves from Q1 to Q4. (Q1-Q4 definitions see Appendix B.2). The diamond marks each factor’s operating point measured on the real platform.

5

Conclusion

In this work, we introduce RobustRLlib, an algorithm-centric library and evaluation platform for robust RL under deployment shifts. It integrates 17 robust RL algorithms with 6 standard RL baselines, organized by algorithm feature, claimed target shift, and base algorithm. A shared interface evaluates MDP-element shifts alongside control-loop latency and task semantics through individual and compound shift scenarios and extends to advanced robotics tasks. Under a unified evaluation over 273 perturbation configurations, robust methods as a group outperform standard RL references. Individual and compound shift evaluations further reveal each algorithmic feature’s strengths and capability boundaries. Therefore, RobustRLlib provides a platform to support more comprehensive evaluation and guide the design of future robust RL methods.

References Andrychowicz, O. M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., Pachocki, J., Petron, A., Plappert, M., Powell, G., Ray, A., et al. (2020). Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20. Arthur, W. B. (1991). Designing economic agents that act like human agents: A behavioral approach to bounded rationality. The American economic review, 81(2):353–359. As, Y., Qu, C. R., Unger, B., Kang, D., van der Hart, M., Shi, L., Coros, S., Wierman, A., and Krause, A. (2026). Spidr: A simple approach for zero-shot safety in sim-to-real transfer. Advances in Neural Information Processing Systems, 38:90882–90924. Berdica, U., Li, K., Beukman, M., Goldie, A. D., Maiolino, P., and Foerster, J. N. (2024). Robust offline learning via adversarial world models. In NeurIPS 2024 Workshop on Open-World Agents. Bertsimas, D., Sim, M., and Zhang, M. (2019). Adaptive distributionally robust optimization. Management Science, 65(2):604–618. Bhardwaj, M., Xie, T., Boots, B., Jiang, N., and Cheng, C.-A. (2023). Adversarial model for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1245–1269. Black, K., Galliker, M., and Levine, S. (2026). Real-time execution of action chunking flow policies. Advances in Neural Information Processing Systems, 38:33383–33407.

13

Blanchet, J., Lu, M., Zhang, T., and Zhong, H. (2023). Double pessimism is provably efficient for distributionally robust offline reinforcement learning: Generic algorithm and robust partial coverage. Advances in Neural Information Processing Systems, 36:66845–66859. Bouteiller, Y., Ramstedt, S., Beltrame, G., Pal, C. J., and Binas, J. (2021). Reinforcement learning with random delays. In ICLR. Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., Ding, T., Driess, D., Dubey, A., Finn, C., et al. (2023). Rt-2: Vision-language-action models transfer web knowledge to robotic control. arXiv preprint arXiv:2307.15818. Chen, J., Xu, L., Venugopal, A., and Schneider, J. (2025). Policy-driven world model adaptation for robust offline model-based reinforcement learning. arXiv preprint arXiv:2505.13709. Ding, W., Shi, L., Chi, Y., and Zhao, D. (2023a). Seeing is not believing: Robust reinforcement learning against spurious correlation. Advances in Neural Information Processing Systems, 36:66328–66363. Ding, W., Xu, C., Arief, M., Lin, H., Li, B., and Zhao, D. (2023b). A survey on safety-critical driving scenario generation—a methodological perspective. IEEE Transactions on Intelligent Transportation Systems, 24(7):6971–6988. Ding, Z., Zhang, A., Tian, Y., and Zheng, Q. (2024). Diffusion world model: Future modeling beyond step-by-step rollout for offline reinforcement learning. arXiv preprint arXiv:2402.03570. Duan, Y., Chen, X., Houthooft, R., Schulman, J., and Abbeel, P. (2016). Benchmarking deep reinforcement learning for continuous control. In International conference on machine learning, pages 1329–1338. PMLR. Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. (2020). An empirical investigation of the challenges of real-world reinforcement learning. arXiv preprint arXiv:2003.11881. Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., et al. (2025). Libero-plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626. Foffano, D., Russo, A., and Proutiere, A. (2026). Adversarial diffusion for robust reinforcement learning. Advances in Neural Information Processing Systems, 38:68627–68658. Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. (2020). D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219. Fujimoto, S. and Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132–20145. Goyal, V. and Grand-Clement, J. (2023). Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research, 48(1):203–226. Gu, S., Shi, L., Wen, M., Jin, M., Mazumdar, E., Chi, Y., Wierman, A., and Spanos, C. (2025). Robust gymnasium: A unified modular benchmark for robust reinforcement learning. In International Conference on Learning Representations, volume 2025, pages 102014–102041. Gu, X., Wang, Y.-J., and Chen, J. (2024). Humanoid-gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.05695. Gulcehre, C., Wang, Z., Novikov, A., Paine, T., Gómez, S., Zolna, K., Agarwal, R., Merel, J. S., Mankowitz, D. J., Paduraru, C., et al. (2020). Rl unplugged: A suite of benchmarks for offline reinforcement learning. Advances in neural information processing systems, 33:7248–7259. Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr. 14

Han, S., Su, S., He, S., Han, S., Yang, H., and Miao, F. (2022). What is the solution for state adversarial multi-agent reinforcement learning? arXiv preprint arXiv:2212.02705. He, T., Luo, Z., He, X., Xiao, W., Zhang, C., Zhang, W., Kitani, K., Liu, C., and Shi, G. (2024). Omnih2o: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. arXiv preprint arXiv:2406.08858. Hu, S., Zhong, Y., Gao, M., Wang, W., Dong, H., Liang, X., Li, Z., Chang, X., and Yang, Y. (2023). Marllib: A scalable and efficient multi-agent reinforcement learning library. Journal of Machine Learning Research, 24(315):1–23. Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraÚjo, J. G. (2022). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1–18. Huang, S., Papernot, N., Goodfellow, I., Duan, Y., and Abbeel, P. (2017). Adversarial attacks on neural network policies. arXiv preprint arXiv:1702.02284. Intelligence, P., Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A. Z., Shi, L. X., Smith, L., Springenberg, J. T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., and Zhilinsky, U. (2025). π0.5 : a vision-language-action model with open-world generalization. Iyengar, G. N. (2005). Robust dynamic programming. Mathematics of Operations Research, 30(2):257–280. Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. (2023). Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36:18964–18993. Klopp, O., Lounici, K., and Tsybakov, A. B. (2017). Robust matrix completion. Probability Theory and Related Fields, 169(1-2):523–564. Kostrikov, I., Nair, A., and Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169. Kozak, J. P., Zhang, R., Porter, M., Song, Q., Liu, J., Wang, B., Wang, R., Saito, W., and Zhang, Y. (2023). Stability, reliability, and robustness of gan power devices: A review. IEEE Transactions on Power Electronics, 38(7):8442–8471. Kumar, A., Fu, Z., Pathak, D., and Malik, J. (2021). Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034. Lecarpentier, E. and Rachelson, E. (2019). Non-stationary markov decision processes, a worst-case approach using model-based reinforcement learning. Advances in neural information processing systems, 32. Li, H., Zhang, Z., Luo, W., Han, C., Hu, Y., Guo, T., and Liao, S. (2024). Towards optimal adversarial robust q-learning with bellman infinity-error. arXiv preprint arXiv:2402.02165. Li, Z., Cheng, X., Peng, X. B., Abbeel, P., Levine, S., Berseth, G., and Sreenath, K. (2021). Reinforcement learning for robust parameterized locomotion control of bipedal robots. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2811–2817. IEEE. Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., and Stoica, I. (2018). Rllib: Abstractions for distributed reinforcement learning. In International conference on machine learning, pages 3053–3062. PMLR. Liang, Y., Sun, Y., Zheng, R., and Huang, F. (2022). Efficient adversarial training without attacking: Worstcase-aware robust reinforcement learning. Advances in neural information processing systems, 35:22547– 22561. 15

Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. (2022). Flow matching for generative modeling. arXiv preprint arXiv:2210.02747. Liu, Q., Li, Y., and Odem-Davis, K. (2015). On robustness of noninferiority clinical trial designs against bias, variability, and nonconstancy. Journal of biopharmaceutical statistics, 25(1):206–225. Liu, Z., Guo, Z., Cen, Z., Zhang, H., Tan, J., Li, B., and Zhao, D. (2022). On the robustness of safe reinforcement learning under observational perturbations. arXiv preprint arXiv:2205.14691. Lu, C., Ball, P., Teh, Y. W., and Parker-Holder, J. (2023). Synthetic experience replay. Advances in Neural Information Processing Systems, 36:46323–46344. Lu, C., Ball, P. J., Parker-Holder, J., Osborne, M. A., and Roberts, S. J. (2021). Revisiting design choices in offline model-based reinforcement learning. arXiv preprint arXiv:2110.04135. Lyu, J., Xu, K., Xu, J., Yan, M., Yang, J., Zhang, Z., Bai, C., Lu, Z., and Li, X. (2024). Odrl: A benchmark for off-dynamics reinforcement learning. Advances in Neural Information Processing Systems, 37:59859–59911. Ma, Y., Zhang, X., Sun, W., and Zhu, J. (2019). Policy poisoning in batch reinforcement learning and control. Advances in Neural Information Processing Systems, 32. Mahmood, A. R., Korenkevych, D., Vasan, G., Ma, W., and Bergstra, J. (2018). Benchmarking reinforcement learning algorithms on real-world robots. In Conference on robot learning, pages 561–591. PMLR. Mandlekar, A., Zhu, Y., Garg, A., Fei-Fei, L., and Savarese, S. (2017). Adversarially robust policy learning: Active construction of physically-plausible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3932–3939. IEEE. Mankowitz, D. J., Levine, N., Jeong, R., Shi, Y., Kay, J., Abdolmaleki, A., Springenberg, J. T., Mann, T., Hester, T., and Riedmiller, M. (2019). Robust reinforcement learning for continuous control with model misspecification. arXiv preprint arXiv:1906.07516. Mittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., Du, P., Serrano-Muñoz, A., Yao, X., Zurbrügg, R., Rudin, N., Wawrzyniak, L., Rakhsha, M., Denzler, A., Heiden, E., Borovicka, A., Ahmed, O., Akinola, I., Anwar, A., Carlson, M. T., Feng, J. Y., Garg, A., Gasoto, R., Gulich, L., Guo, Y., Gussert, M., Hansen, A., Kulkarni, M., Li, C., Liu, W., Makoviychuk, V., Malczyk, G., Mazhar, H., Moghani, M., Murali, A., Noseworthy, M., Poddubny, A., Ratliff, N., Rehberg, W., Schwarke, C., Singh, R., Smith, J. L., Tang, B., Thaker, R., Trepte, M., Wyk, K. V., Yu, F., Millane, A., Ramasamy, V., Steiner, R., Subramanian, S., Volk, C., Chen, C., Jawale, N., Kuruttukulam, A. V., Lin, M. A., Mandlekar, A., Patzwaldt, K., Welsh, J., Zhao, H., Anes, F., Lafleche, J.-F., Moënne-Loccoz, N., Park, S., Stepinski, R., Gelder, D. V., Amevor, C., Carius, J., Chang, J., Chen, A. H., de Heras Ciechomski, P., Daviet, G., Mohajerani, M., von Muralt, J., Reutskyy, V., Sauter, M., Schirm, S., Shi, E. L., Terdiman, P., Vilella, K., Widmer, T., Yeoman, G., Chen, T., Grizan, S., Li, C., Li, L., Smith, C., Wiltz, R., Alexis, K., Chang, Y., Chu, D., Fan, L. J., Farshidian, F., Handa, A., Huang, S., Hutter, M., Narang, Y., Pouya, S., Sheng, S., Zhu, Y., Macklin, M., Moravanszky, A., Reist, P., Guo, Y., Hoeller, D., and State, G. (2025). Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831. Mohan, A., Kriegl, D., and Schön, T. (2026). Road-rl: A unified library and benchmark for robust adversarial reinforcement learning. arXiv preprint arXiv:2606.29867. Nilim, A. and El Ghaoui, L. (2005). Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798. Oikarinen, T., Zhang, W., Megretski, A., Daniel, L., and Weng, T.-W. (2021). Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34:26156–26167. Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in neural information processing systems, 35:32211–32224. 16

Panaganti, K., Xu, Z., Kalathil, D., and Ghavamzadeh, M. (2023). Bridging distributionally robust learning and offline rl: An approach to mitigate distribution shift and partial data coverage. arXiv preprint arXiv:2310.18434. Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. (2018). Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803–3810. IEEE. Pinto, L., Davidson, J., Sukthankar, R., and Gupta, A. (2017). Robust adversarial reinforcement learning. In International conference on machine learning, pages 2817–2826. PMLR. Pumacay, W., Singh, I., Duan, J., Krishna, R., Thomason, J., and Fox, D. (2024). The colosseum: A benchmark for evaluating generalization for robotic manipulation. arXiv preprint arXiv:2402.08191. Qin, R.-J., Zhang, X., Gao, S., Chen, X.-H., Li, Z., Zhang, W., and Yu, Y. (2022). Neorl: A near realworld benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:24753–24765. Queeney, J. and Benosman, M. (2023). Risk-averse model uncertainty for distributionally robust safe reinforcement learning. Advances in Neural Information Processing Systems, 36:1659–1680. Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021). Stable-baselines3: Reliable reinforcement learning implementations. Journal of machine learning research, 22(268):1–8. Raffin, A. and Stulp, F. (2020). Generalized state-dependent exploration for deep reinforcement learning in robotics. Arxiv. Rajeswaran, A., Ghotra, S., Ravindran, B., and Levine, S. (2016). Epopt: Learning robust neural network policies using model ensembles. arXiv preprint arXiv:1610.01283. Rigter, M., Lacerda, B., and Hawes, N. (2022). Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082–16097. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. (2015). Trust region policy optimization. In International conference on machine learning, pages 1889–1897. Pmlr. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Schwarke, C., Mittal, M., Rudin, N., Hoeller, D., and Hutter, M. (2025). Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509.10771. Sferrazza, C., Huang, D.-M., Lin, X., Lee, Y., and Abbeel, P. (2024). Humanoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506. Sharma, S., Sahoo, S., Huang, H., Li, F.-F., Wu, J., Sadigh, D., and Bohg, J. (2026). One demonstration, many objects: Generalizing manipulation via local contact geometry. Shi, L. and Chi, Y. (2024). Distributionally robust model-based offline reinforcement learning with nearoptimal sample complexity. Journal of Machine Learning Research, 25(200):1–91. Shi, L., Li, G., Wei, Y., Chen, Y., Geist, M., and Chi, Y. (2026). The curious price of distributional robustness in reinforcement learning with a generative model. Operations Research, 74(5):2438–2459. Sun, C.-E., Gao, S., and Weng, T.-W. (2024). Breaking the barrier: Enhanced utility and robustness in smoothed drl agents. arXiv preprint arXiv:2406.18062. Sun, K., Liu, Y., Zhao, Y., Yao, H., Jui, S., and Kong, L. (2021). Exploring the training robustness of distributional reinforcement learning against noisy state observations. arXiv preprint arXiv:2109.08776.

17

Tan, J., Zhang, T., Coumans, E., Iscen, A., Bai, Y., Hafner, D., Bohez, S., and Vanhoucke, V. (2018). Sim-to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332. Tan, K. L., Esfandiari, Y., Lee, X. Y., Sarkar, S., et al. (2020). Robustifying reinforcement learning agents via action space adversarial training. In 2020 American control conference (ACC), pages 3959–3964. IEEE. Tanabe, T., Sato, R., Fukuchi, K., Sakuma, J., and Akimoto, Y. (2022). Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification. Advances in neural information processing systems, 35:6967–6981. Tessler, C., Efroni, Y., and Mannor, S. (2019). Action robust reinforcement learning and applications in continuous control. In International Conference on Machine Learning, pages 6215–6224. PMLR. Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. (2017). Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23–30. IEEE. Tversky, A. and Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases: Biases in judgments reveal some heuristics of thinking under uncertainty. science, 185(4157):1124–1131. Wang, H., Shi, L., and Chi, Y. (2024). Sample complexity of offline distributionally robust linear markov decision processes. arXiv preprint arXiv:2403.12946. Wang, Y., Miao, F., and Zou, S. (2022). arXiv:2209.06866.

Robust constrained reinforcement learning.

arXiv preprint

Weng, J., Chen, H., Yan, D., You, K., Duburcq, A., Zhang, M., Su, Y., Su, H., and Zhu, J. (2022). Tianshou: A highly modularized deep reinforcement learning library. Journal of Machine Learning Research, 23(267):1–6. Wiesemann, W., Kuhn, D., and Rustem, B. (2013). Robust markov decision processes. Mathematics of Operations Research, 38(1):153–183. Xiong, Z., Eappen, J., Zhu, H., and Jagannathan, S. (2022). Defending observation attacks in deep reinforcement learning via detection and denoising. arXiv preprint arXiv:2206.07188. Xu, H. and Mannor, S. (2010). Distributionally robust markov decision processes. Advances in Neural Information Processing Systems, 23. Yang, R., Bai, C., Ma, X., Wang, Z., Zhang, C., and Han, L. (2022). Rorl: Robust offline reinforcement learning via conservative smoothing. Advances in neural information processing systems, 35:23851–23866. Yang, R., Wang, J., Wu, G., and Li, B. (2024). Uncertainty-based offline variational bayesian reinforcement learning for robustness under diverse data corruptions. Advances in Neural Information Processing Systems, 37:39748–39783. Younis, O. G., Perez-Vicente, R., Balis, J. U., Dudley, W., Davey, A., and Terry, J. K. (2024). Minari. Software, version 0.5.0. Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S. (2020a). Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on robot learning, pages 1094–1100. PMLR. Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. (2020b). Mopo: Modelbased offline policy optimization. Advances in neural information processing systems, 33:14129–14142. Zhang, H., Chen, H., Boning, D. S., and Hsieh, C.-J. (2021). Robust reinforcement learning on state observations with learned optimal adversary. In International Conference on Learning Representations.

18

Zhang, H., Chen, H., Xiao, C., Li, B., Liu, M., Boning, D., and Hsieh, C.-J. (2020). Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in neural information processing systems, 33:21024–21037. Zhang, S., Wang, Y., Chang, H., Zhao, H., Liu, Y., Guizilini, V., Bobu, A., Wagenmaker, A., Dixit, A., Yu, C., Shah, D., and Simchowitz, M. (2026). Post-training for robotics foundation models dataset and challenge. RSS 2026 Workshop & Challenge. Zouitine, A., Bertoin, D., Clavier, P., Geist, M., and Rachelson, E. (2024). Rrls: Robust reinforcement learning suite. arXiv preprint arXiv:2406.08406.

A

Benchmark construction

A.1

Benchmark Comparison

Benchmark RLlib (Liang et al., 2018) RRLS (Zouitine et al., 2024) ODRL (Lyu et al., 2024) Robust-Gymnasium (Gu et al., 2025) RWRL Suite (Dulac-Arnold et al., 2020) RoAd-RL (Mohan et al., 2026) RobustRLlib (ours)

Shifts

Robust algorithms

MDP

Latency

Semantic

Compound

Task expansion

0 4 0 4 0 0 17

0/4 2/4 1/4 4/4 3/4 1/4 4/4

✗ ✗ ✗ ✗ ✓ ✗ ✓

✗ ✗ ✗ ✓ ✗ ✗ ✓

✗ ❍ ✗ ✓ ✓ ✗ ✓

✓ ✓ ✗ ✓ ❍ ✓ ✓

Table 2: Comparison with RL libraries and robustness benchmarks. Robust algorithms counts robust and robust safe RL algorithms evaluated by each benchmark. A checkmark denotes explicit, evaluated support; ❍ denotes partial support; ✗ indicates that the capability is absent or not demonstrated in the cited release. MDP counts, out of four, the MDP components on which shift is evaluated: observations, actions, transitions, and reward/cost signals. Compound requires multiple shift sources to be active jointly. Task expansion indicates an extensible interface for carrying the benchmark to additional tasks. Table 2 compares evaluated capabilities with 6 related benchmarks. Algorithm counts include robust and robust safe RL method. Shift entries require reported evaluations, whereas task expansion denotes a documented interface for adding task backends. These criteria distinguish a benchmark’s demonstrated scope from its potential extensibility. The compared systems standardize different parts of robust RL evaluation. RLlib provides generalpurpose algorithm infrastructure; RRLS and ODRL compare methods primarily under dynamics-related uncertainty (Liang et al., 2018; Lyu et al., 2024; Zouitine et al., 2024). Robust-Gymnasium offers broad disrupted environments, including semantic tasks, while the RWRL Suite evaluates practical stressors such as latency and their combination (Dulac-Arnold et al., 2020; Gu et al., 2025). RoAd-RL focuses on observation attacks and inference-time defenses (Mohan et al., 2026). Rather than claiming these capabilities individually as new, RobustRLlib combines a structured robust online, offline, and safe RL library with isolated and composed shift evaluations under a shared protocol. Its task-specific adapters further demonstrate how the evaluation methodology extends from standard control to advanced robotics tasks such as Isaac Lab and a VLA pipeline.

A.2

Disruptor toolbox: shift modes, operators and implementations

A perturbation configuration is specified by (source, factor, mode, magnitude, resampling rule, optional schedule, seed). The source identifies where the intervention enters the agent–environment interaction (Section 3.2), the factor names the perturbed quantity (e.g., gravity, friction, or an observation

19

Source Stochastic Observation, action, reward/cost Observation

Operator

Intervention

Magnitude and resampling

gauss, uniform

Add componentwise signal noise

relative

Scale noise by each feature’s reference spread, σi = cσ̂i Draw and hold a sensor offset Sample physical parameters or their multipliers

Standard deviation or sampling range; per step Relative scale c; per step

Latency

bias gauss, uniform, loguniform buffer, interp

Latency

substep

Vary how many simulator substeps hold one policy action

Adversarial Observation

adversarial

Radius ϵ; per step

Action

oppose

Action

rotate

Action

oppose_goal

Search for a policy-dependent perturbation within an ℓ∞ ball Subtract a bounded component along the chosen action Change action direction while preserving magnitude; 2-D actions only Remove part of the goal-directed action component

shift

Add a specified signal offset

Signal units; fixed

scale, set

Multiply or assign a named model parameter

translate hue, tint, swap, brightness, blur, translate

Move a named body, object, or goal Change appearance or relocate a named goal/object

Ratio or parameter value; fixed Metric displacement; fixed Hue angle, tint colour, named swap, brightness factor, blur radius (px), or displacement (m); fixed

Non-stationary Any compatible source

schedule

Vary an operator’s severity by g(t)

Dynamics

push

Apply an intermittent force to a named body

Reward/cost

delay

Release accumulated feedback every k steps or shift it by k steps

Composition Compatible sources

ordered list

Stack specifications with independent random streams

Observation Dynamics

Parametric Observation, action, reward/cost Dynamics Dynamics, semantics Semantics

Delay a buffered command or interpolate timestamped observations

Offset range; per episode Parameter range; per reset Control steps or milliseconds; configured refresh Substeps; per policy decision

Opposition magnitude; per step Rotation angle; per step Opposition strength; per step

Ramp, step, or sinusoidal profile Force, duration, and interval; per event Delay k; flush at episode end Explicit order and seeds

Table 3: Shift operator index. The first three families construct interventions; non-stationary and composition determine how interventions evolve and combine. Each row gives the compatible source, mode, intervention, and magnitude or resampling rule. channel), the mode specifies how the intervention is constructed, and the magnitude sets how far it departs from nominal in the operator’s natural units. The resampling rule determines whether a value is redrawn at each step, held for an episode, or fixed. The optional schedule controls when the intervention is active and how its magnitude evolves, while the seed makes stochastic draws reproducible. The main text summarizes five shift modes; here we detail their source-specific modes, indexed in Table 3. A.2.1

Shift source: Latency

We treat latency as a composable deployment-shift source within this unified robust RL evaluation framework. Specifically, we expose observation delay, action delay, and control-period variation through õt = 20

(κ )

I(Ht , t − δo ), ãt = at−δa , st+1 ∼ P0 t (· | st , ãt ), capturing delayed sensing, delayed execution, and variable control rates, respectively. The three operators act at different points of one decision and are implemented accordingly. • Observation latency. A timestamped history Ht is queried at t − δo using linear interpolation, so sub-step delays are not rounded away (Tan et al., 2018). If the requested time precedes reset, the history returns the reset observation. • Action latency. An integer buffer executes the command from δa control steps earlier, ãt = amax(t−δa ,0) , using the initial command to warm-start the buffer. • Control-period variation. Following Peng et al. (2018), we set ∆tt = ∆t0 + ξt , where ξt ∼ Exp(λ); λ is drawn per episode and ξt per decision. One command is held for κt = round(∆tt /hsim ) simulator substeps, changing its duration time. A.2.2

Operator details

• Stochastic. Signal noise is redrawn per step, whereas bias persists for an episode and dynamics parameters are sampled at reset. loguniform samples in log space to balance reciprocal multipliers. Latency draws act on command buffers, timestamped observations, or action-hold duration. • Adversarial. The observation operator searches for an input that changes the current actor’s output within radius ϵ; it requires a differentiable feed-forward actor. The three action operators respectively oppose the chosen command, rotate a 2-D command, or suppress its goal-directed component. Commands are clipped to the action space before perturbation. • Parametric. These operators apply a specified value rather than draw one. set handles parameters whose nominal value is zero, while translate expresses a displacement in physical units. Semantic operators change appearance or task referents. • Non-stationary. A schedule changes the shift magnitude over time, whereas latency operators change when signals enter the control loop. push creates an intermittent event. For feedback delay, interval release sums rewards over k steps, while shifted release delivers each reward k steps later; both flush pending feedback at episode end. • Composition. An ordered list specifies wrapper order explicitly. Each specification uses its own seeded random stream, making compound conditions reproducible.

B

Evaluation protocol

B.1

Metrics

Normalized score. To compare standard-control tasks with different native metrics, let R̄ denote the mean raw task score over evaluation episodes. We report the unclipped normalized score S(R̄) = 100

R̄ − Rmin , Rmax − Rmin

(2)

where Rmin and Rmax are fixed task-level reference scores shared by all methods, seeds, and shift conditions. We use published D4RL references where available and otherwise use task-defined or reference-policy anchors; all values are listed in Table 7. Normalization precedes cross-task aggregation. For the standard-control experiments, we map each task’s native metric to a common comparable scale. Performance and safety. We report performance and safety separately on the advanced robotic tasks. For G1, performance is PPO-reference-normalized return over five out-of-envelope conditions (Table 7), while safety is the fraction of episodes surviving the 20 s horizon under 20–80 ms action delays. For Drawer, performance is the fraction of episodes opening the drawer beyond 0.30 m over fourteen out-of-envelope conditions; safety is the fraction whose cumulative joint-limit cost remains below the per-episode budget 21

Shift factor

Unit

Q1

Q2

Q3

Q4

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

HalfCheetah-v5

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

Walker2d-v5

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

FetchReachDense-v3

Morphology Actuator gear

× ×

0.8 0.4

1.2 0.2

1.4 0.1

1.6 2

PointMaze_UMaze-v3

Actuator gear Dry friction Adversarial action

× N –

0.8 10 oppose (0.6)

1.2 20 opposegoal (0.6)

1.4 30 opposegoal (0.8)

0.6 40 rotate (70)

CarRacing-v2

Morphology Actuator gear Surface friction

× × ×

0.8, 1.2 0.8, 1.2 0.9, 1.1

0.6 0.6 1.2

1.4 1.4 0.8

1.6, 1.8 0.4, 1.6 0.6, 0.7

Hopper-v5

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

HalfCheetah-v5

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

Walker2d-v5

Gravity Morphology Actuator gear

× × ×

0.9, 1.1 0.9, 1.1 0.9, 1.1

0.8, 1.2 0.8, 1.2 0.8, 1.2

0.7, 1.3 0.7, 1.3 0.7, 1.3

0.6, 1.4 0.6, 1.4 0.6, 1.4

AdroitHandDoor-v1

Actuator gear Latch friction

× ×

0.8 2

1.2 3

1.4 4

0.6 5

PointMaze_Open-v3

Actuator gear Dry friction Adversarial action

× N –

0.4 30 oppose-goal

0.2 45 rotate

0.1 60 rotate (90)

0.05 75 oppose

LunarLander Continuous-v3

Gravity Engine power Wind

× × N

0.9, 1.1 0.9, 1.1 5, 10

1.15 0.75 15

0.85 1.25 20

0.75, 1.25 0.6, 1.4 25, 30

Environment Online Hopper-v5

Offline

Table 4: The 220 perturbation configurations and the shift levels. Each row is one shift factor on one environment; the four cells list every perturbation configuration evaluated on that factor, grouped by shift levels. Q indexes displacement from nominal. Adversarial-action is discrete attack modes with no scalar distance from nominal.

22

Shift source

Perturbation

Mode

Setting

Observation

obs_rel_0.10 obs_uni_0.10 obs_gauss_0.10 mad_0.02 mad_0.05

stochastic stochastic stochastic adversarial adversarial

Gaussian at 0.10 σi , σi the dimension’s own std U [−0.10, 0.10] σ = 0.10 MAD gradient attack, ϵ = 0.02 MAD gradient attack, ϵ = 0.05

Action

act_uni_0.10 act_gauss_0.10 act_offset_0.05 act_oppose_0.10 cmd_oppose_0.20

stochastic stochastic parametric adversarial adversarial

U [−0.10, 0.10] on the clipped action σ = 0.10 on the clipped action +0.05 on every actuator ϵ = 0.10 against the action just chosen ϵ = 0.20, applied to the command before clipping

Dynamics

gear_static_0.8 gear_ramp_0.8 gear_sine_p500 push_60 trans_t1

parametric non-stationary non-stationary non-stationary stochastic

actuator gear × 0.8 for the whole episode gear 1.0 → 0.8, linear over 1000 steps gear cycling in [0.8, 1.2], period 500 steps 60 N on the torso, 12 steps every 250, sign random state noise σq = 10−3 , σv = 10−2 per step

stochastic stochastic stochastic parametric stochastic

delay drawn from [2, 4] ms, interpolated delay drawn from [4, 8] ms delay drawn from [8, 16] ms fixed buffer of 1 control step (8 ms) buffer redrawn each episode, {0, 1} steps

Latency

obs_delay_4ms obs_delay_8ms obs_delay_16ms act_delay_8ms act_delay_rand1 In-training perturbation Reward Gaussian, uniform, constant Latency(Reward) credit ladder

signal corruption three arms matched at σ = 0.5 (uniform ±0.866) interval reward release k ∈ {4, 16, 32, 64} control steps

Table 5: 44 individual shift perturbation configuration. The 22 fixed sets on Hopper-v5 are instantiated separately for online and offline evaluation, composing 44 perturbation configurations used to compute the scores in Section 4.2, and the In-training perturbation means the shift acts during learning, the evaluation is nominal. A control step is 8 ms (frame_skip 4 × 2 ms).

Shift source

Shift factor

Resampling rule

Setting

Hopper-v5 ← OmniH2O (He et al., 2024) Dynamics Dynamics Dynamics Dynamics Action Latency

Foot/link friction Link mass Actuator gear Torso push Torque noise Action delay

log-uniform, per episode log-uniform, per episode log-uniform, per episode onset phase per episode, sign per push Gaussian, per step, pre-clip uniform integer, per episode

×[0.38, 1.06] ×[0.81, 1.17] ×[0.84, 1.14] 119 N, every 625 steps, 10 steps σ = 0.06 [1, 2] steps

Pusher-v5s ← Peng et al. (2018) Table I Dynamics Dynamics Dynamics Dynamics Latency Observation Semantics

Link mass Object mass Joint damping Actuator gear Control period Sensor noise Goal displacement

log-uniform, per episode log-uniform, per episode log-uniform, per episode log-uniform, per episode λ per episode, dt per step Gaussian, per step, per-feature std deterministic

×[0.03, 32] ×[0.11, 3.8] ×[0.02, 1.8×103 ] ×[0.18, 5.7] dt = dt0 +Exp(1/λ), λ ∈ [50, 400] 0.125 σi 0.25 m diagonal

Table 6: Reference-derived compound shifts scenario configuration. Every factor of a profile is applied jointly in one mode; this sheet lists the reference system’s own factor settings.

23

Rmin

Rmax

episodic return episodic return episodic return

−20.27 −280.18 1.63

3234.3 12135.0 4592.3

D4RL v2 random / expert D4RL v2 random / expert D4RL v2 random / expert

AdroitHandDoor-v1 PointMaze_Open-v3 LunarLanderContinuous-v3

episodic return goals per 300 steps episodic return

−56.51 0.0 −213.9

2880.57 3.28 136.9

D4RL v2 random / expert no goal/ IQL nominal random policy / IQL nominal

PointMaze_UMaze-v3 FetchReachDense-v3 CarRacing-v2 Pusher-v5s Unitree G1

goals per 300 steps neg. goal distance episodic return episodic return episodic return

0.23 −8.42 −30.44 −146.2 0

5.5 0.0 900.0 −23.97 37.30

random policy / kinematic ceiling random policy / reward maximum random policy / solved threshold random policy / PPO nominal zero floor / PPO nominal

Task

Metric

Hopper-v5 HalfCheetah-v5 Walker2d-v5

Source

Table 7: Normalization anchors (Rmin , Rmax ). The pairs entering equation 2, fixed per environment and shared by every algorithm, dataset quality, and perturbation configurations. Source names the construction behind each pair: D4RL v2 where a published pair exists, and otherwise a random-policy floor with either a task-defined or a reference-learner ceiling. Shift factor

Training setting

Evaluation setting

Contact friction Link mass Base CoM, x and y Torque noise Action delay Actuator Kp , Kd Velocity push

U [0.2, 1.1] × U [0.7, 1.3] ±0.1 m ±10% of the effort limit 1–3 policy steps (20–60 ms) × U [0.75, 1.25] 1 m/s, random direction, every 5 s

0.1 × 1.5 +0.15 m along x 20% 4 steps (80 ms) boundary only, 0.75 and 1.25 boundary only, 1 m/s

Table 8: G1: training and evaluation configuration. of 20. We average episodes within each training seed, then report the mean and standard deviation across seeds.

B.2

Evaluation grids

The main robust RL algorithms evaluation grid varies one shift factor at a time across online and offline tasks, grouping its non-nominal settings into four task–factor-specific shift levels (Table 4). The grid is fixed and same for every algorithm: 220 non-nominal conditions, 110 online and 110 offline, plus the nominal setting of every factor. The individual shift grid instead varies intervention modes within a shift source, allowing method-level specialization to be diagnosed (Table 5). Finally, the compound grid activates physical, control, sensing, and timing factors jointly in reference-derived Hopper and Pusher tasks (Table 6). Thus, the three grids measure robustness breadth, shift-specific behavior, and robustness under interacting deployment factors, respectively. The advanced tasks bring their own grids, and both place every method on the same conditions. On Isaac Lab the conditions are positioned relative to the reference envelope (He et al., 2024) of Tables 8 and 9: G1 has 31 per seed, six inside that envelope, eighteen on its boundary, five beyond it, and flat and rough nominal. Drawer has 16, two controls and fourteen beyond the reference ranges, each moving one axis while the rest keep their training randomization. The VLA evaluation tests eight shift factors across five shift sources (Figure 9). Each factor has its own Q1–Q4 ladder, ordered outward from its nominal setting. Hardware logs place execution delay (83 ms) and command noise (0.020 rad) at Q2, so the sweep includes both milder and more severe conditions than those measured on the platform. We report the actual input change for perception and command shifts separately; the three dynamics shifts are specified in physical units.

24

Shift factor

Training setting

Evaluation setting

Robot friction Handle friction Cabinet mass Arm Kp Arm Kd Drawer damping Action delay

static and dynamic U [0.8, 1.25] static U [1.0, 1.25], dynamic U [1.25, 1.5] × U [0.8, 1.2] × U [0.9, 1.1] × U [0.9, 1.1] × U [0.5, 1.5] 0–2 policy steps

both 0.4; both 1.8 both 0.5; both 2.0 × 0.6; × 1.4 × 0.75; × 1.25 × 0.75; × 1.25 × 0.25; × 2.0 3 steps (50 ms); 4 steps (66.7 ms)

Table 9: Drawer: training and evaluation configuration.

Shift levels of each factor Gravity (×) Joint damping (×) Dry friction (+N) Command noise ( rad) Execution delay (steps) Proprio. noise ( rad) Brightness (×) Blur radius (px)

×1

0.8 1.2

0.6 1.4

0.25 2.0

×1

25

8 20

50 500

+0

0.25 1.0

1.5 4

8 64

0

0.01

0.02

0.05

0.10

0

2

5

10

20

0

0.005

0.01

0.02

0.05

×1

0.75

0.5

0.25

0.1

0

1

2

3

4

nominal Q1 Q2 Q3 Q4 shift level (nearest farthest from nominal) dynamics action

latency observation

semantic real robot's measured operating point

Figure 9: VLA evaluation grid. Each row is one shift factor, coloured by its shift source, with the nominal value on the left and the setting of each level above its circle. Q1–Q4 order the levels from nearest to farthest from nominal; the three dynamics shift have three levels. The diamond marks the operating point measured on the real robot: it sits at Q2, so Q1 is milder than the hardware and Q3–Q4 go beyond it.

C

Benchmark tasks and datasets

In this section we present an overview of the environments the benchmark evaluates on, the control interface each of them exposes, and the offline datasets learn from. The main experiment in Section 4.1 covers locomotion, manipulation, navigation, and vehicle control; Pusher and DoorCausal provide the additional semantic-channel tests and compound-profile in Section 4.2. Table 10 gives the default interface of every task we included in our benchmark. Offline methods learn from the fixed datasets in Table 11. We use Minari (Younis et al., 2024), including its D4RL ports (Fu et al., 2020), wherever a corresponding dataset is available; only for LunarLander do we

25

collect our own dataset. Task

Observation

Action (rate)

Horizon

Task type

Standard control Hopper-v5 Walker2d-v5 HalfCheetah-v5 AdroitHandDoor-v1 FetchReachDense-v3 Pusher-v5s DoorCausal PointMaze_UMaze-v3 PointMaze_Open-v3 CarRacing-v2 LunarLanderContinuous-v3

11-D state 17-D state 17-D state 39-D state 13-D proprioception and relative goal 23-D state 54-D State 6-D state and desired goal 6-D state and desired goal 96 × 96 × 3 pixels 8-D state

3-D 6-D 6-D 28-D 4-D 7-D 8-D 2-D 2-D 3-D 2-D

1,000 1,000 1,000 200 50 100 300 300 300 1,000 1,000

Locomotion Locomotion Locomotion Manipulation Manipulation Manipulation Manipulation Navigation Navigation Driving Driving

123-D state 31-D state 14-D joint state and three 180×320×3 camera streams

37-D (50 Hz) 8-D (60 Hz) 14-D (60 Hz)

1,000 480 Demo

Locomotion Manipulation Bimanual manipulation

Advanced robotics Unitree G1 Franka Drawer insert-mouse-battery

Table 10: Benchmark tasks. Observation and Action specify the default policy interface; control rates are shown in parentheses where reported. Numeric horizons are maximum environment steps per episode, while Demo follows the length of the recorded demonstration used for VLA evaluation.

Task

Dataset

Source or collection

Hopper, Walker2d, HalfCheetah AdroitHandDoor PointMaze_Open LunarLanderContinuous

Minari medium-v0

Collected in Gymnasium v5 environments

D4RL/door/expert-v2 D4RL/pointmaze/open-v3 Collected for this benchmark

Minari port of D4RL trajectories Minari port of D4RL trajectories Deterministic SAC checkpoints trained for 1.5 million steps; selected medium corpus of 1,741 episodes

Table 11: Offline datasets. Each dataset contains approximately one million transitions.

Advanced robotic tasks. These tasks test robust safe algorithms and show our interface extensibility meanwhile. In Isaac Lab (Mittal et al., 2025), Unitree G1 (Figure 10) locomotion measures return and survival under delay, and Franka drawer manipulation measures success and joint-limit cost under physical shifts. The VLA task (Figure 11) fine-tunes π0.5 (Intelligence et al., 2025) on expert demonstrations for the bimanual insert-mouse-battery task from the RSS 2026 post-training challenge (Zhang et al., 2026), then evaluates perception and execution shifts through camera and robot-state interfaces.

26

Figure 10: Isaac Lab tasks. Left: Humanoid G1 locomotion task; right: Franka drawer manipulation task. 1s

11 s

21 s

31 s

40 s

Figure 11: VLA task. Bimanual insert-mouse-battery demonstrations of the RSS 2026 post-training challenge (Zhang et al., 2026).

D

Algorithms

The library holds six standard RL baselines without any robust design, thirteen robust RL algorithms, and two robust safe algorithms. They are organized by the taxonomy of Section 3.1.

D.1

Offline algorithms

D.1.1

Standard RL baselines.

IQL IQL (Kostrikov et al., 2021) avoids querying out-of-distribution actions altogether: a value function is fitted to the critic by expectile regression, the twin critics bootstrap from that value function, and the actor is extracted by advantage-weighted regression. TD3+BC TD3+BC (Fujimoto and Gu, 2021) adds a behavior-cloning term to the TD3 actor loss and normalizes states. The term is balanced against the critic by dividing by the mean absolute Q over the batch. MOPO MOPO (Yu et al., 2020b) learns an ensemble dynamics model, branches short rollouts from dataset states, and penalizes the rollout reward by the model’s predicted uncertainty before running SAC on the mixture of real and model data. SynthER SynthER (Lu et al., 2023) fits a diffusion model to logged transitions and trains an unmodified offline learner on a much larger sample from it. We count it as data augmentation rather than a generative method: it samples individual transitions to enlarge the dataset, where FWM and PLR-PVL sample multistep futures conditioned on the current state and turn them into critic targets.

27

D.1.2

Learner-centric algorithms.

RFQI RFQI (Panaganti et al., 2022) replaces the fitted-Q target by a worst-case Bellman backup over a total-variation uncertainty set of radius ρ around the empirical transitions; the inner maximization is solved through a dual variable fitted per batch, and the actor and critic follow the release’s BCQ-style architecture. RORL RORL (Yang et al., 2022) makes an SAC learner conservative through smoothing: the critic and policy are regularized to vary little within an ϵ-ball of dataset states, and Q-values at perturbed states are penalized by the disagreement of a ten-critic ensemble. ATLA-IQL The offline ATLA adaptation (Zhang et al., 2021) keeps ATLA’s principle, a learned observation adversary alternated with the learner, but the adversary is trained by a differentiable objective on the critic by minimizing the minimum Q at the policy’s action under the perturbed state since no environment is available. D.1.3

Data-centric algorithms.

RSC-IQL RSC-IQL (Ding et al., 2023a) applies RSC’s counterfactual rewriting to logged data where a causal mask over state dimensions is fitted from the dataset, and transitions are rewritten by permuting or perturbing the dimensions the mask marks as non-causal. RAMBO RAMBO (Rigter et al., 2022) augments the dataset with short rollouts from a learned dynamics ensemble and replaces the uncertainty penalty by adversarial training of the model: the ensemble is updated to lower the critic’s value of the policy’s transitions while a maximum-likelihood term keeps it close to the data. D.1.4

Generative model.

ROMB ROMB (Chen et al., 2025) adapts a learned world model against the policy under a constrained maximin objective: the model is pushed toward transitions that lower the policy’s value while a supervised loss keeps it on the data, and the policy is trained on rollouts from the adapted model. FWM FWM transfers the multi-step future modeling of Diffusion World Model (Ding et al., 2024) to a flow-matching (Lipman et al., 2022) world model: instead of a one-step bootstrap, the critic regresses an h-step return whose future is sampled from the generative model conditioned on the current state, action and return-to-go. Writing x1 for a length-T window of states and rewards and x0 ∼ N (0, I), the diffusion model’s denoising objective becomes a regression onto the straight-line velocity of the linear path xt = (1−t) x0 +t x1 , 2

L = Et∼U [0,1], x0 , (x1 ,c,g) vθ (xt , t, c, g) − (x1 − x0 ) 2 ,

(3)

with c the current state and action and g the return-to-go, dropped with probability 0.2 so one network carries both the conditional and the unconditional velocity. Sampling then replaces the reverse diffusion chain by an ODE integrated from t = 0 to t = 1 in N = 5 midpoint steps on the guidance-mixed velocity: ṽθ = w vθ (· | c, g) + (1 − w) vθ (· | ∅),

w = 2.0.

(4)

PLR-PVL PLR-PVL (Berdica et al., 2024) trains a collection of world models consistent with the offline data, treats each as a level in the sense of unsupervised environment design, and prioritizes them during policy learning by the value loss they induce. We adapt the method by replacing their world model with the flow-matching model of equation 3: the ensemble is K flow models differing only in training seed, each proposing one h-step future for the same conditioning, and the critic target is taken per sample from the model whose rollout carries the largest positive value loss, k ⋆ = arg max k

(k)

where δt

h
1X (k) max δt , 0 , h t=1

is the GAE-λ residual of the current critic along the k-th rollout. 28

(5)

D.2

Online algorithms

D.2.1

Standard RL baselines.

SAC SAC (Haarnoja et al., 2018) is the off-policy reference: twin critics with a clipped double-Q target and a stochastic actor whose objective adds an entropy bonus with automatically tuned temperature. PPO PPO (Schulman et al., 2017) is the on-policy reference: a clipped probability-ratio surrogate on trajectories from the current policy, with generalized advantage estimation. TRPO TRPO (Schulman et al., 2015) replaces PPO’s clipped surrogate by an explicit KL trust region solved with conjugate gradients. It appears only as the base of RARL-TRPO. D.2.2

Learner-centric algorithms

ATLA ATLA (Zhang et al., 2021) trains a policy against an optimal observation adversary in the stateadversarial MDP (Zhang et al., 2020): the adversary is itself an RL agent that outputs a bounded perturbation of the observation, is rewarded by the protagonist’s negative return, and is updated in alternation with the protagonist. ATLA-SA ATLA-SA (Zhang et al., 2021) adds the SA-PPO regularizer (Zhang et al., 2020) to ATLA: a KL penalty between the policy at a state and at the worst-case neighbor found by stochastic gradient Langevin dynamics, and a recurrent (LSTM) policy with a 100-step history so that the agent can infer a persistent perturbation from context. RSC RSC (Ding et al., 2023a) learns a causal graph and dynamics model over the state variables from replayed experience, then generates counterfactual transitions by intervening on the variables the graph marks as non-causal for the reward. Mixing these into replay removes the spurious correlations a policy would otherwise exploit. D.2.3

Environment-centric algorithms.

RARL RARL (Pinto et al., 2017) casts robustness as a two-player zero-sum game: an adversary applies bounded external forces to the robot and is trained to minimize the protagonist’s return, and the two are updated in alternating blocks. We implement RARL-PPO and RARL-TRPO in our library. DR Domain randomization (Peng et al., 2018; Tobin et al., 2017) samples simulator parameters from fixed ranges at every episode so that the policy meets a distribution of dynamics during training. We implement DR-PPO and DR-SAC in our library. D.2.4

Robust safe methods.

RAMU RAMU (Queeney and Benosman, 2023) applies a risk measure over sampled next-state perturbations to both the reward and the cost targets; the original is defined for TD learners, and the row reported here is a PPO adaptation that applies the same transform to PPO’s value targets. SPiDR SPiDR (As et al., 2026) trains a constrained policy under domain randomization and adds a pessimistic cost penalty proportional to the disagreement of next-state predictions across sampled dynamics; it runs with the authors’ solver and a joint-limit cost budget.

29

E

Training Settings

E.1

Training protocol

We train each method with five independent seeds to its budget in Table 14. Unless otherwise noted, we freeze the final checkpoint and evaluate each policy for 20 episodes per condition using the same evaluation seed. RFQI retains its native best-checkpoint selection rule. Our default protocol trains on the nominal task and introduces shifts only when evaluating the frozen policy. Reward shift and latency delivery are exceptions: they affect learning feedback but cannot alter the actions of a frozen policy. We therefore apply these shifts during training and evaluate the resulting policies on the nominal task. The Isaac Lab and VLA studies are reported under their own protocol, and the departures are the following. Isaac Lab tasks use three training seeds and score 128 episodes per condition. They match the retained primary training interactions within each task to the budget in Table 15. The five methods share one base learner, so the comparison is between mechanisms rather than between releases. And they do not hold the shift grid out from training: PPO and RAMU-PPO train on the nominal specification, whereas DR-PPO, ATLA-PPO and SPiDR-PPO each train inside a randomization envelope. Table 8 and Table 9 show the envelopes applied in our benchmark, which is transferred from a humanoid sim-to-real pipeline (He et al., 2024). The VLA task finetunes a single π0.5 model on demonstrations, and evaluated by joint-space tracking against the recorded demonstration.

E.2

Training budget alignment

This part presents our training budget and provides the evidence of our training protocol alignment. We train SAC-based algorithms for 1M environment steps and PPO-based algorithms for 2M steps, consistent with commonly used training budgets (Raffin and Stulp, 2020). This asymmetric budget accounts for PPO’s lower sample efficiency: as an on-policy method, PPO generally requires more environment interactions to reach a comparable level of training maturity. Figure 12 shows that the additional PPO steps provide meaningful learning progress. PPO (2M)

Nominal evaluation return

Hopper

Walker2d

SAC (1M)

HalfCheetah

FetchReach

PointMaze_UMaze

CarRacing

4500

6300

12300

0

6

1080

3000

4200

8200

−1

4

720

1500

2100

4100

−2

2

360

0

0

0

−3

0

0

−1500 0.0

0.5

1.0

1.5

2.0

−2100 0.0

0.5

1.0

1.5

2.0

−4100 0.0

0.5

1.0

1.5

2.0

−4 0.0

0.5

1.0

1.5

2.0

−2 0.0

0.5

1.0

1.5

2.0

−360 0.0

0.5

1.0

1.5

2.0

Environment steps (millions)

Figure 12: Online training curves. However, more environment steps do not imply a larger computational budget. On Walker2d, PPO at 2M steps requires only 0.80 ± 0.29 hours, compared with 1.60 ± 0.03 hours for SAC at 1M steps on H100 GPUs (Table 12). Thus, the 2M-step PPO budget compensates for its lower sample efficiency without giving it an advantage in wall-clock compute. For ATLA and ATLA-SA, their official configuration remains unchanged, which is about 5M protagonist steps rather than the 2M the other PPO rows use. The asymmetry is also the one the ATLA paper itself reports under: its released configurations give vanilla PPO and SA-PPO 976 iterations of the same 2048-step rollout, i.e. the same 2M our PPO row uses, against 2441 for ATLA and ATLA-SA. Our PPOversus-ATLA comparison therefore reproduces the budget pairing those results were established with rather than introducing a new one. The full online training budget is in Table 14. Taken together, the online budget is set by the base algorithm. Table 13 summarizes the resulting protocol. Methods sharing the same base algorithm are trained with the same budget, while ATLA and ATLA-SA retain their official ∼5M-step configuration.

30

Task

Method

Env. steps

Wall-clock time (h)

Walker2d-v5

PPO PPO SAC

1M 2M 1M

0.40 ± 0.15 0.80 ± 0.29 1.60 ± 0.03

Table 12: Walker2d training wall-clock time on H100 GPUs. Values are mean ± sample standard deviation over three training seeds. Wall-clock time is measured from the start of training to the corresponding checkpoint save. Budget rule

Methods

Steps

SAC-based PPO/TRPO-based Official ATLA configuration

SAC, DR-SAC, RSC-SAC PPO, RARL-PPO, RARL-TRPO ATLA, ATLA-SA

1M 2M ∼5M

Table 13: Online training budget summary.

Algorithm

Feature

Base algorithm

Budget

Online SAC PPO ATLA ATLA-SA RSC-SAC RARL-PPO / -TRPO DR-SAC RAMU SPiDR

Standard Standard Learner-centric Learner-centric Learner-centric Env.-centric Env.-centric Safe RL Safe-robust

— — PPO PPO (LSTM) SAC PPO / TRPO SAC PPO PPO

1M steps 2M steps 2441×2048 steps 2441×2048 steps 1M steps 2M steps 1M steps Isaac Lab budget Isaac Lab budget

Offline IQL TD3+BC MOPO SynthER RFQI RORL ATLA-IQL RSC-IQL RAMBO ROMB FWM PLR-PVL

Standard Standard Standard Standard Learner-centric Learner-centric Learner-centric Data-centric Data-centric Generative model Generative model Generative model

— — SAC IQL FQI SAC IQL IQL SAC IQL IQL IQL

1M, batch 256 1M, batch 256 1M, batch 256 1M, batch 256 500k, batch 1000 3M (MuJoCo, Door) / 1M 1M, batch 256 1M, batch 256 2M (MuJoCo, Door) / 1M 1M, batch 256 1M, batch 256 1M, batch 256

Table 14: Training budget. Budgets are environment steps for online methods and policy/critic gradient updates for offline methods. Training budget alignment explanation is in Section E.2. For the offline algorithms’ training budgets, Table 14 shows nine of the twelve offline columns share the same 1M-update budget. RAMBO and RORL (Rigter et al., 2022; Yang et al., 2022) follow their official budgets on the tasks their releases cover and keep the 1M budget elsewhere. Thus, our MuJoCo comparisons with 1M-update references preserve native budgets. RFQI (Panaganti et al., 2022) follows its official budget and reports its own best checkpoint, because forcing it onto the common protocol would break its mechanism.

31

Task

PPO, DR-PPO, RAMU-PPO

ATLA-PPO

SPiDR-PPO

G1 Drawer

1500 × 4096 × 24 3200 × 512 × 96

750 × 2 × 4096 × 24 1600 × 2 × 512 × 96

900 × 512 × 320 3200 × 512 × 96

B 147,456,000 157,286,400

Table 15: Training budgets for Isaac Lab tasks. Iterations are chosen so that the retained primary training interactions B match within each task.

F

Hyperparameter settings

F.1

Online algorithm hyperparameters

The tables below report algorithm hyperparameters only. H, HC, and W2D denote Hopper , HalfCheetah, and Walker2d, respectively. A combined H/HC/W2D column is used only when all listed values agree across the three tasks. Dashes indicate that a parameter does not apply. Parameter

H/HC/W2D PointMaze FetchReach CarRacing

Learning rate Hidden layers Discount factor γ GAE parameter λ PPO clipping parameter Value-loss coefficient Entropy coefficient Rollout length Minibatch size Optimization epochs Maximum gradient norm

3 × 10−4 2 × 64 2 × 64 0.99 0.95 0.2 0.5 0 2048 64 10 0.5

2 × 64

NatureCNN

Table 16: PPO hyperparameters.

Parameter

H/HC/W2D PointMaze FetchReach −4

−4

−4

CarRacing

Learning rate Hidden layers (actor / critic) Target smoothing τ Replay-buffer size Learning starts Train frequency / gradient steps

3 × 10 2 × 256 0.005 106 100 1/1

3 × 10 2 × 256 0.005 106 100 1/1

3 × 10 2 × 256 0.005 106 100 1/1

10−4 NatureCNN +2 × 256 0.02 2 × 105 104 8/8

Discount factor γ Target entropy Minibatch size

automatic

automatic

0.99 automatic 256

automatic (−3)

Table 17: SAC hyperparameters.

32

Parameter

Hopper Walker2d HalfCheetah PointMaze

Adversary target foot both feet Adversary magnitude 5N 5N Update ratio 1/1 1/1 Adversary warm-up 0 0 Zero-force probability p0 0 0 Protagonist/adversary lr 3 × 10−4 3 × 10−4 Entropy coefficient 0 0

torso and feet 5N 1/1 0 0 3 × 10−4 0

Discount factor γ GAE parameter λ PPO clipping parameter

particle 10 N 2/1 5 × 104 0.25 3 × 10−4 0

FetchReach

CarRacing

Cartesian action hull 0.1 5 1/1 2/1 0 4 × 104 0 0.25 3 × 10−4 2.5 × 10−4 0.01 0.01

0.99 0.95 0.2

Table 18: RARL-PPO hyperparameters. The update ratio denotes protagonist-to-adversary update blocks.

Parameter Adversary target Adversary magnitude Update ratio Adversary warm-up Zero-force probability p0 Protagonist learning rate Adversary learning rate

Hopper Walker2d HalfCheetah PointMaze foot 5N 1/1 0 0 10−3 10−3

both feet 5N 1/1 0 0 10−3 10−3

torso and feet 5N 1/1 0 0 10−3 10−3

Discount factor γ GAE parameter λ Target KL divergence CG steps / damping Critic updates / policy update

particle 10 N 2/1 5 × 104 0.25 10−3 10−3

FetchReach

CarRacing

Cartesian action 0.1 1/1 0 0 10−3 10−3

hull 2 5/1 3 × 105 0.5 10−4 5 × 10−5

0.99 0.95 0.01 15/0.1 10

Table 19: RARL-TRPO hyperparameters. The update ratio denotes protagonist-to-adversary update blocks; CG denotes conjugate gradient.

Parameter

H/HC/W2D

PointMaze

FetchReach

CarRacing

Randomized parameters g/morph./gear friction/gear morph./gear friction/morph./gear Randomization range [0.9, 1.1] [0, 7.5]/[0.9, 1.1] [0.9, 1.1] [0.9, 1.1] Learning rate 3 × 10−4 3 × 10−4 10−3 10−4 Learning-rate schedule constant constant constant linear to 3 × 10−5 Discount factor γ 0.99 0.99 0.95 0.99 Target smoothing τ Entropy coefficient Target entropy

automatic

0.005 automatic tuning automatic automatic

−2 (fixed)

Table 20: DR-SAC hyperparameters. Physical parameters are sampled independently at reset and remain fixed within an episode. Morph. represents Morphology

33

Parameter

H/HC/W2D PointMaze FetchReach

Actor learning rate Critic learning rate Learning-rate schedule Causal/RSC warm-up Replay augmentation / RSC mix Selected fraction Reference size Dynamics learning rate

−4

−4

3 × 10 10−3 constant 105 0.25 – 128 –

−4

3 × 10 3 × 10−4 constant 3 × 105 0.10 0.25 – 10−4

Discount factor γ

3 × 10 10−3 constant 105 0.25 – 128 –

CarRacing 10−4 10−4 linear to 3 × 10−5 4 × 105 0.05 0.25 – 10−4

0.99 Table 21: RSC-SAC hyperparameters.

Parameter

Hopper Walker2d HalfCheetah PointMaze FetchReach CarRacing

Observation radius ϵ 0.075 Policy learning rate 3 × 10−4 Value learning rate 2.5 × 10−4 Adversary policy lr 3 × 10−3 Adversary value lr 3 × 10−5 Adversary updates per block 1 Initial policy std. dev. 1.0

0.05 4 × 10−4 3 × 10−4 10−3 10−4 1 1.0

0.15 10−4 10−3 3 × 10−3 3 × 10−4 1 0.5

Discount factor γ GAE parameter λ PPO clipping parameter

0.075 3 × 10−4 2.5 × 10−4 3 × 10−3 3 × 10−5 1 1.0

0.075 3 × 10−4 2.5 × 10−4 3 × 10−3 3 × 10−5 1 1.0

8/255 10−4 10−4 10−4 – 4 –

0.99 0.95 0.2

Table 22: ATLA-PPO hyperparameters.

Parameter Observation radius ϵ SA regularization weight Policy learning rate Value learning rate Adversary policy lr Adversary value lr Adversary updates per block Initial policy std. dev. History length

Hopper Walker2d HalfCheetah PointMaze FetchReach CarRacing 0.075 0.3 10−3 10−2 10−2 10−2 1 1.0 100

0.05 0.3 10−3 3 × 10−2 3 × 10−4 10−2 1 1.0 100

0.15 0.03 10−3 3 × 10−3 10−2 3 × 10−4 1 0.5 100

Discount factor γ GAE parameter λ PPO clipping parameter State-adversarial inner steps

0.075 0.3 10−3 10−2 10−2 10−2 1 1.0 100

0.075 0.3 10−3 10−2 10−2 10−2 1 1.0 100

8/255 0.1 10−4 10−4 10−4 – 4 – –

0.99 0.95 0.2 2

Table 23: ATLA-SA-PPO hyperparameters. State-task ϵ values act on normalized observations; CarRacing uses 8 raw pixel levels, equivalent to 8/255 after image normalization.

34

F.2

Offline algorithm hyperparameters

The tables below give the algorithm hyperparameters behind the reported offline results. Dashes indicate that a parameter does not apply. Parameter

Hopper Walker2d HalfCheetah Door PointMaze LunarLander

AWR temperature β Policy dropout

3.0 —

3.0 —

3.0 —

Expectile τ Advantage clip Learning rate Hidden layers Discount factor γ Target EMA rate Batch size

0.5 0.1

3.0 —

3.0 —

0.7 100 3 × 10−4 2 × 256 0.99 0.005 256 Table 24: IQL hyperparameters.

Parameter

Value

Parameter

Value

BC weight α Noise clip Learning rate Discount factor γ State normalization

2.5 0.5 3 × 10−4 0.99 on

Target policy noise Policy delay Hidden layers Target EMA rate Batch size

0.2 2 2 × 256 0.005 256

Table 25: TD3+BC hyperparameters.

Parameter

Hopper Walker2d HalfCheetah

Rollout horizon h Penalty coeff. λ Penalty type Real data ratio

42 37.28 ens.-std 0.05

Dynamics ensemble Rollouts per refresh Target entropy Learning rate Discount factor γ Target EMA rate Batch size

3 1.0 ens.-std 0.5

5 1.0 max-aleat. 0.05

Door 1 5.0 max-aleat. 0.5

PointMaze LunarLander 5 1.0 max-aleat. 0.05

5 1.0 max-aleat. 0.05

7 networks, 5 elites 5 × 104 −3 3 × 10−4 0.99 0.005 256

Table 26: MOPO hyperparameters. The penalty type names the ensemble statistic the reward is discounted by: max-aleat. is the maximum per-member aleatoric standard deviation of the original paper, ens.-std the standard deviation across members, which (Lu et al., 2021) introduce and find to work better than the original penalty at much longer rollout horizons. The Hopper column takes its three values directly.

35

Parameter

Value

Denoiser Layers Fourier features σdata Learning rate Batch size

residual MLP 6 random 1.0 3 × 10−4 1024

Parameter

Value

Width Time embedding Sampling steps Schurn Schedule Terminals modeled

2048 128 128 80 cosine yes

Table 27: SynthER hyperparameters. The consumer is the IQL agent of Table 24, trained on the generated dataset with normalization statistics taken from the real one.

Parameter

Hopper Walker2d HalfCheetah

Adversary weight λadv 3 × 10−4 Horizon h 5 BC anchor weight 1.0 Target entropy −3

0 5 — −6

3 × 10−4 5 — −6

Real data ratio Dynamics ensemble Rollouts per refresh Critic / actor lr Adversary lr Discount factor γ Target EMA rate Batch size

Door

PointMaze LunarLander

3 × 10−4 1 1.0 −90

3 × 10−4 1 1.0 −2

3 × 10−4 1 1.0 −2

0.5 7 networks, 5 elites 5 × 104 −4 3 × 10 / 1 × 10−4 3 × 10−4 0.99 5 × 10−3 256

Table 28: RAMBO hyperparameters. The BC anchor is a behaviour-cloning term kept on the actor throughout training.

Parameter

Hopper Walker2d HalfCheetah

Door

Uncertainty radius ρ Actor lr Critic lr

0.5 1 × 10−4 6 × 10−4

0.5 3 × 10−4 8 × 10−4

0.3 3 × 10−4 8 × 10−4

0.3 3 × 10−4 8 × 10−4

Perturbation limit Φ Convex weight λ Dual tolerance / max steps Target / policy samples Discount factor γ Target EMA rate Batch size

0.1 0.75 10−3 / 104 10 / 100 0.99 0.005 1000 Table 29: RFQI hyperparameters.

36

PointMaze LunarLander 0.3 3 × 10−4 8 × 10−4

0.3 3 × 10−4 8 × 10−4

Parameter

Hopper Walker2d HalfCheetah

Smoothing samples 20 Policy smooth ϵ 0.005 Policy smooth weight 0.1 Q smooth ϵ 0.005 Q OOD ϵ 0.01 Q OOD weight 0.5 Uncert. weight, init / min 2.0 / 0.1 Uncertainty decay 10−6

20 0.01 1.0 0.01 0.01 0.1 1.0 / 0.1 5 × 10−7

10 0.001 0.1 0.001 0 0 0/0 0

Door

PointMaze LunarLander

20 0.01 1.0 0.01 0.01 0.1 1.0 / 0.1 5 × 10−7

20 0.005 0.1 0.005 0.01 0.5 — —

20 0.005 0.1 0.005 0.01 0.5 — —

1.0, from update 2 × 104 10 critics 10−4 / 0.2 3 × 256 3 × 10−4 0.99 0.005 256

BC anchor weight Critic ensemble Q smooth weight / τ Hidden layers Policy / critic lr Discount factor γ Target EMA rate Batch size

Table 30: RORL hyperparameters.

Parameter

Value

Parameter

Value

Adversary radius ϵ Adversarial actor loss L2 regulariser

0.025 0.1 0.02

ϵ ramp length Adversary lr EMA

1.5 × 104 updates 3 × 10−4 0.995

Table 31: ATLA-IQL hyperparameters.

Parameter

Value

Rewrite mode Mask selection threshold Mask sparsity weight

causal permutation 0.7 0.5

Parameter

Value

Rewritten fraction of batch Perturbation probability scale

0.25 0.3

Table 32: RSC-IQL hyperparameters.

Parameter

Value

Dynamics ensemble GAE parameter λ Supervised anchor weight Stackelberg coefficient

Parameter

7 networks, 5 elites 0.95 3000 0

Rollout horizon Adversary clip range Model share of critic target

Table 33: ROMB hyperparameters.

37

Value 5 0.2 0.5

Parameter

Value

Horizon h ODE sampling steps / solver Rollout window T Learning rate Batch size

Parameter

5 5 / midpoint 12 10−4 256

Value

Guidance weight RTG conditioning Backbone Conditioning dropout

2.0 on U-Net, 128 features 0.2

Table 34: FWM hyperparameters.

Parameter

Value

Parameter

Value

Flow-model ensemble PVL discount γλ PVL horizon

3 0.9405 h=5

PVL decay λ Ensemble diversity Advantage clipping

0.95 training seeds positive part

Table 35: PLR-PVL hyperparameters.

F.3

Robust safe algorithm hyperparameters Parameter

Value

G1 Actor and critic Learning rate Discount γ Entropy coefficient Drawer Actor and critic Learning rate Discount γ Entropy coefficient

Parameter

Value

256-128-128, ELU 10−3 , adaptive 0.99 0.008

Observation normalization KL target GAE λ Clip ϵ

off 0.01 0.95 0.2

256-128-64, ELU 5 × 10−4 , fixed 0.99 0.001

Observation normalization KL target GAE λ Clip ϵ

off 0.02 0.95 0.2

Table 36: PPO hyperparameters.

Parameter G1 Observation corruption Drawer Robot friction Cabinet mass Drawer damping

Value on U[0.8, 1.25] × U[0.8, 1.2] × U[0.5, 1.5]

Parameter Push, every 10–15 s Handle friction Arm Kp , Kd Action delay

Value vx , vy ∼ U[−0.5, 0.5] m/s static U[1.0, 1.25], dynamic U[1.25, 1.5] × U[0.9, 1.1] 0–2 policy steps

Table 37: DR-PPO hyperparameters.

38

Parameter

Value

Adversary network Adversary clip Perturbation

128-128, ELU 0.2 ϵ tanh(·), ϵ = 0.05

Parameter

Value

Adversary lr Adversary entropy coefficient Phases per outer iteration

10−3 0 1+1

Table 38: ATLA-PPO hyperparameters.

Parameter Synthetic next states Wang distortion η Episode cost budget Dual learning rate KL guard, mean / max

Value 5 0.75 20 0.01 0.03 / 0.3

Parameter

Value

Uncertainty ε Cost discount Initial multiplier Cost EMA β Step halvings

0.1 0.9 0.01 0.9 8

Table 39: RAMU-PPO hyperparameters.

Parameter

Value

Parameter

Auxiliary members per env Variance clip Cost GAE λ Initial multiplier

8 103 0.95 0.01

Uncertainty penalty Cost discount Episode cost budget Dual learning rate

Value 0.075 × Var(s′ ) 0.9 20 0.01

Table 40: SPiDR-PPO hyperparameters.

G

Additional results and analysis

G.1

Analysis of robust methods’ gains

Figure 4 reports two gains for each algorithmic feature family. The darker column is the base-relative gain: each method with a matched base learner is compared with that base algorithm under the same conditions, and the differences are averaged within the family. Standalone recipes without a matched reference learner, including RFQI, RORL, and RAMBO, are excluded; the number of included methods is reported as n in Table 41. The lighter column is the standard-pool gain: the full family’s mean minus the mean of all six Standard RL references. It therefore reflects both the robustness mechanism and the strength of its underlying learner. Brackets denote 95% seed-level stratified bootstrap intervals from 10,000 replicates. Each training seed contributes its mean over 20 evaluation episodes, and seeds are resampled within each (method, task) stratum, with the same draw reused across all conditions of that task. Methods and base algorithms are separate runs and are resampled independently. The two different comparisons indicate that both online families retain positive base-relative gains (+5.5 and +6.6), whereas offline gains are smaller. The generative family offers the cleanest comparison: all three methods use IQL, yielding a +9.8 gain over the standard pool but +2.5 over IQL. Its aggregate advantage thus reflects both the added mechanism and a strong base algorithm, while fixed-data coverage may limit the incremental benefit available offline.

39

Algorithm feature Learner-centric (Online) Environment-centric Learner-centric (Offline) Data-centric Generative model

Paired gain

Pooled gain

n paired

+5.5 [+2.8, +8.3] +6.6 [+3.8, +9.6] +3.8 [+0.8, +6.4] −0.1 [−2.7, +2.1] +2.5 [−0.3, +4.9]

+9.5 [+7.2, +11.8] +9.6 [+7.3, +11.8] +2.9 [+1.3, +4.4] +7.6 [+4.2, +11.1] +9.8 [+7.9, +11.7]

3 2 1 1 3

Table 41: The two columns of the algorithm feature panel, with intervals. Point estimates and 95% seed-level stratified bootstrap intervals from 10,000 replicates. Paired subtracts each member’s own base algorithm; pooled subtracts the mean of all six Standard RL references. n is the number of members that have a matched base learner and therefore enter the paired column.

G.2

Analysis of task capability: On-policy vs Off-policy

On-policy methods update a policy using trajectories collected by that policy or a recent version of it, as in PPO (Schulman et al., 2017). Off-policy methods can also learn from data generated by earlier or different policies, often through experience replay, as in SAC (Haarnoja et al., 2018). This distinction concerns which data can be used for an update. Figure 13 shows that the evaluated off-policy methods generally achieve higher scores on the MuJoCo locomotion tasks, while on-policy methods are stronger on PointMaze-UMaze and CarRace. In locomotion, frequent reward feedback allows replay-based learners to reuse informative transitions across many updates, which may contribute to the off-policy methods’ advantage. PointMaze requires discovering goal-reaching trajectories, while CarRacing couples control with visual input; updates from recent on-policy rollouts may be more useful in these settings.

G.3

Results of advanced robotic tasks

Figure 14 indicates that across the Isaac Lab tasks, SPiDR-PPO most consistently balances performance and safety: it maintains high success while satisfying the cost budget on Drawer, and leads on both return and survival under G1 action delays. These results show why high return alone is insufficient for sim-to-real evaluation. Robust-safe methods must be assessed for constraint satisfaction alongside task performance.

G.4

Supplementary results

Figures 15-16 show our complete comparison on 220 different configurations across 4 task families in Section 4.1. Figures 17 and 18 show our full shift-isolated evaluation on 44 distinct perturbation configurations. Figure 19 further shows full results on 9 perturbation configurations including compound-shifts scenario.

40

Normalized score

PPO

HalfCheetah

RARL-P

RARL-T

Hopper

ATLA

Walker2d

ATLA-SA

SAC

DR

RSC

PointMaze-Umaze

FetchReach

CarRace

125 100 75 50 25 0

On-policy

Off-policy

On-policy

Off-policy

On-policy

Off-policy

On-policy

Off-policy

On-policy

Off-policy

On-policy

Off-policy

Figure 13: On-policy and off-policy performance aggregated by task.

Performance

Locomotion · G1 100 80 60 40 20 0 20

mass ×1.5

0

25

50

75

CoM +0.15 m

100

Success rate (%)

0

25

50

75

torque 20%

100

0

25

50

75

delay 40 ms

100

0

25

50

75

delay 60 ms

100

0

25

50

75

delay 80 ms

100

0

25

50

75

100

75

100

Manipulation · Drawer Performance

robot fric.

handle fric.

cabinet mass

arm gains

damping

delay

100 75 50 25 0

0

25

50

75

100

Success rate (%)

0

25

50

75

100

0

PPO

25

DR-PPO

50

75

100

ATLA-PPO

0

25

50

RAMU-PPO

75

100

0

25

50

75

100

0

25

50

SPiDR-PPO

Figure 14: Isaac Lab results. Each panel repeats Figure 8 on one factor: safety success rate against performance, one point per algorithm, bars the s.d. over three training seeds.

41

Standard

RARL-P

RARL-T

ATLA

ATLA-SA

DR

RSC

Hopper - Gravity ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

DR

RARL-T

ATLA-SA

DR

DR

DR

DR

ATLA-SA

Hopper - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

ATLA-SA

ATLA-SA

ATLA-SA

ATLA-SA

ATLA-SA

ATLA-SA

ATLA-SA

ATLA-SA

Hopper - Actuator gear ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

RSC

RSC

RSC

DR

RARL-T

RSC

DR

RARL-T

HalfCheetah - Gravity ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

100 80 60 40 20 0

DR

DR

DR

DR

DR

DR

DR

DR

HalfCheetah - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

100 80 60 40 20 0

DR

DR

DR

DR

DR

DR

DR

DR

HalfCheetah - Actuator gear ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

100 80 60 40 20 0

DR

DR

DR

DR

DR

DR

DR

DR

Walker2d - Gravity ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

DR

ATLA-SA

DR

RARL-P

RARL-T

DR

DR

DR

Walker2d - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0 -50

ATLA

DR

DR

DR

RARL-T

RARL-T

RARL-T

DR

Walker2d - Actuator gear ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

Standard

Standard

DR

DR

RARL-P

RARL-P

DR

Figure 15: Supplementary online results of Section 4.1.

42

DR

Standard

RARL-P

RARL-T

ATLA

ATLA-SA

DR

RSC

FetchReach - Morphology ×0.8

×1.2

×1.4

×1.6

100 75 50 25 0

RARL-T

DR

DR

DR

FetchReach - Actuator gear ×0.1

×0.2

ATLA

ATLA

×0.4

×2

100 75 50 25 0

ATLA-SA

ATLA-SA

PointMaze-UMaze - Actuator gear ×0.6

×0.8

×1.2

×1.4

125 100 75 50 25 0

DR

DR

DR

DR

PointMaze-UMaze - Dry friction 10 N

20 N

30 N

40 N

ATLA

ATLA

100 75 50 25 0

DR

DR

PointMaze-UMaze - Adversarial action oppose (0.6)

oppose-goal (0.6) oppose-goal (0.8)

rotate (70)

100 75 50 25 0

ATLA

ATLA

ATLA

ATLA

CarRace - Morphology ×0.6

×0.8

×1.2

×1.4

×1.6

×1.8

100 75 50 25 0 -25

DR

DR

RARL-P

RARL-P

RARL-P

RARL-P

CarRace - Actuator gear ×0.4

×0.6

×0.8

×1.2

×1.4

×1.6

125 100 75 50 25 0

DR

DR

DR

DR

RARL-P

RARL-P

CarRace - Surface friction ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

125 100 75 50 25 0

RARL-P

RARL-P

RARL-P

RARL-P

DR

DR

Figure 15: Supplementary online results of Section 4.1.

43

Standard (IQL, MOPO, TD3BC, SynthER)

ATLA

RSC-IQL

ROMB

FWM

PLR-PVL

RAMBO

RFQI

RORL

HalfCheetah - Gravity ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

RAMBO

RAMBO

FWM

ATLA

FWM

RORL

RORL

RORL

HalfCheetah - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

80 60 40 20 0

RFQI

RFQI

RFQI

FWM

RSC-IQL

RORL

RORL

RORL

HalfCheetah - Actuator gear ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

RORL

RORL

RAMBO

RAMBO

RAMBO

RAMBO

RAMBO

RAMBO

Hopper - Gravity ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

ATLA

RORL

RORL

RORL

RFQI

RSC-IQL

RSC-IQL

RSC-IQL

Hopper - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

125 100 75 50 25 0

PLR-PVL

ROMB

RSC-IQL

RSC-IQL

FWM

RORL

RORL

RORL

Hopper - Actuator gear ×0.6

×0.7

×0.8

FWM

FWM

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0

PLR-PVL

RORL

RORL

RAMBO

FWM

RORL

Walker2d - Gravity ×0.6

×0.9

×1.1

×1.2

×1.3

×1.4

200 150 100 50 0

RORL

RORL

RORL

RORL

RORL

RORL

RORL

RORL

Walker2d - Morphology ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

150 100 50 0 -50

RFQI

RFQI

FWM

RORL

RORL

RORL

RORL

RORL

Walker2d - Actuator gear ×0.6

×0.7

×0.8

×0.9

×1.1

×1.2

×1.3

×1.4

200 150 100 50 0

RAMBO

RAMBO

RORL

RORL

RORL

RORL

RORL

Figure 16: Supplementary offline results of Section 4.1.

44

RORL

Standard (IQL, MOPO, TD3BC, SynthER)

ATLA

RSC-IQL

ROMB

FWM

PLR-PVL

RAMBO

RFQI

PointMaze - Actuator gear ×0.05

×0.1

×0.2

×0.4

80 60 40 20 0

ROMB

RSC-IQL

RSC-IQL

RSC-IQL

PointMaze - Dry friction 30 N

45 N

FWM

FWM

60 N

75 N

100 80 60 40 20 0

PLR-PVL

ATLA

PointMaze - Adversarial action oppose-goal

rotate

rotate 90°

oppose

80 60 40 20 0

RSC-IQL

FWM

ROMB

ATLA

Door - Actuator gear ×0.6

×0.8

×1.2

×1.4

150 100 50 0 -50

RSC-IQL

ATLA

ATLA

ATLA

Door - Latch 2×

3×

4×

5×

150 100 50 0 -50

ATLA

ATLA

RAMBO

RAMBO

LunarLander - Gravity ×0.75

×0.85

×0.9

×1.1

×1.15

×1.25

150 100 50 0

ATLA

ROMB

ATLA

PLR-PVL

RFQI

RFQI

LunarLander - Engine ×0.6

×0.75

×0.9

×1.1

×1.25

FWM

FWM

×1.4

150 100 50 0

Standard

RFQI

RFQI

ROMB

LunarLander - Wind 5

10

15

20

25

30

150 100 50 0

RAMBO

RAMBO

RFQI

FWM

RAMBO

FWM

Figure 16: Supplementary offline results of Section 4.1.

45

RORL

RARL-P

Observation

obs rel 0.10

RARL-T

obs uni 0.10

ATLA

ATLA-SA

DR

obs gauss 0.10

RSC

mad 0.02

mad 0.05

150 100 50 0

DR

ATLA-SA

act uni 0.10

ATLA-SA

DR

DR

cmd oppose 0.20

act gauss 0.10

act offset 0.05

act oppose 0.10

DR

DR

DR

DR

gear static 0.8

gear ramp 0.8

gear sine p500

100 50

Dynamics

0

Execution latency

push 60

trans t1

100 50 0

Reward shift

ATLA-SA

150

RSC

DR

RSC

RSC

ATLA-SA

obs delay 4ms

obs delay 8ms

obs delay 16ms

act delay 8ms

act delay rand1

DR

DR

DR

DR

DR

150 100 50 0

gauss

uniform

shift

150 100 50 0

DR

DR

ATLA-SA

Credit latency

Action

150

k=4

k=16

k=32

100 50 0

DR

ATLA-SA

RARL-T

Figure 17: Supplementary online results of Section 4.2.

46

k=64

150

ATLA-SA

ATLA

RSC-IQL

Observation

obs rel 0.10

ROMB

obs uni 0.10

FWM

PLR-PVL

RAMBO

obs gauss 0.10

RFQI

RORL

mad 0.02

mad 0.05

100 50 0

PLR-PVL

RSC-IQL

act uni 0.10

ATLA

act gauss 0.10

RFQI

act offset 0.05

RFQI

act oppose 0.10

cmd oppose 0.20

Action

100 75 50 25 0

PLR-PVL

RFQI

gear static 0.8

RORL

gear ramp 0.8

RORL

gear sine p500

FWM

push 60

trans t1

60 40 20

Reward shift

Execution latency

0

FWM

PLR-PVL

obs delay 4ms

ATLA

obs delay 8ms

ATLA

obs delay 16ms

RORL

act delay 8ms

act delay rand1

100 75 50 25 0

RORL

gauss

RORL

uniform

shift

150 100 50 0

ATLA

RORL

FWM

RORL

Credit latency

Dynamics

80

k=4

RORL

k=16

k=32

k=64

150 100 50 0 ATLA

PLR-PVL

Figure 18: Supplementary offline results of Section 4.2.

47

RORL

RFQI

PLR-PVL

Nominal

Action

Dynamic

Latency

Compound

Normalized score

Hopper - Online PPO

RARL-P

RARL-T

ATLA

ATLA-SA

SAC

Latency

Compound

DR

RSC

ROMB

FWM

150 100 50 0 -50

Nominal

Action

Dynamic

TD3BC

SynthER

MOPO

ATLA

RSC-IQL

PLR-PVL

RAMBO

RFQI

RORL

Observation

Semantic

150 100 50 0

Normalized score

-50

150 100 50 0 -50

Nominal

Latency

Dynamic

Compound

Pusher Normalized score

Normalized score

Hopper - Offline IQL

PPO

ATLA-SA

RARL-P

SAC

DR

125 100 75 50 25 0

Figure 19: Supplementary results of Section 4.2.

48

Related documents

• RobustRLlib: A Unified Library and Benchmark for Robust Reinforcement Learning Algorithms#1448486
• Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization#1064211
• REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement#1035078
• Reinforcement Learning via Advantage Guidance#1354468
• Safe and Robust Neural Policy Learning with Statistical Verification for Sim-to-Real Deployment in Robotics#1187513