Context-Aware Safe Reinforcement Learning for Non-Stationary Environments

Abstract

Safety is a critical concern when deploying reinforcement learning agents for realistic tasks. Recently, safe reinforcement learning algorithms have been developed to optimize the agent's performance while avoiding violations of safety constraints. However, few studies have addressed the non-stationary disturbances in the environments, which may cause catastrophic outcomes. In this paper, we propose the context-aware safe reinforcement learning (CASRL) method, a meta-learning framework to realize safe adaptation in nonstationary environments. We use a probabilistic latent variable model to achieve fast inference of the posterior environment transition distribution given the context data. Safety constraints are then evaluated with uncertainty-aware trajectory sampling. The high cost of safety violations leads to the rareness of unsafe records in the dataset. We address this issue by enabling prioritized sampling during model training and formulating prior safety constraints with domain knowledge during constrained planning. The algorithm is evaluated in realistic safety-critical environments with non-stationary disturbances. Results show that the proposed algorithm significantly outperforms existing baselines in terms of safety and robustness.

On the Robustness of Safe Reinforcement Learning under Observational Perturbations

“…Safe RL. One type of approaches utilize domain knowledge of the target problem to improve the safety of an RL agent, such as designing a safety filter [21], assuming a sophistic system dynamics model [20,22,23], or incorporating expert interventions [24,25]. Constrained Markov Decision Process (CMDP) is another commonly used framework to model the safe RL problem, which can be solved via many constrained optimization techniques [4].…”

Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability

“…SAVED [181] and RCE [109] both use an ensemble of neural networks as the dynamics model to estimate the dynamics prediction uncertainty and solve the constrained optimization problem via a model-predictive-control fashion, where the former formulates chance constraints to ensure safety from a probabilistic perspective, while the latter one considers the worst case unsafe scenario. CASRL [27] further extends previous approaches from stationary environments to non-stationary environments by modeling the non-stationary disturbances as probabilistic latent variables.…”

Robustifying Reinforcement Learning Policies with $\mathcal{L}_1$ Adaptive Control

“…Learning to adapt. Meta-RL has recently been proposed to achieve fast adaptation of a pre-trained policy in the presence of dynamic variations [18]- [23]. Despite impressive performance mainly in terms of fast adaptation demonstrated by these methods, the intermediate policies learned during the adaptation phase will most likely still fail.…”