Robust Reinforcement Learning: A Review of Foundations and Recent Advances

Abstract

Reinforcement learning (RL) has become a highly successful framework for learning in Markov decision processes (MDP). Due to the adoption of RL in realistic and complex environments, solution robustness becomes an increasingly important aspect of RL deployment. Nevertheless, current RL algorithms struggle with robustness to uncertainty, disturbances, or structural changes in the environment. We survey the literature on robust approaches to reinforcement learning and categorize these methods in four different ways: (i) Transition robust designs account for uncertainties in the system dynamics by manipulating the transition probabilities between states; (ii) Disturbance robust designs leverage external forces to model uncertainty in the system behavior; (iii) Action robust designs redirect transitions of the system by corrupting an agent’s output; (iv) Observation robust designs exploit or distort the perceived system state of the policy. Each of these robust designs alters a different aspect of the MDP. Additionally, we address the connection of robustness to the risk-based and entropy-regularized RL formulations. The resulting survey covers all fundamental concepts underlying the approaches to robust reinforcement learning and their recent advances.

On the Robustness of Safe Reinforcement Learning under Observational Perturbations

“…Robust RL. The robustness definition in the RL context has many interpretations [31], including the robustness against action perturbations [32], reward corruptions [33,34], domain shift [35,36], and dynamics uncertainty [37][38][39][40][41][42][43]. The most related works are investigating the observational robustness of an RL agent under state adversarial attacks [17,19].…”

Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability

“…There is a surge of interest in researching effective attacks and defenses in RL. Detailed reviews of both fields can be found in [28,76,121].…”

“…It has rich meanings that go beyond its literal sense, and motivates a comprehensive framework that includes multiple principles, requirements, and criteria [3]. Recently, there has been exciting progress in the area of trustworthy RL [2,5,48,107,108,121,129,137,140,145,148,165,171,201], which greatly help to advance our understanding of intrinsic vulnerabilities in RL and potential solutions in particular aspects of trustworthy RL. It is clear that the next leap toward trustworthy RL will require a holistic and fundamental understanding of the challenges of these problems, the weakness, and advantages of existing trustworthy RL approaches, and a paradigm shift of trustworthy RL based on existing work.…”

“…Despite these interconnecting aspects of trustworthiness, existing surveys mainly focus on limited parts. García et al [52], Gu et al [62], and Brunke et al [25] work on safe RL, while Moos et al [121] and Kirk et al [88] focus on robustness and generalization, respectively. They advance the field by providing concrete descriptions for one aspect of trustworthy RL, yet lack a comprehensive characterization of the intrinsic vulnerabilities of RL as we envisioned above.…”

Intelligent air defense task assignment based on hierarchical reinforcement learning

“…Reinforcement learning was first introduced in the 1950s ( Minsky, 1954 ) with the central idea of allowing an agent to learn in its environment and continuously refine its behavioral strategies through constant interaction with the environment and exploration by trial and error ( Moos et al, 2022 ). With the continuous development of RL, algorithms such as Q-learning ( Watkins and Dayan, 1992 ) and SARSA ( Chen et al, 2008 ) have been proposed.…”