A Review of Safe Reinforcement Learning: Methods, Theory and Applications

Gu, Shangding; Yang, Lihong; Du, Yuchen; Chen, Guang; Walter, Florian; Wang, Jun; Yang, Yanfei; Knoll, Alois

doi:10.48550/arxiv.2205.10330

Cited by 32 publications

(42 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Safety Constraints: finding an optimal policy that satisfies external user-specified safe constraints (Chow et al, 2018a;Gu et al, 2022).…”

Section: A Additional Related Workmentioning

confidence: 99%

Quasi-optimal Learning with Continuous Treatments

Li¹,

Zhou²,

Zhu³

2023

Preprint

View full text Add to dashboard Cite

Many real-world applications of reinforcement learning (RL) require making decisions in continuous action environments. In particular, determining the optimal dose level plays a vital role in developing medical treatment regimes. One challenge in adapting existing RL algorithms to medical applications, however, is that the popular infinite support stochastic policies, e.g., Gaussian policy, may assign riskily high dosages and harm patients seriously. Hence, it is important to induce a policy class whose support only contains near-optimal actions, and shrink the action-searching area for effectiveness and reliability. To achieve this, we develop a novel quasi-optimal learning algorithm, which can be easily optimized in off-policy settings with guaranteed convergence under general function approximations. Theoretically, we analyze the consistency, sample complexity, adaptability, and convergence of the proposed algorithm. We evaluate our algorithm with comprehensive simulated experiments and a dose suggestion real application to Ohio Type 1 diabetes dataset.

show abstract

“…• Safety Constraints: finding an optimal policy that satisfies external user-specified safe constraints (Chow et al, 2018a;Gu et al, 2022).…”

Section: A Additional Related Workmentioning

confidence: 99%

Quasi-optimal Learning with Continuous Treatments

Li¹,

Zhou²,

Zhu³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…In the setting above, the agent is allowed to explore the entire state and action space without any constraints, where one issue is [37]: how can we guarantee safety when we apply RL for real-world applications? In CRL, however, the space is explored under safe constraints, where the goal of policy optimization is to maximize the rewards whilst satisfying the certain safe constraints [10].…”

Section: B Reinforcement Learning From Distributional and Constrained...mentioning

confidence: 99%

Trustworthy Reinforcement Learning for Quadrotor UAV Tracking Control Systems

Wang¹,

Boyle²

2023

Preprint

View full text Add to dashboard Cite

Simultaneously accurate and reliable tracking control for quadrotors in complex dynamic environments is challenging. As aerodynamics derived from drag forces and moment variations are chaotic and difficult to precisely identify, most current quadrotor tracking systems treat them as simple 'disturbances' in conventional control approaches. We propose a novel, interpretable trajectory tracker integrating a Distributional Reinforcement Learning disturbance estimator for unknown aerodynamic effects with a Stochastic Model Predictive Controller (SMPC). The proposed estimator 'Constrained Distributional Reinforced disturbance estimator' (ConsDRED) accurately identifies uncertainties between true and estimated values of aerodynamic effects. Simplified Affine Disturbance Feedback is used for control parameterization to guarantee convexity, which we then integrate with a SMPC. We theoretically guarantee that ConsDRED achieves at least an Θ(1/ √ T ) global convergence rate and an Θ(1/ √ T ) sublinear rate if constraints are violated with an error Θ(1/m H 4 ) decreases as the width and the layer of neural network, i.e., m and H, increase. To demonstrate practicality, we show convergent training in simulation and real-world experiments, and empirically verify that ConsDRED is less sensitive to hyperparameter settings compared with canonical constrained RL approaches. We demonstrate our system improves accumulative tracking errors by at least 62% compared with the recent art. Importantly, the proposed framework, ConsDRED-SMPC, balances the tradeoff between pursuing high performance and obeying conservative constraints for practical implementations 1 .

show abstract

“…Safe RL. Safe RL has garnered significant attention in recent years, as researchers aim to address safety concerns associated with deploying RL agents in safety-critical domains (Garcıa and Fernández 2015;Gu et al 2022;Baheri et al 2020;Baheri 2022). A prevalent approach to safe RL involves formulating the problem as a constrained optimization task, where the primary objective is to maximize the expected return while satisfying given safety constraints (Achiam et al 2017).…”

Section: Related Workmentioning

confidence: 99%

Policy Learning with Constraints in Model-free Reinforcement Learning: A Survey

Liu

Halev

Liu

2021

Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Reinforcement Learning (RL) algorithms have had tremendous success in simulated domains. These algorithms, however, often cannot be directly applied to physical systems, especially in cases where there are constraints to satisfy (e.g. to ensure safety or limit resource consumption). In standard RL, the agent is incentivized to explore any policy with the sole goal of maximizing reward; in the real world, however, ensuring satisfaction of certain constraints in the process is also necessary and essential. In this article, we overview existing approaches addressing constraints in model-free reinforcement learning. We model the problem of learning with constraints as a Constrained Markov Decision Process and consider two main types of constraints: cumulative and instantaneous. We summarize existing approaches and discuss their pros and cons. To evaluate policy performance under constraints, we introduce a set of standard benchmarks and metrics. We also summarize limitations of current methods and present open questions for future research.

show abstract

A Review of Safe Reinforcement Learning: Methods, Theory and Applications

Cited by 32 publications

References 0 publications

Quasi-optimal Learning with Continuous Treatments

Quasi-optimal Learning with Continuous Treatments

Trustworthy Reinforcement Learning for Quadrotor UAV Tracking Control Systems

Policy Learning with Constraints in Model-free Reinforcement Learning: A Survey

Contact Info

Product

Resources

About