Safe Model-Based Reinforcement Learning for Systems With Parametric Uncertainties

Abstract

Reinforcement learning has been established over the past decade as an effective tool to find optimal control policies for dynamical systems, with recent focus on approaches that guarantee safety during the learning and/or execution phases. In general, safety guarantees are critical in reinforcement learning when the system is safety-critical and/or task restarts are not practically feasible. In optimal control theory, safety requirements are often expressed in terms of state and/or control constraints. In recent years, reinforcement learning approaches that rely on persistent excitation have been combined with a barrier transformation to learn the optimal control policies under state constraints. To soften the excitation requirements, model-based reinforcement learning methods that rely on exact model knowledge have also been integrated with the barrier transformation framework. The objective of this paper is to develop safe reinforcement learning method for deterministic nonlinear systems, with parametric uncertainties in the model, to learn approximate constrained optimal policies without relying on stringent excitation conditions. To that end, a model-based reinforcement learning technique that utilizes a novel filtered concurrent learning method, along with a barrier transformation, is developed in this paper to realize simultaneous learning of unknown model parameters and approximate optimal state-constrained control policies for safety-critical systems.

Safe adaptive output‐feedback optimal control of a class of linear systems

“…In recent years, significant progress has been made in developing safe model-based reinforcement learning (SMBRL) techniques to learn safe controllers for different classes of systems. [6][7][8][9][10][11][12][13][14][15][16] While Markov decision process (MDP) based SMBRL methods have been available for discrete time systems with finite state and action spaces, [6][7][8][9] synthesizing online controllers for systems in continuous time, under output feedback, while guaranteeing stability and safety is still a challenging problem.…”

“…While the control barrier function results in safety guarantees, the existence of a smooth value function, in spite of a nonsmooth cost function, needs to be assumed. 16 This article is inspired by the nonlinear coordinate transformation first introduced in Reference 18. Leveraging the results of Reference 18, a barrier transformation (BT) to construct an equivalent, unconstrained optimal control problem from a state-constrained optimal control problem was introduced in Reference 14.…”

“…Applications such as manned aviation demand deterministic safety guarantees, and as such, online, real‐time learning in such systems is challenging. SMBRL techniques for CT deterministic systems have been studied in results such as References 13–16. In Reference 13, a SMBRL method has been developed to synthesize a real‐time safe controller online by incorporating the proximity penalty method developed in Reference 17 with the framework of control barrier functions.…”

“…In Reference 13, a SMBRL method has been developed to synthesize a real‐time safe controller online by incorporating the proximity penalty method developed in Reference 17 with the framework of control barrier functions. While the control barrier function results in safety guarantees, the existence of a smooth value function, in spite of a nonsmooth cost function, needs to be assumed 16 …”

What, how, and when? A hybrid system approach to multi-region search and rescue

“…Introducing theoretical guarantees on reinforcement learning methods is an active area of research. 10 The problem of defining a search termination criterion like we seek in the multi-region scenarios appears novel in the problem space.…”