A General Framework to Increase Safety of Learning Algorithms for Dynamical Systems Based on Region of Attraction Estimation

Abstract

Although the state-of-the-art learning approaches exhibit impressive results for dynamical systems, only a few applications on real physical systems have been presented. One major impediment is that the intermediate policy during the training procedure may result in behaviors that are not only harmful to the system itself but also to the environment. In essence, imposing safety guarantees for learning algorithms is vital for autonomous systems acting in the real world. In this article, we propose a computationally effective and general safe learning framework, specifically for complex dynamical systems. With a proper definition of the safe region, a supervisory control strategy, which switches the actions applied on the system between the learning-based controller and a predefined corrective controller, is given. A simplified system facilitates the estimation of the safe region for the high-dimensional dynamical system. During the learning phase, the belief of the safe region is updated with the actual execution results of the corrective controller, which in turn enables the learning-based controller to have more freedom in choosing its actions. Two examples are given to demonstrate the performance of the proposed framework, one simple inverted pendulum to illustrate the online adaptation method, and one quadcopter control task to show the overall performance. Index Terms-Deep learning in robotics and automation, learning and adaptive systems, robot safety, safe reinforcement learning.

Learning a Low-Dimensional Representation of a Safe Region for Safe Reinforcement Learning on Dynamical Systems

“…In particular, it is challenging to compute a safe region for a complex dynamical system. For this reason, [26] introduces an SRL framework that utilizes a supervisory control strategy based on finding a simplified system by means of physically inspired model order reduction [27]. A simplified safe region is constructed from the simplified system, which functions as an approximation for the safe region of the full dynamics.…”

“…In this paper, we consider the same supervisory control strategy as used in [26] to construct a general SRL framework that is applicable to complex dynamical systems. However, to overcome the limitations of physically inspired model order reduction, we propose a novel data-driven approach to identify the supervisor, i.e., the low-dimensional representation of the safe region.…”

“…In this article, we consider SRL in order to optimize a learning-based policy with respect to a predefined reward function, while ensuring that the system state remains in a safe region of the state space. In this section, we outline a general SRL framework for dynamical systems, see also [26]. The SRL framework first identifies a safe state-space region as the safe region.…”

“…The aforementioned SRL framework is not directly applicable to complex dynamical systems, as in such cases, calculating the safe region S is computationally infeasible [31]. An SRL framework based on estimating safety with a low-dimensional representation of the safe region is introduced to overcome this problem [26].…”

“…where t safe denotes the first time point at which the probability p(x ∈ S) is not larger than the threshold p t , i.e., p(x ∈ S) = (y) ≤ p t . More details of this SRL framework are given in [26].…”