Safe Exploration of State and Action Spaces in Reinforcement Learning

Abstract

In this paper, we consider the important problem of safe exploration in reinforcement learning. While reinforcement learning is well-suited to domains with complex transition dynamics and high-dimensional state-action spaces, an additional challenge is posed by the need for safe and efficient exploration. Traditional exploration techniques are not particularly useful for solving dangerous tasks, where the trial and error process may lead to the selection of actions whose execution in some states may result in damage to the learning system (or any other system). Consequently, when an agent begins an interaction with a dangerous and high-dimensional state-action space, an important question arises; namely, that of how to avoid (or at least minimize) damage caused by the exploration of the state-action space. We introduce the PI-SRL algorithm which safely improves suboptimal albeit robust behaviors for continuous state and action control tasks and which efficiently learns from the experience gained from the environment. We evaluate the proposed method in four complex tasks: automatic car parking, pole-balancing, helicopter hovering, and business management

Safe- to-Explore State Spaces: Ensuring Safe Exploration in Policy Search with Hierarchical Task Optimization

“…These methods are typically combined with low-dimensional policy representations that are initialized on prior data, e. g., human demonstrations or trajectories optimized using an initial model [4], [5], [3]. Limiting policy updates between iterations lowers the probability of straying into unexplored state space, but does not provide any guarantees [22]. The associated risk is also evidenced by the results reported in Section III: if the search space includes obstacles, collisions can and, in general, will occur between the robot and obstacles.…”

Probabilistic Policy Reuse for Safe Reinforcement Learning

“…While most Reinforcement Learning (RL) tasks [28] are focused on maximizing a long-term cumulative reward, RL researchers are also paying increasing attention to the safety of the approaches (e.g., avoiding visits to undesirable situations, collisions, crashes, etc.) during the training process [9,10]. Thus, when using RL techniques in dangerous control tasks, an important question arises; namely, how can we ensure that the exploration of the state-action space will not cause damage or injury while, at the same time, learning (near-)optimal policies?…”

“…Instead, evolutionary approaches [16,20,21] are usually based on evolving populations of neural networks (each one representing a different policy) with the aim of finding a better policy in each generation. However, the random generation of the initial population and the random mutation of the neural networks during evolution are made to visit undesirable states repeatedly [9]. Therefore, most of these approaches suffer from the same problem: the absence of mechanisms for avoiding the visit to undesirable situations during the exploration of the state and action spaces.…”

“…One of these algorithms is the Policy Improvement through Safe Reinforcement Learning (PI-SRL) algorithm [9]. PI-SRL is an algorithm for safe exploration in dangerous and continuous control tasks.…”

“…Certainly, not knowing what to do in a given situation increases the probability of ending up in trouble. Thus, the concept of risk is associated to the concept of unknown [9]. In case of an unknown state, the risk is maximum, and the agent has to ask the teacher for advice.…”