Safe- to-Explore State Spaces: Ensuring Safe Exploration in Policy Search with Hierarchical Task Optimization

Abstract

Policy search reinforcement learning allows robots to acquire skills by themselves. However, the learning procedure is inherently unsafe as the robot has no a-priori way to predict the consequences of the exploratory actions it takes. Therefore, exploration can lead to collisions with the potential to harm the robot and/or the environment. In this work we address the safety aspect by constraining the exploration to happen in safeto-explore state spaces. These are formed by decomposing target skills (e.g., grasping) into higher ranked sub-tasks (e.g., collision avoidance, joint limit avoidance) and lower ranked movement tasks (e.g., reaching). Sub-tasks are defined as concurrent controllers (policies) in different operational spaces together with associated Jacobians representing their joint-space mapping. Safety is ensured by only learning policies corresponding to lower ranked sub-tasks in the redundant null space of higher ranked ones. As a side benefit, learning in sub-manifolds of the state-space also facilitates sample efficiency. Reaching skills performed in simulation and grasping skills performed on a real robot validate the usefulness of the proposed approach.

Null Space Based Efficient Reinforcement Learning with Hierarchical Safety Constraints

“…The prior work constrains the agent to explore in Safe-To-Explore-State-Spaces (STESS) [4], which decomposes a robotic skill into prioritized elemental tasks and a normalized Radial Basis Function (RBF) [4] network is used to represent the learning policy. We continue with STESS framework in this paper and further construct several phases for different period constraints.…”

“…We take advantage of STESS [4] to enable safe exploration of lower ranked RL task in the null space of higher ranked tasks.All tasks are solved in the acceleration space and the objective function can be formulated as…”

“…Reinforcement learning (RL) involves performing a number of exploratory actions, often with a degree of randomness, which can lead to damage of the robot or its environment. This problem has been previously addressed by learning in simulation [1], [2], [3], safety exploration [4], [5], imitation learning [6], [7] and learning from demonstration (LfD) [8], [9]. However, some of these solutions do not guarantee safety, while others face difficulties when transferring from simulated to real environments.…”

“…We leverage a hierarchical framework that can guarantee constraint satisfaction in the least-square sense during the robot's trial-and-error exploration, where constraints are used to describe safety conditions. We base our work on Safe-To-Explore State Space (STESS) [4], which restricts the robotic operational space to a collision-free space. This is accomplished by decomposing a robotic skill (e. g., putting a book into a cabinet) into different prioritized tasks.…”

Teleoperation of Collaborative Robot for Remote Dementia Care in Home Environments

“…The motion planning and control of the robot are completed in MoveIt, an open-source project in ROS. In this case, we modified an open-source unified robot description format (URDF) model of YuMi provided by Lundell et al [35]. Because YuMi's manipulators and grippers are controlled independently based on different IP addresses, we divided the whole robot into four motion planning groups after URDF remodeling: left arm, right arm, left hand and right hand.…”