Relational reinforcement learning with guided demonstrations

Abstract

Model-based reinforcement learning is a powerful paradigm for learning tasks in robotics. However, in-depth exploration is usually required and the actions have to be known in advance. Thus, we propose a novel algorithm that integrates the option of requesting teacher demonstrations to learn new domains with fewer action executions and no previous knowledge. Demonstrations allow new actions to be learned and they greatly reduce the amount of exploration required, but they are only requested when they are expected to yield a significant improvement because the teacher's time is considered to be more valuable than the robot's time. Moreover, selecting the appropriate action to demonstrate is not an easy task, and thus some guidance is provided to the teacher. The rule-based model is analyzed to determine the parts of the state that may be incomplete, and to provide the teacher with a set of possible problems for which a demonstration is needed. Rule analysis is also used to find better alternative models and to complete subgoals before requesting help, thereby minimizing the number of requested demonstrations. These improvements were demonstrated in a set of experiments, which included domains from the international planning competition and a robotic task. Adding teacher demonstrations and rule analysis reduced the amount of exploration required by up to 60% in some domains, and improved the success ratio by 35% in other domains.

Teaching a Robot the Semantics of Assembly Tasks

“…In contrast, we use the REX-D algorithm [61], which combines relational RL and active demonstration requests. REX-D requests demonstrations only when they can save a lot of time, because teacher's time is considered to be very valuable, and uses autonomous exploration otherwise.…”

“…We introduced the REX-D algorithm [61] to address the learning phase, which is an efficient model-based reinforcement learning (RL) method combined with additional human demonstrations upon request. It can take three alternative strategies: one is to explore the state space to improve the model and achieve better rewards in the long term; another is to exploit the available knowledge by executing the manipulations that maximize the reward with the current learned model [76]; and the last one is to request a demonstration from the teacher [60].…”

Robot motion adaptation through user intervention and reinforcement learning

“…Generally, in PS approaches, the robot is taught an initial trajectory that is then improved through autonomously-generated rollouts and policy updates [7]. A model-based RL proved to be efficient for learning action sequences with the user in the role of a teacher [19]. Instead of applying RL to teach the robot new actions and their effects, in this work, RL is applied to modify the robot trajectory segment selected by the user.…”

Cleaning Tasks Knowledge Transfer Between Heterogeneous Robots: a Deep Learning Approach

“…MDP is also used to clear objects from a table in fully-observable problems with uncertainty [27]. The same authors employ REX-D algorithm that integrates active teacher demonstration for increasing learning speed in order to sweep lentils from a plane [28]. Interactive RL approach with contextual affordances is developed by Cruz et al to clean a table using state-action-reward-state-action (SARSA) [7].…”