Interactive Learning with Corrective Feedback for Policies Based on Deep Neural Networks

Abstract

Deep Reinforcement Learning (DRL) has become a powerful strategy to solve complex decision making problems based on Deep Neural Networks (DNNs). However, it is highly data demanding, so unfeasible in physical systems for most applications. In this work, we approach an alternative Interactive Machine Learning (IML) strategy for training DNN policies based on human corrective feedback, with a method called Deep COACH (D-COACH). This approach not only takes advantage of the knowledge and insights of human teachers as well as the power of DNNs, but also has no need of a reward function (which sometimes implies the need of external perception for computing rewards). We combine Deep Learning with the COrrective Advice Communicated by Humans (COACH) framework, in which non-expert humans shape policies by correcting the agent's actions during execution. The D-COACH framework has the potential to solve complex problems without much data or time required. Experimental results validated the efficiency of the framework in three different problems (two simulated, one with a real robot), with state spaces of low and high dimensions, showing the capacity to successfully learn policies for continuous action spaces like in the Car Racing and Cart-Pole problems faster than with DRL.

Learning to Interactively Learn and Assist

“…The traditional means of passing task information to an agent include specifying a reward function (Barto and Sutton 1998) that can be hand-crafted for the task (Singh, Lewis, and Barto 2009;Levine et al 2016;Chebotar et al 2017) and providing demonstrations (Schaal 1999;Abbeel and Ng 2004) before the agent starts training. More recent works explore the concept of the human supervision being provided throughout training by either providing rewards during training (Isbell et al 2001;Thomaz et al 2005;Warnell et al 2018;Perez-Dattari et al 2018) or demonstrations during training; either continuously (Ross, Gordon, and Bagnell 2011b;Kelly et al 2018) or at the agent's discretion (Ross, Gordon, and Bagnell 2011a;Borsa et al 2017;Xu et al 2018;Hester et al 2018;James, Bloesch, and Davison 2018;Yu et al 2018a;Krening 2018;Brown, Cui, and Niekum 2018). In all of these cases, however, the reward and demonstrations are the sole means of interaction.…”

Learning to Interactively Learn and Assist

“…The traditional means of passing task information to an agent include specifying a reward function [4,6] that can be hand-crafted for the task [46,30,9] and providing demonstrations [44,1] before the agent starts training. More recent works explore the concept of the human supervision being provided throughout training by either providing rewards during training [51,47,23,39] or demonstrations during training; either continuously [25,42] or at the agent's discretion [53,20,24,55,7,27,41,8]. In all of these cases, however, the reward and demonstrations are the sole means of interaction.…”

Corrective Feedback in Learning Interaction: Integration of Surface Strategy Taxonomy

“…Khaled Karim and Hossein (2019) explained that there are four categories in corrective feedback analysis: clarification request, Recast, Elicitation, and Metalinguistic Feedback. Through corrective feedback, students realize where their mistakes are and deepen their understanding of the knowledge gained through learning experiences so that learning difficulties can be overcome and ultimately, the quality of learning outcomes will be better (Celemin, & Ruiz-del-Solar, 2019;Pérez-Dattari et al, 2018;Chen et al, 2018). Corrective feedback is a lecturer's response to student learning errors.…”