Exploring implicit human responses to robot mistakes in a learning from demonstration task

Abstract

As robots enter human environments, they will be expected to accomplish a tremendous range of tasks. It is not feasible for robot designers to pre-program these behaviors or know them in advance, so one way to address this is through end-user programming, such as learning from demonstration (LfD). While significant work has been done on the mechanics of enabling robot learning from human teachers, one unexplored aspect is enabling mutual feedback between both the human teacher and robot during the learning process, i.e., implicit learning. In this paper, we explore one aspect of this mutual understanding, grounding sequences, where both a human and robot provide non-verbal feedback to signify their mutual understanding during interaction. We conducted a study where people taught an autonomous humanoid robot a dance, and performed gesture analysis to measure people's responses to the robot during correct and incorrect demonstrations.

Understanding and Resolving Failures in Human-Robot Interaction: Literature Review and Model Development

“…Almost all errors researched in the literature exemplified technical failures (e.g., Gieselmann, 2006 ; Kim and Hinds, 2006 ; Gieselmann and Ostendorf, 2007 ; Spexard et al, 2008 ; Kim et al, 2009 ; Groom et al, 2010 ; Lee et al, 2010 ; Takayama et al, 2011 ; Desai et al, 2012 , 2013 ; Kahn et al, 2012 ; Rosenthal et al, 2012 ; Shiomi et al, 2013 ; Yasuda and Matsumoto, 2013 ; Kaniarasu and Steinfeld, 2014 ; Lohan et al, 2014 ; Cha et al, 2015 ; Gehle et al, 2015 ; Giuliani et al, 2015 ; Gompei and Umemuro, 2015 ; Hamacher, 2015 ; Knepper et al, 2015 ; Mirnig et al, 2015 , 2017 ; Mubin and Bartneck, 2015 ; Salem et al, 2015 ; Bajones et al, 2016 ; Brooks et al, 2016 ; Hamacher et al, 2016 ; Hayes et al, 2016 ; Ragni et al, 2016 ; Robinette et al, 2016 ; Engelhardt and Hansson, 2017 ; Law et al, 2017 ; Sarkar et al, 2017 ; van der Woerdt and Haselager, 2017 ; Kwon et al, 2018 ). Only a few evaluated the impact of social norm violations (e.g., Short et al, 2010 ; Salem et al, 2013 ; Giuliani et al, 2015 ; Mirnig et al, 2015 , 2017 ; van der Woerdt and Haselager, 2017 ), and none focused on human errors.…”

“…A robot's failure symptoms in the literature include the robot not completing a given task (e.g., Takayama et al, 2011 ; Rosenthal et al, 2012 ; Brooks et al, 2016 ; Robinette et al, 2016 ; Mirnig et al, 2017 ; Kwon et al, 2018 ), running into obstacles (e.g., Brooks et al, 2016 ), performing the wrong action (e.g., Kim et al, 2009 ; Lee et al, 2010 ; Desai et al, 2012 , 2013 ; Yasuda and Matsumoto, 2013 ; Kaniarasu and Steinfeld, 2014 ; Gehle et al, 2015 ; Mubin and Bartneck, 2015 ; Salem et al, 2015 ; Brooks et al, 2016 ; Hayes et al, 2016 ; Robinette et al, 2016 ; Mirnig et al, 2017 ; Sarkar et al, 2017 ; van der Woerdt and Haselager, 2017 ), performing the right action incorrectly or incompletely (e.g., Takayama et al, 2011 ; Shiomi et al, 2013 ; Cha et al, 2015 ; Hamacher, 2015 ; Brooks et al, 2016 ; Hamacher et al, 2016 ; Adubor et al, 2017 ; Sarkar et al, 2017 ; van der Woerdt and Haselager, 2017 ; Kwon et al, 2018 ), producing no action or speech (irresponsiveness) (e.g., Gieselmann, 2006 ; Lohan et al, 2014 ; Bajones et al, 2016 ; Robinette et al, 2016 ; Lucas et al, 2017 , 2018 ), timing speech improperly (e.g., Mirnig et al, 2017 ), failing to produce speech (e.g., Gieselmann and Ostendorf, 2007 ; Mirnig et al, 2017 ), producing inappropriate speech or erroneous instruction (e.g., Gieselmann, 2006 ; Gieselmann and Ostendorf, 2007 ; Short et al, 2010 ; Gehle et al, 2015 ; Gompei and Umemuro, 2015 ; Lucas et al, 2017 , 2018 ; Mirnig et al, 2017 ; Sarkar et al, 2017 ), repeating s...…”

“…Error recovery strategies and reactions to errors have been evaluated using surveys (e.g., Lee et al, 2010 ; Takayama et al, 2011 ; Cha et al, 2015 ; Brooks et al, 2016 ; Adubor et al, 2017 ; Kim et al, 2017 ; Rossi et al, 2017b ; van der Woerdt and Haselager, 2017 ; Kwon et al, 2018 ), video analysis of HRIs (e.g., Giuliani et al, 2015 ; Mirnig et al, 2015 ), and unstructured observational studies (e.g., Gieselmann, 2006 ; Gehle et al, 2015 ), however most studies used controlled user experiments (e.g., Spexard et al, 2008 ; Short et al, 2010 ; Desai et al, 2013 ; Salem et al, 2013 , 2015 ; Gompei and Umemuro, 2015 ; Knepper et al, 2015 ; Hayes et al, 2016 ; Ragni et al, 2016 ; Robinette et al, 2016 ; Mirnig et al, 2017 ; Lucas et al, 2018 ). One study introduced an idea on how to improve situation awareness (SA; see Comprehension and Memory section) in erroneous situations without any formal evaluation (Cassenti, 2007 ).…”

To Err Is Robot: How Humans Assess and Act toward an Erroneous Social Robot

“…A deviation from the social script resulted in a different strategy in the human gaze coordination (social signals). Hayes et al (2016) performed a user study in which participants were instructed to teach a dance to a robot. They explored how humans implicitly responded when the robot made a mistake.…”

Expressing Robot Incapability

“…Transparency about the causes of incapability also helps observers assign blame more accurately [16]. If observers also understand what the robot was trying to do, they are be er able to help the robot complete the task [14,22,31].…”