Recovering from failure by asking for help

Abstract

Robots inevitably fail, often without the ability to recover autonomously. We demonstrate an approach for enabling a robot to recover from failures by communicating its need for specific help to a human partner using natural language. Our approach automatically detects failures, then generates targeted spoken-language requests for help such as "Please give me the white table leg that is on the black table." Once the human partner has repaired the failure condition, the system resumes full autonomy. We present a novel inverse semantics algorithm for generating effective help requests. In contrast to forward semantic models that interpret natural language in terms of robot actions and perception, our inverse semantics algorithm generates requests by emulating the human's ability to interpret a request using the Generalized Grounding Graph (G 3 ) framework. To assess the effectiveness of our approach, we present a corpusbased online evaluation, as well as an end-to-end user study, demonstrating that our approach increases the effectiveness of human interventions compared to static requests for help.Ross A. Knepper and Stefanie Tellex have contributed equally to this paper. This is one of several papers published in Autonomous Robots comprising the "Special Issue on Robotics Science and Systems".

To Err Is Robot: How Humans Assess and Act toward an Erroneous Social Robot

“…To enable a robot to generate help requests in case of an error situation, Knepper et al (2015) developed their inverse semantics algorithm. It allows the robot to phrase precise requests that specify the kind of help that is needed.…”

A review of methodologies for natural-language-facilitated human–robot cooperation

“…Robot gaps, which are constraints such as a robot's physical structure strength, capable actions and operation precision [97]; (3). user gaps, which are missing information caused by abstract, ambiguous, or incomplete human NL instructions [98] [99]. Filling up these knowledge gaps enhances robot capability in adapting dynamic environments and various tasks or users.…”

“…The first step of gap filling is gap detection. Gap detection methods mainly include the following: (1) hierarchical knowledge structure checking, which detects knowledge gaps by checking real-world-available knowledge from top-level goals to low-level NLC execution parameters defined in a hierarchical knowledge structure [38][97], (2) knowledgeapplicability assessment, which detects knowledge gaps by checking the similarities between theoretical scenarios and real-world scenarios [48] [97], and (3) performancetriggered knowledge gap estimation, which detects knowledge gaps by considering the final execution performances [99] [100]. Hierarchical knowledge structure checking has the rationale that if desired knowledge defined in a knowledge structure is missing in real-world situations, then knowledge gaps exist.…”

Generating machine-executable plans from end-user's natural-language instructions

“…Human-machine collaborative manufacturing combines human intelligence on high-level task planning and the robot physical capability (e.g., precision and speed) on low-level task execution [1]. Toward this direction, intuitive and natural communication between the human and the machine has been an active research area in the last decade with the goal to enable seamless human-machine cooperation [2] [3]. Natural-Language-instructed human-machine interaction is expected to enable an advanced manufacturing machine, such as a Computer Numerical Control machine or an industrial robot, to autonomously perform tasks such as rough/fine finishing [4] [5], assembly [2] [6] and packaging [7] [8] according to the end-user's NL instructions, which are given based on the user's judgement of the task progress and environmental situations.…”

“…Toward this direction, intuitive and natural communication between the human and the machine has been an active research area in the last decade with the goal to enable seamless human-machine cooperation [2] [3]. Natural-Language-instructed human-machine interaction is expected to enable an advanced manufacturing machine, such as a Computer Numerical Control machine or an industrial robot, to autonomously perform tasks such as rough/fine finishing [4] [5], assembly [2] [6] and packaging [7] [8] according to the end-user's NL instructions, which are given based on the user's judgement of the task progress and environmental situations. Compared with other input methods, including human hand force [9][10], hand gesture [11] [12], and body motions [13] [14][15] [16], the NL instruction method has two main advantages.…”