An augmented reality system for teaching sequential tasks to a household robot

Abstract

Abstract-We present a method of instructing a sequential task to a household robot using a hand-held augmented reality device. The user decomposes a high-level goal such as "prepare a drink" into steps such as delivering a mug under a kettle and pouring hot water into the mug. The user takes a photograph of each step using the device and annotates it with necessary information via touch operation. The resulting sequence of annotated photographs serves as a reference for review and reuse at a later time. We created a working prototype system with various types of robots and appliances.

Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces

“…Sometimes systems combine user evaluations techniques with demonstration [111,382] or technical evaluations [15,406]. In observational studies [58,369,382], researchers can also get user feedback through observations [135,448]. Finally, some systems also ran lab studies through expert interviews [10,116,340] to get specific feedback from the expert's perspectives.…”

On the Integration of Adaptive and Interactive Robotic Smart Spaces

“…For instance, [67] describes a household robot able to learn how to collaborate with humans in cooking tasks, while [68] and [70] show how a robot can decompose a demonstrated task into sequential manipulation primitives in order to learn how to manipulate everyday objects. Another interesting approach, that could be combined with the ones above to enable user-driven definition or personalization of services in a cognitive robotic ecology, is the one presented in [69], where a hand-held augmented reality device is used to instruct a sequential task to a household robot. Human-Aware Planning: Supporting cooperative and personalised system-user interaction will require a substantial improvement in the capabilities of planbased CRE solutions.…”

A knowledge-driven shared autonomy human-robot interface for tablet computers

“…[11] [21]. However, it would get more difficult for an operator to pick a desired action from the list represented by such an input element if the set of possible actions gets large due to enhanced capabilities of the robot and complex world states.…”

“…Their HRI restricts the movement of a surveillance robot based on the relationship between the robot and a chosen object allowing simplified inspection. Fung et al [21] teach sequential tasks to a robot by augmenting photographies of task related objects with icons for possible actions. The selected actions are manually parametrized by the operator and afterwards checked for semantic validity.…”