Understanding and executing instructions for everyday manipulation tasks from the World Wide Web

Abstract

Service robots will have to accomplish more and more complex, open-ended tasks and regularly acquire new skills. In this work, we propose a new approach to generating plans for such household robots. Instead composing them from atomic actions, we propose to transform task descriptions on web sites like ehow.com into executable robot plans. We present methods for automatically converting the instructions given in natural language into a formal, logic-based representation, for resolving the word senses using the WordNet database and the Cyc ontology, and for exporting the generated plans into the mobile robot's plan language RPL. We discuss the problems of inferring information missing in these descriptions, of grounding the abstract task descriptions in the perception and action system, and we propose techniques for solving them. The whole system works autonomously without human interaction. It has successfully been tested with a set of about 150 natural language directives, of which up to 80% could be correctly transformed.

Acquiring knowledge about human goals from Search Query Logs

“…In applications, goal knowledge can benefit various domains including text analysis (Kröll et al, 2010), human computer interaction (Smith & Lieberman, 2010) or planning (Tenorth et al, 2010). These applications would benefit from additional knowledge about human goals, e.g.…”

Priming transformational planning with observations of human activities

“…The whole process starts with the generation of a default plan (lower left corner). This initial plan may be manually created or automatically generated from web sites as demonstrated in [2] and [3]. These default plans specify the main plan steps to achieve a goal, but usually fail to perform the task efficiently.…”

“…• a system for generating plans from natural-language task descriptions on web sites like ehow.com [2], • methods for projecting and debugging these plans to infer missing information [3], and • TRANER, a planning system, can modify high-level plans using a set of transformations [1]. With these modules, robots can autonomously create plans for new tasks, fix plan flaws, and apply transformations to optimize the performance of a plan.…”

From Structured Task Instructions to Robot Task Plans

“…One way to meet this requirement is to provide robots with the ability to understand and follow the task instructions that are defined by human users (Burgard et al, 1999;Rybski et al, 2007;Tenorth et al, 2010;Cantrell et al, 2012). A robot can find and extract knowledge of these instructions from human-robot dialogue, as well as web sources such as open knowledge bases like Cyc 1 and Open Mind Indoor Common Sense (OMICS) 2 .…”

“…Procedural knowledge specifies a plan skeleton of how to accomplish a task. Given procedural knowledge, the robot needs to find out a possible sequence of actions which meets the restrictions of the plan skeleton to accomplish the task (Burgard et al, 1999;Tenorth et al, 2010). Normally, both knowledge are arbitrarily introduced in user-defined task instructions.…”