Grounded language interpretation of robotic commands through structured learning

Vanzo, Andrea; Croce, Danilo; Bastianelli, Emanuele; Basili, Roberto; Nardi, Daniele

doi:10.1016/j.artint.2019.103181

Cited by 22 publications

(8 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Grounding referents probabilistically can be based on sensor data and human language (Vanzo, Croce, Bastianelli, Basili, & Nardi, 2019;Walter, Hemachandra, Homberg, Tellex, & Teller, 2013) in addition to static information (Mohan, Mininger, & Laird, 2013). Generalized grounding graphs facilitate both understanding and the generation of language requests regarding the shared environment (Tellex, Knepper, Li, Rus, & Roy, 2014).…”

Section: Instructing Robots Using Natural Languagementioning

confidence: 99%

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Thomason

Padmakumar²,

Sinapov³

et al. 2020

jair

109

View full text Add to dashboard Cite

In this work, we present methods for using human-robot dialog to improve language understanding for a mobile robot agent. The agent parses natural language to underlying semantic meanings and uses robotic sensors to create multi-modal models of perceptual concepts like red and heavy. The agent can be used for showing navigation routes, delivering objects to people, and relocating objects from one location to another. We use dialog clari_cation questions both to understand commands and to generate additional parsing training data. The agent employs opportunistic active learning to select questions about how words relate to objects, improving its understanding of perceptual concepts. We evaluated this agent on Amazon Mechanical Turk. After training on data induced from conversations, the agent reduced the number of dialog questions it asked while receiving higher usability ratings. Additionally, we demonstrated the agent on a robotic platform, where it learned new perceptual concepts on the y while completing a real-world task.

show abstract

Section: Instructing Robots Using Natural Languagementioning

confidence: 99%

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Thomason

Padmakumar²,

Sinapov³

et al. 2020

jair

109

View full text Add to dashboard Cite

show abstract

“…Ontologies and semantic rules have been jointly used to formally describe a domain of interest and infer new knowledge [10][11][12][13]. Normally, knowledge representation and reasoning approaches can be classified into three categories: distributional semantic representation, model-theoretic semantic representation and frame semantic representation [14].…”

Section: Literature Reviewmentioning

confidence: 99%

A Rule-Based Heuristic Methodology for Su-Field Analysis in Industrial Engineering Design

et al. 2022

View full text Add to dashboard Cite

Industrial engineering design is a crucial issue in manufacturing. To meet the competitive global market, manufacturers are continuously seeking solutions to design industrial products and systems inventively. Su-Field analysis, which is one of the TRIZ analysis tools for inventive design problems, has been used to effectively improve the performance of industrial systems. However, the inventive standards used for engineering design are summarized and classified according to a large number of patents in different fields. They are built on a highly abstract basis and are independent of specific application fields, making their use require much more technical knowledge than other TRIZ tools. To facilitate the use of invention standards, in particular to capture the uncertainty or imprecision described in the standards, this paper proposes a rule-based heuristic approach. First, Su-Field analysis ontology and fuzzy analysis ontology are constructed to represent precise and fuzzy knowledge in the process of solving inventive problems respectively. Then, SWRL (Semantic Web Rule Language) reasoning and fuzzy reasoning are executed to generate heuristic conceptual solutions. Finally, we develop a software prototype and elaborate the resolution of “Auguste Piccard’s Stratostat ” in the prototype.

show abstract

“…With respect to robotics, the understanding and acquisition of language can take advantage of the situational nature of a robot, as it is placed in a dedicated environment where tasks and context are known ( Taniguchi et al, 2019 ). Research works have focused on specific contexts for extractions and interpretations of robot instructions, such as manipulation ( Misra et al, 2016 ), grasping ( Chen et al, 2021 ), intention recognition ( Mi et al, 2020 ; Sun et al, 2021 ) and grounding ( Misra et al, 2016 ; Shridhar et al, 2020 ; Vanzo et al, 2020 ). Other approaches interpret natural language through human-robot dialog ( Thomason et al, 2015 ), or utilize additional sensor modalities, such as vision ( Sun et al, 2021 ; Chen et al, 2021 ).…”

Section: Related Workmentioning

confidence: 99%

Coordinating Shared Tasks in Human-Robot Collaboration by Commands

et al. 2021

View full text Add to dashboard Cite

Human-robot collaboration is gaining more and more interest in industrial settings, as collaborative robots are considered safe and robot actions can be programmed easily by, for example, physical interaction. Despite this, robot programming mostly focuses on automated robot motions and interactive tasks or coordination between human and robot still requires additional developments. For example, the selection of which tasks or actions a robot should do next might not be known beforehand or might change at the last moment. Within a human-robot collaborative setting, the coordination of complex shared tasks, is therefore more suited to a human, where a robot would act upon requested commands.In this work we explore the utilization of commands to coordinate a shared task between a human and a robot, in a shared work space. Based on a known set of higher-level actions (e.g., pick-and-placement, hand-over, kitting) and the commands that trigger them, both a speech-based and graphical command-based interface are developed to investigate its use. While speech-based interaction might be more intuitive for coordination, in industrial settings background sounds and noise might hinder its capabilities. The graphical command-based interface circumvents this, while still demonstrating the capabilities of coordination. The developed architecture follows a knowledge-based approach, where the actions available to the robot are checked at runtime whether they suit the task and the current state of the world. Experimental results on industrially relevant assembly, kitting and hand-over tasks in a laboratory setting demonstrate that graphical command-based and speech-based coordination with high-level commands is effective for collaboration between a human and a robot.

show abstract

Grounded language interpretation of robotic commands through structured learning

Abstract: For guidance on citations see FAQs.

Cited by 22 publications

References 27 publications

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

Jointly Improving Parsing and Perception for Natural Language Commands through Human-Robot Dialog

A Rule-Based Heuristic Methodology for Su-Field Analysis in Industrial Engineering Design

Coordinating Shared Tasks in Human-Robot Collaboration by Commands

Contact Info

Product

Resources

About