Using dialog and human observations to dictate tasks to a learning robot assistant

Rybski, Paul E.; Stolarz, Jeremy; Yoon, Kevin; Veloso, Manuela

doi:10.1007/s11370-008-0016-5

Cited by 30 publications

(11 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The notion of spatial description clauses is influenced by many similar formalisms for reasoning about the semantics of natural language directions [7], [8], [9], [10], [11], [12]. The structure of the spatial description clause builds on the work of Landau and Jackendoff [13], and Talmy [14], providing a computational instantiation of their formalisms.…”

Section: Related Workmentioning

confidence: 99%

Natural language command of an autonomous micro-air vehicle

Huang

Tellex

Bachrach

et al. 2010

2010 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

Abstract-Natural language is a flexible and intuitive modality for conveying directions and commands to a robot but presents a number of computational challenges. Diverse words and phrases must be mapped into structures that the robot can understand, and elements in those structures must be grounded in an uncertain environment.In this paper we present a micro-air vehicle (MAV) capable of following natural language directions through a previously mapped and labeled environment. We extend our previous work in understanding 2D natural language directions to three dimensions, accommodating new verb modifiers such as go up and go down, and commands such as turn around and face the windows. We demonstrate the robot following directions created by a human for another human, and interactively executing commands in the context of surveillance and search and rescue in confined spaces. In an informal study, 71% of the paths computed from directions given by one user terminated within 10 m of the desired destination.

show abstract

Section: Related Workmentioning

confidence: 99%

Natural language command of an autonomous micro-air vehicle

Huang

Tellex

Bachrach

et al. 2010

2010 IEEE/RSJ International Conference on Intelligent Robots and Systems

View full text Add to dashboard Cite

show abstract

“…Forbes et al [9] implemented a set of robust parametrized motion procedures on a PR2 mobile manipulator and converted the NL sentences into intended procedures and their parameter instantiations. Rybski et al [10] designed a task representation called Directed Acyclic Graph (DAG), which includes a preconditioned list of the behavior, the behavior and the link to the next behavior. The outputted formal representations are action plans in fact, and can be implemented by the robot sequentially.…”

Section: Related Workmentioning

confidence: 99%

Detecting Target Objects by Natural Language Instructions Using an RGB-D Camera

Bao

Jia

Cheng

et al. 2016

Sensors

View full text Add to dashboard Cite

Controlling robots by natural language (NL) is increasingly attracting attention for its versatility, convenience and no need of extensive training for users. Grounding is a crucial challenge of this problem to enable robots to understand NL instructions from humans. This paper mainly explores the object grounding problem and concretely studies how to detect target objects by the NL instructions using an RGB-D camera in robotic manipulation applications. In particular, a simple yet robust vision algorithm is applied to segment objects of interest. With the metric information of all segmented objects, the object attributes and relations between objects are further extracted. The NL instructions that incorporate multiple cues for object specifications are parsed into domain-specific annotations. The annotations from NL and extracted information from the RGB-D camera are matched in a computational state estimation framework to search all possible object grounding states. The final grounding is accomplished by selecting the states which have the maximum probabilities. An RGB-D scene dataset associated with different groups of NL instructions based on different cognition levels of the robot are collected. Quantitative evaluations on the dataset illustrate the advantages of the proposed method. The experiments of NL controlled object manipulation and NL-based task programming using a mobile manipulator show its effectiveness and practicability in robotic applications.

show abstract

“…It has been demonstrated that learning can be used to infer meanings from spoken utterances [12,13]. These scenarios are usually restricted to a constrained set of vocabulary or using manual annotation to improve the learning algorithm.…”

Section: Passive Sensing Vs Active Teachingmentioning

confidence: 99%

How to train your robot - teaching service robots to reproduce human social behavior

Liu¹,

Glas²,

Kanda³

et al. 2014

The 23rd IEEE International Symposium on Robot and Human Interactive Communication

View full text Add to dashboard Cite

Developing interactive behaviors for social robots presents a number of challenges. It is difficult to interpret the meaning of the details of people's behavior, particularly non-verbal behavior like body positioning, but yet a social robot needs to be contingent to such subtle behaviors. It needs to generate utterances and non-verbal behavior with good timing and coordination. The rules for such behavior are often based on implicit knowledge and thus difficult for a designer to describe or program explicitly. We propose to teach such behaviors to a robot with a learning-by-demonstration approach, using recorded human-human interaction data to identify both the behaviors the robot should perform and the social cues it should respond to. In this study, we present a fully unsupervised approach that uses abstraction and clustering to identify behavior elements and joint interaction states, which are used in a variable-length Markov model predictor to generate socially-appropriate behavior commands for a robot. The proposed technique provides encouraging results despite high amounts of sensor noise, especially in speech recognition. We demonstrate our system with a robot in a shopping scenario.

show abstract

Using dialog and human observations to dictate tasks to a learning robot assistant

Cited by 30 publications

References 22 publications

Natural language command of an autonomous micro-air vehicle

Natural language command of an autonomous micro-air vehicle

Detecting Target Objects by Natural Language Instructions Using an RGB-D Camera

How to train your robot - teaching service robots to reproduce human social behavior

Contact Info

Product

Resources

About