Integrating Language, Vision and Action for Human Robot Dialog Systems

Rickert, Markus; Foster, Mary Ellen; Giuliani, Manuel; By, Tomas; Panin, Giorgio; Knoll, Alois

doi:10.1007/978-3-540-73281-5_108

Cited by 33 publications

(29 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The robot torso (Figure 1) being built as part of the project [10,11] consists of a pair of mechanical arms with grippers and an animatronic talking head. The input channels consist of speech recognition, object recognition, gesture recognition, and robot sensors; the outputs include synthesized speech, emotional expressions, head motions, and robot actions.…”

Section: The Jast Robotmentioning

confidence: 99%

Attention driven visual processing for an interactive dialog robot

Müller

Knoll

2009

Proceedings of the 2009 ACM Symposium on Applied Computing

Self Cite

View full text Add to dashboard Cite

In this paper we propose an attention-based vision system for the JAST interactive dialog robot. The robotic vision system incorporates three submodules: object recognition, gesture recognition and self recognition.The performance boost of our biologically inspired vision system is based on two assumptions: first, generally attention is attracted by regions of high intensity or hue gardients as well as scene dynamics (bottom-up attention attraction), and second, attentioninal focus can be directed by higher level modules, whether volitional or not, in an inhibitory or reinforcing way (top-down attention control). The system proposed in this paper is able to utilize these assumptions and organize its computational efforts accordingly.Integrated into an efficient data management architecture, the vision system is capable of continuously publishing results to the cognitive layer of the robot and thus enables operations in realtime. Furthermore, the modular system structure and the asynchronous communication paradigm allows for efficient integration of additional modules, be it visual or any other sensory input data.The main contribution of this work is the application of neuroscience findings and biologically plausible theories of attention based visual processing to a real-world robotic setup. Here, our experimental results show tremendous speedups using either the bottom-up attention attractors or the principle of top-down attention control as input data filters for further visual analysis, reaching the peak in a combination of the two.

show abstract

Section: The Jast Robotmentioning

confidence: 99%

Attention driven visual processing for an interactive dialog robot

Müller

Knoll

2009

Proceedings of the 2009 ACM Symposium on Applied Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…Studies on the social interaction of human-computer interfaces have included conversations with robots [11][12][13] and virtual agents [14][15][16][17]. An important cue to recognize social interaction is nonverbal information.…”

Section: Introductionmentioning

confidence: 99%

“…An important cue to recognize social interaction is nonverbal information. These studies employ some multimodal information such as hand gestures, head nods, face direction, and gaze direction as well as spoken language to build teamwork in the collaboration with a robot [11,12] or to create a chance to address the user [13]. Meanwhile, Maatman et al [14] and Kopp et al [15] studied the natural behavior of the agent while the user is speaking.…”

Section: Introductionmentioning

confidence: 99%

Estimating a User's Internal State before the First Input Utterance

Chiba

Ito

2012

Advances in Human-Computer Interaction

View full text Add to dashboard Cite

This paper describes a method for estimating the internal state of a user of a spoken dialog system before his/her first input utterance. When actually using a dialog-based system, the user is often perplexed by the prompt. A typical system provides more detailed information to a user who is taking time to make an input utterance, but such assistance is nuisance if the user is merely considering how to answer the prompt. To respond appropriately, the spoken dialog system should be able to consider the user's internal state before the user's input. Conventional studies on user modeling have focused on the linguistic information of the utterance for estimating the user's internal state, but this approach cannot estimate the user's state until the end of the user's first utterance. Therefore, we focused on the user's nonverbal output such as fillers, silence, or head-moving until the beginning of the input utterance. The experimental data was collected on a Wizard of Oz basis, and the labels were decided by five evaluators. Finally, we conducted a discrimination experiment with the trained user model using combined features. As a three-class discrimination result, we obtained about 85% accuracy in an open test.

show abstract

“…Schrempf et al (2005) proposed a method to synchronize robot and human actions using a Dynamic Bayesian Network. Rickert et al (2007) presented a collaborative robot that is equipped with speech recognition and visual object recognition, and is able to follow the operator hands. This robot uses these informations to anticipate on the next task.…”

Section: Human-robot Collaboration In Manufacturingmentioning

confidence: 99%

Multi-users online recognition of technical gestures for natural human–robot collaboration in manufacturing

2018

View full text Add to dashboard Cite

Human-Robot Collaboration in industrial context requires a smooth, natural and efficient coordination between robot and human operators. The approach we propose to achieve this goal is to use online recognition of technical gestures. In this paper, we present together, and analyze, parameterize and evaluate much more thoroughly, three findings previously unveiled separately by us in several conference presentations: 1/ we show on a real prototype that multi-users continuous real-time recognition of technical gestures on an assembly-line is feasible (≈ 90% recall and precision in our case-study), using only non-intrusive sensors (depth-camera with a top-view, plus inertial sensors placed on tools); 2/ we formulate an end-to-end methodology for designing and developing such a system; 3/ we propose a method for adapting to new users our gesture recognition. Furthermore we present here two new findings: 1/ by comparing recognition performances using several sets of features, we highlight the importance of choosing features that focus on the effective part of gestures, i.e. usually hands movements; 2/ we obtain new results suggesting that enriching a multiusers training set can lead to higher precision than using a separate training dataset for each operator.

show abstract

Integrating Language, Vision and Action for Human Robot Dialog Systems

Cited by 33 publications

References 13 publications

Attention driven visual processing for an interactive dialog robot

Attention driven visual processing for an interactive dialog robot

Estimating a User's Internal State before the First Input Utterance

Multi-users online recognition of technical gestures for natural human–robot collaboration in manufacturing

Contact Info

Product

Resources

About