Using vision, acoustics, and natural language for disambiguation

Abstract

Creating a human-robot interface is a daunting experience. Capabilities and functionalities of the interface are dependent on the robustness of many different sensor and input modalities. For example, object recognition poses problems for state-of-the-art vision systems. Speech recognition in noisy environments remains problematic for acoustic systems. Natural language understanding and dialog are often limited to specific domains and baffled by ambiguous or novel utterances. Plans based on domain-specific tasks limit the applicability of dialog managers. The types of sensors used limit spatial knowledge and understanding, and constrain cognitive issues, such as perspective-taking.In this research, we are integrating several modalities, such as vision, audition, and natural language understanding to leverage the existing strengths of each modality and overcome individual weaknesses. We are using visual, acoustic, and linguistic inputs in various combinations to solve such problems as the disambiguation of referents (objects in the environment), localization of human speakers, and determination of the source of utterances and appropriateness of responses when humans and robots interact. For this research, we limit our consideration to the interaction of two humans and one robot in a retrieval scenario. This paper will describe the system and integration of the various modules prior to future testing.

Sensor Network for Structuring People and Environmental Information

“…Moreover, for robotic use, much accuracy is required for approaching a person or for interacting movements such as making gestures or keeping eye contacts. Finding people by on-board sensors installed on robots are another field that is actively studied (Gockley et al, 2007;Fransen et al, 2007). However, this is effective only after the robots have approached close enough to the person in target.…”

Crossmodal content binding in information-processing architectures

“…However, we would argue that it would be a grave mistake to discard earlier work on symbol grounding. If, for example, the systems of [12,16,3,7,4], could be utilised in one and the same system, we would truly be able to take a step forward as a community. Therefore non-intrusiveness is an important requirement on our binding system [10].…”

“…When an object is placed in front of the robot, the visual subarchitecture processes the object as described previously, ultimately creating a visual proxy for it. When the tutor makes an assertion about the object (or relation), we use recency information to bind the communication proxy for the deictic reference to the newest visual proxy 4 . The communication proxy contains binding features for all of the adjectives used in the utterance.…”

Structuring information on people and environment for supporting robotic services

“…Finding people by on-board sensors installed on robots are another field that is actively studied [13,14]. However, this is effective only after the robots have approached close enough to the person in target.…”