Combining deictic gestures and natural language for referent identification

Abstract

In virtually all current natural-language dialog systems, users can only refer to objects by using linguistic descriptions. However, in human face-to-face conversation, participants fre= quently use various sorts of deictic gestures as well. In this paper, we will present the referent identification component of XTRA, a system for a natural-language access to expert systems. XTRA allows the user to combine NL input together with pointing gestures on the terminal screen in order Io refer to objects on the display. Information about the location and type of this deietic gesture, as well as about the linguistic description of the referred object, the case frame, and the dialog memory are utilized for identifying the object. The system is tolerant in respect to impreciseness of both the deictic and the natural language input. The user can thereby refer to objects more easily, avoid referential failures, and employ vague everyday terms instead of precise technical notions.

Plan-based integration of natural language and graphics generation

“…[36,62]). WIP grew out of the results of our previous research into multimodal interaction, particularly in the VITRA and XTRA projects.…”

“…Over the past several years, a number of projects have entered the area between natural language processing and multimodal communication, often focusing on a single specific functionality, such as the use of pointing gestures parallel to verbal descriptions for referent identification (e.g., [13,36,43]). The automatic design of complete multimodal presentations has only recently received significant attention in artificial intelligence research.…”

Robot deictics

“…Deictic gestures are often used to augment or replace verbal descriptions of the object being referred to, also called the referent [16]. The importance of deictic gestures in communication is shown in pre-verbal children, who will use deictic gestures as a way of communicating with their caretakers prior to their ability to form utterances to describe their wants and needs [6].…”

Multimodal definite clause grammar

“…Recent work in the design of direct manipulation interfaces has also explored the notion of integrating a set of diverse inputs. Other papers exploring multimodal interfaces include [ 1,6,7,11,161. Most of this work, however, has focused on the application of the ideas, and not on the principles for integrating the different inputs.…”