Multimodal Interaction for Information Access: Exploiting Cohesion

Abstract

Multimodality is a powerful concept for dealing with dialogue cohesion in a human-computer natural language (NL)-centered system. This work is a modest step toward more effective exploitation of the potentially large bandwidth of communication provided by this situation. The relations between exploration, navigation, and NLbased communication are discussed in general and with reference to two prototypes. Light cognitive load feedback and direct manipulation are proposed so that user and system can cooperate in mutually establishing the structure of the ongoing dialogue. The main points are: (i) use of an appropriate dialogue structure to constrain inference in the anaphora resolution process; (ii) use of a graphical representation of the structure, to limit the problem of opacity; (iii) allowance for the possibility of direct manipulation on this representation, to avoid the necessity of operating linguistically at the metalevel. The context of the work is within NL-centered multimodal information access systems, in which basic entities are pairs (most commonly question and answer). A dialogue model is provided by a modified version of the centering model; it is both sufficiently simple to be displayed in an intuitive fashion on the screen, and sufficiently powerful to give accurate results. An extension of the discourse model, oriented to the treatment of deixis, is also proposed. Finally, steps toward an overall approach to the integration of navigational and mediated aspects of interaction are discussed.

Cognitive Principles in Robust Multimodal Interpretation

“…Considerable effort has been devoted to studying user multimodal behavior (Cohen, 1984;Oviatt, 1999a) and mechanisms to interpret user multimodal inputs (Chai et al, 2004b;Gustafson et al, 2000;Huls, Bos, & Classen, 1995;Johnston, Cohen, McGee, Oviatt, Pittman, & Smith, 1997;Johnston, 1998;Johnston & Bangalore, 2000;Kehler, 2000;Koons et al, 1993;Neal & Shapiro, 1991;Oviatt, DeAngeli, & Kuhn, 1997;Stent et al, 1999;Stock, 1993;Wahlster, 1998;Wu & Oviatt, 1999;Zancanaro, Stock, & Strapparava, 1997).…”

“…It then checks semantic constraints such as the type of the candidate objects being referenced and their properties for reference resolution. A modified centering model for multimodal reference resolution is also introduced in previous work (Zancanaro et al, 1997). The idea is that based on the centering movement between turns, segments of discourse can be constructed.…”

Intelligent Multimedia Communication

“…The generation system, part of the output presentation system, is influenced by the user's interest model that develops in the course of the multimodal interaction. Another aspect developed in this system is cross-modal feedback (Zancanaro et al [13]). The user is provided fast graphical feedback of the discourse references interpretation by the system, exploiting profitably the large bandwidth of communication we have in a multimodal system.…”

Multimodal Information Exploration

“…PREPARATORY exploits the mechanism of focus spaces, described in Zancanaro [12]. This mechanism at the same time takes into account the changes in the environment, and the preceding interactions.…”