A generic platform for addressing the multimodal challenge

Abstract

Multimodal interactive systems support multiple interaction techniques such as the synergistic use of speech and direct manipulation. The flexibility they offer results in an increased complexity that current software tools do not address appropriately. One of the emerging technical problems in multimodal interaction is concerned with the fusion of information produced through distinct interaction techniques. In this article, we present a generic fusion engine that can be embedded in a multi-agent architecture modelling technique. We demonstrate the fruitful symbiosis of our fusion mechanism with PAC-Amodeus, our agentbased conceptual model, and illustrate the applicability of the approach with the implementation of an effective interactive system: MATIS, a Multimodal Airline Travel Information System.

Test of the ICARE Platform Fusion Mechanism

“…Two types of elementary components are defined: "device components" (DC) and "interaction language components" (ILC). A modality is the coupling of a device and an interaction language [13]. For example, in Fig.…”

“…Thus, the fusion algorithm must be rigorously and carefully developed and validated, assuring that the multimodal action performed by the user is actually accomplished by the application. So several frameworks is dedicated to the multimodal interactions, such as [6] and [13].…”

Mobile Collaborative Augmented Reality: The Augmented Stroll

“…Our software design solution for implementing the augmented stroll draws upon our software architecture model PAC-Amodeus [15]. PAC-Amodeus uses the Arch model [24] as the foundation for the functional partitioning of an interactive system and populates the key element of this organization, i.e., the Dialogue Controller, with PAC agents [7].…”

“…PAC-Amodeus uses the Arch model [24] as the foundation for the functional partitioning of an interactive system and populates the key element of this organization, i.e., the Dialogue Controller, with PAC agents [7]. Although the software architecture model is not new, we show here how the model, dedicated to the software design of multimodal systems [15], can be applied to the design of an AR system. In particular we highlight the software design of the gateway between the real and the virtual.…”

Software Design and Development of Mutimodal Interaction

“…speech and gesture inputs), information is typically integrated at the syntactic or semantic levels (Nigay et al, 1995). In this type of architecture, current approaches for modality integration include framebased methods, multimodal grammars and agent-based frameworks.…”