Benchmarking fusion engines of multimodal interactive systems

Abstract

This article proposes an evaluation framework to benchmark the performance of multimodal fusion engines. The paper first introduces different concepts and techniques associated with multimodal fusion engines and further surveys recent implementations. It then discusses the importance of evaluation as a mean to assess fusion engines, not only from the user perspective, but also at a performance level. The article further proposes a benchmark and a formalism to build testbeds for assessing multimodal fusion engines. In its last section, our current fusion engine and the associated system HephaisTK are evaluated thanks to the evaluation framework proposed in this article. The article concludes with a discussion on the proposed quantitative evaluation, suggestions to build useful testbeds, and proposes some future improvements.

A Proposal for Processing and Fusioning Multiple Information Sources in Multimodal Dialog Systems

“…The general architecture used for the development of multimodal applications can be separated in four different components: input modalities and their recognizers, output modalities and their respective synthesizers, the integration committee, and the application logic [8]. Indeed, using multimodality efficiently implies a clear abstraction between the results of the user's input analysis, the processing of this input, answer generation and output modalities selection.…”

“…According to [8], fusion of input sources in these systems must be approached in a global way: from the point of view of the architecture of a multimodal system as a whole, then, from the point of view of multimodal dialog modeling, and finally from an algorithmic point of view.…”

Assessing the support provided by a toolkit for rapid prototyping of multimodal systems

“…The concise model of the right side of Figure 3, depicted with the editor of HephaisTK [6], specifies the behavior of a prototype implementing the put-that-there interaction technique. The prototype will initially be at state Start awaiting for the sequence of events whose detection will cause the movement of an object.…”

“…This movement is executed by the subroutine put that there action whereas the stream of events that will cause the execution of this subprogram is represented as a set of nested rectangles. HephaisTK's notation allow four different types of rectangles to declare composite events [6]. Each type of rectangle indicates the temporal constraints among the events annotated within it.…”

“…In the remainder of this work, these toolkits will be referred to as toolkits for rapid prototyping of multimodal systems. Some existing toolkits for rapid prototyping of multimodal systems are ICon [5], Squidy [13], CoGenIVE [4], HephaisTK [6] and PetShop [9]. They are rather different one from another, since they provide different features, target different domains, use different programming paradigms and/or expect different skills from their users.…”