Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML

Abstract

This article introduces the problem of modeling multimodal interaction, in the form of markup languages. After an analysis of the current state of the art in multimodal interaction description languages, nine guidelines for languages dedicated at multimodal interaction description are introduced, as well as four different roles that such language should target: communication, configuration, teaching and modeling. The article further presents the SMUIML language, our proposed solution to improve the time synchronicity aspect while still fulfilling other guidelines. SMUIML is finally mapped to these guidelines as a way to evaluate their spectrum and to sketch future works.

A domain-specific textual language for rapid prototyping of multimodal interactive systems

“…The visual models of HephaisTK [8] include symbols to group simultaneous and successive events. However, it can only group fixed-length sequences of events.…”

“…Some of them provide domain-specific visual languages [3,7,6,17,8] that enable modeling a wide variety of composite events. However, these specifications easily degenerate into complex diagrams for relatively simple commands such as the put-that-there [5].…”

A graphical editor for the SMUIML multimodal user interaction description language

“…In this article we therefore present the graphical SMUIML editor as the last member of the triad of tools providing multimodal interaction practitioners with a complete set of tools for the creation of multimodal interfaces. While we provide a general overview of SMUIML and HephaisTK, a detailed description of these two components can be found in previously published work [25,23,27] 3 .…”

“…The spectrum of multimodal dialogue description language uses, on a scale from highly expressive to highly usable, was presented in [25]. Through various workshops, numerous informal discussions and a study of the current state of the art, we envisioned the three description language approaches shown in Figure 3.…”

“…Along these three approaches, a formal language can also be used as a learning tool helping teachers in communicating the features of a particular application domain to their students. We have presented nine guidelines for a multimodal description language [25], which should be used as design tools or language analysis criteria:…”