Mudra

Hoste, Lode; Dumas, Bruno; Signer, Beat

doi:10.1145/2070481.2070500

Cited by 44 publications

(16 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first defines the input fact pattern to be recognised while the second the UI behaviour. Mudra [22], a follow-up research from the same group, extends Midas for multimodal interfaces. It unifies the input stream coming from different devices, exploiting different modalities.…”

Section: Gesture Description Modelsmentioning

confidence: 99%

DEICTIC: A compositional and declarative gesture description based on hidden markov models

Carcangiu

Spano

Fumera

et al. 2019

International Journal of Human-Computer Studies

View full text Add to dashboard Cite

The consumer-level devices that track the user's gestures eased the design and the implementation of interactive applications relying on body movements as input. Gesture recognition based on computer vision and machine-learning focus mainly on accuracy and robustness. The resulting classifiers label precisely gestures after their performance, but they do not provide intermediate information during the execution. Human-Computer Interaction research focused instead on providing an easy and effective guidance for performing and discovering interactive gestures. The compositional approaches developed for solving such problem provide information on both the whole gesture and on its sub-parts, but they exploit heuristic techniques that have a low recognition accuracy. In this paper, we introduce DEICTIC, a compositional and declarative description for stroke gestures, which uses basic Hidden Markov Models (HMMs) to recognise meaningful predefined primitives (gesture sub-parts) and it composes them to recognise complex gestures. It provides information for supporting gesture guidance and it reaches an accuracy comparable with state-of-the-art approaches, evaluated on two datasets from the literature. Through a developer evaluation, we show that the implementation of a guidance system with DEICTIC requires an effort comparable to compositional approaches, while the definition procedure and the perceived recognition accuracy is comparable to machine learning.

show abstract

Section: Gesture Description Modelsmentioning

confidence: 99%

DEICTIC: A compositional and declarative gesture description based on hidden markov models

Carcangiu

Spano

Fumera

et al. 2019

International Journal of Human-Computer Studies

View full text Add to dashboard Cite

show abstract

“…The literature describes several MMI architectures and frameworks supporting application development considering different approaches such as agents [5,6], components [7,8], layers [9], and server-based [10,11]. However, these architectures lack the modularity that would enable a truly generic approach since, in many aspects, components are tightly coupled, making them hard to expand, replace, or reuse in different applications.…”

Section: Background and Related Workmentioning

confidence: 99%

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Almeida

Teixeira

Silva

et al. 2019

Sensors

View full text Add to dashboard Cite

Technologies, such as smart sensors, actuators, and other kinds of devices, are often installed in our environments (e.g., our Homes) and available to integrate our daily lives. Despite their installation being motivated by the pursuit of automation and increased efficiency, making these environments usable, acceptable and enjoyable in a sustainable, energy efficient way is not only a matter of automation. Tackling these goals is a complex task demanding the combination of different perspectives including building and urban Architecture, Ubiquitous Computing and Human-Computer Interaction (HCI) to provide occupants with the means to shape these environments to their needs. Interaction is of paramount relevance in the creation of adequate relations of users with their environments, but it cannot be seen independently from the ubiquitous sensing and computing or the environment’s architecture. In this regard, there are several challenges to HCI, particularly in how to integrate this multidisciplinary effort. Although there are several solutions to address some of these challenges, the complexity and dynamic nature of the smart environments and the diversity of technologies involved still present many challenges, particularly for its development. In general, the development is complex, and it is hard to create a dynamic environment providing versatile and adaptive forms of interaction. To participate in the multidisciplinary effort, the development of interaction must be supported by tools capable of facilitating co-design by multidisciplinary teams. In this article, we address the development of interaction for complex smart environments and propose the AM4I architecture and framework, a novel modular approach to design and develop adaptive multiplatform multilingual multi-device multimodal interactive systems. The potential of the framework is demonstrated by proof-of-concept applications in two different smart environment contexts, non-residential buildings and smart homes.

show abstract

“…The architecture of this interface is highly dependent on whether a fusion method is integrated into a RIS platform or stand alone. The latter include frameworks like OpenInterface [54], Mudra [40], or SSI [55]. They provide functionality for integrating sensors and performing fusion with the goal to foster modularity and reusability.…”

Section: Contextmentioning

confidence: 99%

“…An elaborated overview is given by [14,39]. Fusion methods can be categorized within a spectrum that extends from early fusion, comprising low-level data stream-oriented approaches, to late fusion, comprising high-level semantic inference-based approaches [4,40,41].…”

Section: Introductionmentioning

confidence: 99%

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Zimmerer

Fischbach

Latoschik

2018

MTI

View full text Add to dashboard Cite

Semantic fusion is a central requirement of many multimodal interfaces. Procedural methods like finite-state transducers and augmented transition networks have proven to be beneficial to implement semantic fusion. They are compliant with rapid development cycles that are common for the development of user interfaces, in contrast to machine-learning approaches that require time-costly training and optimization. We identify seven fundamental requirements for the implementation of semantic fusion: Action derivation, continuous feedback, context-sensitivity, temporal relation support, access to the interaction context, as well as the support of chronologically unsorted and probabilistic input. A subsequent analysis reveals, however, that there is currently no solution for fulfilling the latter two requirements. As the main contribution of this article, we thus present the Concurrent Cursor concept to compensate these shortcomings. In addition, we showcase a reference implementation, the Concurrent Augmented Transition Network (cATN), that validates the concept’s feasibility in a series of proof of concept demonstrations as well as through a comparative benchmark. The cATN fulfills all identified requirements and fills the lack amongst previous solutions. It supports the rapid prototyping of multimodal interfaces by means of five concrete traits: Its declarative nature, the recursiveness of the underlying transition network, the network abstraction constructs of its description language, the utilized semantic queries, and an abstraction layer for lexical information. Our reference implementation was and is used in various student projects, theses, as well as master-level courses. It is openly available and showcases that non-experts can effectively implement multimodal interfaces, even for non-trivial applications in mixed and virtual reality.

show abstract

Mudra

Cited by 44 publications

References 16 publications

DEICTIC: A compositional and declarative gesture description based on hidden markov models

DEICTIC: A compositional and declarative gesture description based on hidden markov models

The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Semantic Fusion for Natural Multimodal Interfaces using Concurrent Augmented Transition Networks

Contact Info

Product

Resources

About