A Model-based Approach for Engineering Multimodal Mobile Interactions

Abstract

Mobile phones have benefited from many enhancements since several years, including an increasing number of their embedded sensors (accelerometer, light sensor...). Created for giving contextaware abilities, these sensors also allow new types of interaction and pave the way to a great expansion of multimodal mobile interactions. Unfortunately, the current context of mobile software development makes difficult the development of multimodal applications. We propose our model-based solution that aims to facilitate the development of multimodal mobile interfaces. We adopt the principles of the Model-Driven Engineering (MDE) which is particularly fitted for such context. Our proposition includes M4L modeling language, library of high-level sensor-based modalities and MIMIC framework. Our approach respects the main criteria of the MDE in order to define an efficient model-based approach. Results of the first evaluation suggest that using our approach facilitates the development of sensor-based multimodal mobile interfaces.

Using programming by demonstration for multimodality in mobile-human interactions

“…MIMIC (MobIle MultImodality Creator) [14] est un Framework à base de modèles pour la construction d'applications mobiles multimodales. Il permet la modélisation graphique des interactions multimodales en entrée et en sortie avec un langage de modélisation appelé M4L (Mobile MultiModality Modeling Language) ainsi que la génération automatique du code de ces interactions (sous Android, iOS et HTML5/CSS3) [13].…”

Multimodal Interaction Framework Based on Firebase Real-Time Database

“…Elouali et al [14] suggested benefiting from this mobile technology for multimodal interaction. More specifically, smartphones are nowadays equipped with multiple sensors (gyroscope, compass, microphone, camera…) which can be used to acquire a large spectrum of user input and utilizing a various interaction modalities.…”