Multimodal interaction: a survey from model driven engineering and mobile perspectives

Elouali, Nadia; Rouillard, José; Pallec, Xavier Le; Tarby, Jean-Claude

doi:10.1007/s12193-013-0126-z

Cited by 16 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…According to Bellik and Teil [6] [8] formally present the modality as: m=(d,r)|(m,r), where "d" denotes the physical I/O device, "r" an interaction language (representational system) and "m" an interaction modality. For example, the speech modality can be defined using the "Microphone" as a physical device and "Pseudo-natural language" as an interaction language.…”

Section: Interaction Modalitiesmentioning

confidence: 99%

A Multimodal Interaction Framework for Blended Learning

Vidakis

2017

EAI Endorsed Transactions on Creative Technologies

View full text Add to dashboard Cite

Humans interact with each other by utilizing the five basic senses as input modalities, whereas sounds, gestures, facial expressions etc. are utilized as output modalities. Multimodal interaction is also used between humans and their surrounding environment, although enhanced with further senses such as equilibrioception and the sense of balance. Computer interfaces that are considered as a different environment that human can interact with, lack of input and output amalgamation in order to provide a close to natural interaction. Multimodal human-computer interaction has sought to provide alternative means of communication with an application, which will be more natural than the traditional "windows, icons, menus, pointer" (WIMP) style. Despite the great amount of devices in existence, most applications make use of a very limited set of modalities, most notably speech and touch. This paper describes a multimodal framework enabling deployment of a vast variety of modalities, tailored appropriately for use in blended learning environment and introduces a unified and effective framework for multimodal interaction called COALS.

show abstract

Section: Interaction Modalitiesmentioning

confidence: 99%

A Multimodal Interaction Framework for Blended Learning

Vidakis

2017

EAI Endorsed Transactions on Creative Technologies

View full text Add to dashboard Cite

show abstract

“…Some of them define criteria whose respect will lead to a better efficiency (speed, comfort and accuracy) of modeling languages as well as any associated modeling environment. The main criteria are: pertinence/concision of concepts, model guidance, model reuse, model checking and model enactment [9]:…”

Section: Effective Mde-based Approach For the Mobile Multimodalitymentioning

confidence: 99%

A Model-based Approach for Engineering Multimodal Mobile Interactions

Elouali

Pallec

Rouillard

et al. 2014

Proceedings of the 12th International Conference on Advances in Mobile Computing and Multimedia

Self Cite

View full text Add to dashboard Cite

Mobile phones have benefited from many enhancements since several years, including an increasing number of their embedded sensors (accelerometer, light sensor...). Created for giving contextaware abilities, these sensors also allow new types of interaction and pave the way to a great expansion of multimodal mobile interactions. Unfortunately, the current context of mobile software development makes difficult the development of multimodal applications. We propose our model-based solution that aims to facilitate the development of multimodal mobile interfaces. We adopt the principles of the Model-Driven Engineering (MDE) which is particularly fitted for such context. Our proposition includes M4L modeling language, library of high-level sensor-based modalities and MIMIC framework. Our approach respects the main criteria of the MDE in order to define an efficient model-based approach. Results of the first evaluation suggest that using our approach facilitates the development of sensor-based multimodal mobile interfaces.

show abstract

“…The cooperation between these tools is difficult; if not impossible. They are not only technically different, but also based on different modeling paradigms (State machine, Petri net, Hierarchy and Component paradigm [6]). Models are, therefore, not compatible with each other.…”

Section: Conflicts Between Sensors-based Interactionsmentioning

confidence: 99%

Mimic

Elouali

Pallec

Rouillard

et al. 2014

CHI '14 Extended Abstracts on Human Factors in Computing Systems

Self Cite

View full text Add to dashboard Cite

In recent years, there has been an increasing interest in the presence of sensors in mobile devices. Emergence of multiple modalities based on these sensors greatly enriches the human-mobile interaction. However, mobile applications slightly involve sensors and rarely combine them simultaneously. In this paper, we seek to remedy this problem by detailing the key challenges that face developers who want to integrate several sensor-based modalities and combine them. We then present our model-based approach solution. We introduce M4L modeling language and MIMIC framework that aim to produce easily sensor-based multimodal mobile applications by generating up to 100% of their interfaces.

show abstract

Multimodal interaction: a survey from model driven engineering and mobile perspectives

Cited by 16 publications

References 27 publications

A Multimodal Interaction Framework for Blended Learning

A Multimodal Interaction Framework for Blended Learning

A Model-based Approach for Engineering Multimodal Mobile Interactions

Mimic

Contact Info

Product

Resources

About