The AM4I Architecture and Framework for Multimodal Interaction and Its Application to Smart Environments

Abstract

Technologies, such as smart sensors, actuators, and other kinds of devices, are often installed in our environments (e.g., our Homes) and available to integrate our daily lives. Despite their installation being motivated by the pursuit of automation and increased efficiency, making these environments usable, acceptable and enjoyable in a sustainable, energy efficient way is not only a matter of automation. Tackling these goals is a complex task demanding the combination of different perspectives including building and urban Architecture, Ubiquitous Computing and Human-Computer Interaction (HCI) to provide occupants with the means to shape these environments to their needs. Interaction is of paramount relevance in the creation of adequate relations of users with their environments, but it cannot be seen independently from the ubiquitous sensing and computing or the environment’s architecture. In this regard, there are several challenges to HCI, particularly in how to integrate this multidisciplinary effort. Although there are several solutions to address some of these challenges, the complexity and dynamic nature of the smart environments and the diversity of technologies involved still present many challenges, particularly for its development. In general, the development is complex, and it is hard to create a dynamic environment providing versatile and adaptive forms of interaction. To participate in the multidisciplinary effort, the development of interaction must be supported by tools capable of facilitating co-design by multidisciplinary teams. In this article, we address the development of interaction for complex smart environments and propose the AM4I architecture and framework, a novel modular approach to design and develop adaptive multiplatform multilingual multi-device multimodal interactive systems. The potential of the framework is demonstrated by proof-of-concept applications in two different smart environment contexts, non-residential buildings and smart homes.

An Accessible Smart Home Based on Integrated Multimodal Interaction

“…For users to be in control of their homes, it is necessary to provide a way of interacting with them, allowing access to relevant information and control of the different devices. There are several input and output modalities that can be used for interaction, including text, speech, graphical, touch, gestures, and gaze [ 19 ].…”

“…Besides a natural and integrated interaction, another important aspect to take into account is the diversity of users of smart homes [ 17 ], as well as the dynamic context of interaction [ 1 , 19 ]. If interaction does not adapt to the user, many users can be excluded due to not being able to access their own home.…”

“…An architecture and framework for multimodal interaction was proposed and demonstrated through proof-of-concept applications in smart environments, including smart homes [ 19 , 41 ]. In another contribution, a multimodal interaction system based on three different modalities (eye blinking, speech, and touch) was implemented with the aim of allowing people with limited physical mobility to control the devices and appliances of a smart home [ 43 ].…”

“…This issue can be minimised with adaptive interaction, where the assistant and modalities are able to adapt to the current user(s) and context. This adaptation can be achieved by relying on information obtained from sensors embedded in the device used for interaction and/or dedicated sensors deployed at home [ 19 ].…”

“…Evolution of AM4I Architecture—Addition of support for adaptation to the user and context to the state-of-the-art interaction architecture AM4I (Adaptive Multiplatform Multidevice Multilingual Multimodal Interaction) [ 19 ], and the demonstration of the AM4I capabilities for smart home interaction.…”