Adaptive User-Centered Multimodal Interaction towards Reliable and Trusted Automotive Interfaces

Abstract

With the recently increasing capabilities of modern vehicles, novel approaches for interaction emerged that go beyond traditional touch-based and voice command approaches. Therefore, hand gestures, head pose, eye gaze, and speech have been extensively investigated in automotive applications for object selection and referencing. Despite these significant advances, existing approaches mostly employ a one-model-fits-all approach unsuitable for varying user behavior and individual differences. Moreover, current referencing approaches either consider these modalities separately or focus on a stationary situation, whereas the situation in a moving vehicle is highly dynamic and subject to safety-critical constraints. In this paper, I propose a research plan for a user-centered adaptive multimodal fusion approach for referencing external objects from a moving vehicle. The proposed plan aims to provide an open-source framework for user-centered adaptation and personalization using user observations and heuristics, multimodal fusion, clustering, transfer-of-learning for model adaptation, and continuous learning, moving towards trusted human-centered artificial intelligence. CCS CONCEPTS• Human-centered computing → User centered design; HCI theory, concepts and models; HCI design and evaluation methods;• Computing methodologies → Neural networks.

Towards Adaptive User-centered Neuro-symbolic Learning for Multimodal Interaction with Autonomous Systems

“…Although these approaches apply to different domains, we focus on the automotive domain as an example of the rich work on driver personalization. More specifically, we demonstrate our suggestion on some of our previous work in the field of adaptive user interaction for the automotive domain (Gomaa et al, 2020;Gomaa, 2022;Gomaa et al, 2022;Meiser et al, 2022;Feld et al, 2019); however, the underlying learning techniques are valid for other domains as well.…”

“…The first stage of the proposed plan is to understand the variances in driver behavior when performing the multimodal referencing task as in (Gomaa et al, 2020;2022;Rümelin et al, 2013). As an example, in the automotive domain, drivers perform different multimodal gestures to control the vehicle and query surrounding objects.…”

“…Figure 2. Proposed approach for model adaptation to generate personalized models through transfer and incremental learning techniques from (Gomaa, 2022).…”