Addressing Data Scarcity in Multimodal User State Recognition by Combining Semi-Supervised and Supervised Learning

Abstract

Detecting mental states of human users is crucial for the development of cooperative and intelligent robots, as it enables the robot to understand the user's intentions and desires. Despite their importance, it is difficult to obtain a large amount of high quality data for training automatic recognition algorithms as the time and effort required to collect and label such data is prohibitively high. In this paper we present a multimodal machine learning approach for detecting dis-/agreement and confusion states in a human-robot interaction environment, using just a small amount of manually annotated data. We collect a data set by conducting a human-robot interaction study and develop a novel preprocessing pipeline for our machine learning approach. By combining semi-supervised and supervised architectures, we are able to achieve an average F1-score of 81.1% for dis-/agreement detection with a small amount of labeled data and a large unlabeled data set, while simultaneously increasing the robustness of the model compared to the supervised approach. CCS CONCEPTS• Human-centered computing → HCI theory, concepts and models; HCI design and evaluation methods; User studies.

Few Labels are Enough! Semi-supervised Graph Learning for Social Interaction

“…Furthermore, AL techniques can encounter challenges with technically complex data, as designing an effective query strategy is non-trivial, and uninformative examples may be selected. Recently, Voß and colleagues [48] tackled multimodal dis-/agreement classification in human-robot interactions and YouTube videos using semi-supervised deep architectures. While their work demonstrates promising results, it still relies on a supervised branch for the final classification, necessitating a significant amount of labeled examples to generalize effectively.…”