Affective computing using speech and eye gaze: a review and bimodal system proposal for continuous affect prediction

Abstract

Speech has been a widely used modality in the field of affective computing. Recently however, there has been a growing interest in the use of multi-modal affective computing systems. These multi-modal systems incorporate both verbal and non-verbal features for affective computing tasks. Such multi-modal affective computing systems are advantageous for emotion assessment of individuals in audio-video communication environments such as teleconferencing, healthcare, and education. From a review of the literature, the use of eye gaze features extracted from video is a modality that has remained largely unexploited for continuous affect prediction. This work presents a review of the literature within the emotion classification and continuous affect prediction sub-fields of affective computing for both speech and eye gaze modalities. Additionally, continuous affect prediction experiments using speech and eye gaze modalities are presented. A baseline system is proposed using open source software, the performance of which is assessed on a publicly available audio-visual corpus. Further system performance is assessed in a cross-corpus and cross-lingual experiment.The experimental results suggest that eye gaze is an effective supportive modal-$ Paper submission for review

Feasibility Study on the Role of Personality, Emotion, and Engagement in Socially Assistive Robotics: A Cognitive Assessment Scenario

“…We followed the procedure that is commonly adopted in the cognitive assessment scenario (see Table 2), by including all the necessary tools for assessing the factors of interest. Given the advances in affective computing [49] and automatic personality detection [50] fields, we would like to adopt these solutions to improve the perception capabilities of the robotic platform. It will allow the robot to automatically infer the individual's traits.…”

Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

“…Their results further showed that gaze features were even more helpful than speech. In a study by O'Dwyer et al [49], the addition of eye gaze features to speech yielded an improvement of 19.5% for valence prediction and 3.5% for arousal prediction. An opposing pattern was found by O'Dwyer et al [50] who employed pupillometry and gaze features for valence and arousal estimation on the RECOLA dataset [55].…”