Eye-based Continuous Affect Prediction

Abstract

Eye-based information channels include the pupils, gaze, saccades, fixational movements, and numerous forms of eye opening and closure. Pupil size variation indicates cognitive load and emotion, while a person's gaze direction is said to be congruent with the motivation to approach or avoid stimuli. The eyelids are involved in facial expressions that can encode basic emotions. Additionally, eye-based cues can have implications for human annotators of emotions or feelings. Despite these facts, the use of eye-based cues in affective computing is in its infancy, however, and this work is intended to start to address this. Eye-based feature sets, incorporating data from all of the aforementioned information channels, that can be estimated from video are proposed. Feature set refinement is provided by way of continuous arousal and valence learning and prediction experiments on the RECOLA validation set. The eye-based features are then combined with a speech feature set to provide confirmation of their usefulness and assess affect prediction performance compared with group-of-humans-level performance on the RECOLA test set. The core contribution of this paper, a refined eye-based feature set, is shown to provide benefits for affect prediction. It is hoped that this work stimulates further research into eye-based affective computing.

Gaze-enhanced Crossmodal Embeddings for Emotion Recognition

“…To improve performance, they suggested to include additional modalities and gaze features in future work. O'Dwyer et al [50] explored the use of a larger gaze feature set to train an LSTM network for the task of continuous affect prediction from the RECOLA dataset [55]. They found that their model performed better for arousal prediction when trained on gaze features.…”

“…For example, gaze aversion was shown to impair the perception of anger and happiness [2,10], and embarrassment is connected to more downward gaze than amusement [34]. Despite the importance of gaze, relatively few works have studied emotion recognition based on gaze location and pupil size [5] or combined gaze with other channels of affective information [3,50,62]. While the performance improvements demonstrated by these approaches underline the importance of integrating gaze into emotion recognition systems, they either rely on video information only [62] or assume both video-based gaze features and speech input to be available both at training and test time [3,50].…”

“…More specifically, our approach builds on the crossmodal emotion embeddings model (EmoBed) by Han et al [26]. We augment the visual pipeline of EmoBed with a state-of-the-art gaze feature representation for gaze-based emotion recognition [50]. The resulting novel feature representation of the video modality acts as a helper modality during training for speech-only testing, or is supported by speech during training when performing video-only test evaluations.…”

“…Second, we conduct experiments on the One-Minute Gradual Emotion Recognition dataset [9], improving over the previous state of the art both for video-only as well as audio-only testing. Third, we perform extensive ablation experiments and report results for different gaze integration strategies, including early versus model-level fusion, as well as different state-of-the-art gaze feature representations [50,62].…”

Sensors and Artificial Intelligence Methods and Algorithms for Human–Computer Intelligent Interaction: A Systematic Mapping Study

“…Eye-tracking-based UIs are, for example, various assistive technology solutions for people with severe disabilities (e.g., [ 47 ]) that cannot use arms and standard input devices. However, eye-based cues (e.g., eye gaze) are another field of increasing interest to the research community for automatic emotion classification and affect prediction [ 48 ].…”