Automatic Classification and Shift Detection of Facial Expressions in Event-Aware Smart Environments

Abstract

Affective application developers often face a challenge in integrating the output of facial expression recognition (FER) software in interactive systems: although many algorithms have been proposed for FER, integrating the results of these algorithms into applications remains difficult. Due to interand within-subject variations further post-processing is needed. Our work addresses this problem by introducing and comparing three post-processing classification algorithms for FER output applied to an event-based interaction scheme to pinpoint the affective context within a time window. Our comparison is based on earlier published experiments with an interactive cycling simulation in which participants were provoked with game elements and their facial expression responses were analysed by all three algorithms with a human observer as reference. The three post-processing algorithms we investigate are mean fixed-window, matched filter, and Bayesian changepoint detection. In addition, we introduce a novel method for detecting fast transition of facial expressions, which we call emotional shift. The proposed detection pattern is suitable for affective applications especially in smart environments, wherever users' reactions can be tied to events.

Context-Aware Emotion Recognition in the Wild Using Spatio-Temporal and Temporal-Pyramid Models

“…In some cases, emotions cannot be interpreted without context [4]. In video-based emotion recognition, facial expression representation often includes three periods, onset, apex and offset [5,6], as shown in Figure 1. The lengths of the periods differ; the onset and offset periods tend to be shorter than the apex period.…”

“…The duration of each varies, leading to unclear temporal borders. In addition, the appearance of spontaneous expressions leads to the presence of multiple apexes [6].…”