Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition

Abstract

Automatic affect analysis has attracted great interest in various contexts including the recognition of action units and basic or non-basic emotions. In spite of major efforts, there are several open questions on what the important cues to interpret facial expressions are and how to encode them. In this paper, we review the progress across a range of affect recognition applications to shed light on these fundamental questions. We analyse the state-of-the-art solutions by decomposing their pipelines into fundamental components, namely face registration, representation, dimensionality reduction and recognition. We discuss the role of these components and highlight the models and new trends that are followed in their design. Moreover, we provide a comprehensive analysis of facial representations by uncovering their advantages and limitations; we elaborate on the type of information they encode and discuss how they deal with the key challenges of illumination variations, registration errors, head-pose variations, occlusions, and identity bias. This survey allows us to identify open issues and to define future directions for designing real-world affect recognition systems.

Uncertainty Flow Facilitates Zero-Shot Multi-Label Learning in Affective Facial Analysis

“…The research on multi-label affective discrimination is also in line with the finding that the decision boundaries among classes are less ostentatious in affective analysis compare to other categorisation problems, e.g., object classification [2,3]. Benefited from previous researches on multi-label classification in general, it appears straightforward to extend affective computing along this direction.…”

“…The same facial expression may be interpreted differently dependent upon its associated contexts, spatial and temporal cues [1,2]. Hence, affective classification, in a nutshell, is ambiguous [3]. The past effort in resolving this ambiguity has been reflected in lowering the single-label dependency in producing emotion categories [4].…”

“…This pioneer research should be credited under following contributions: (1) We develop a novel inductive transfer learning [3] based computational framework that allows multi-label affective prediction on single evoked source. (2) Unlike conventional inductive transfer learning, the proposed Uncertainty Flow focuses on model uncertainty rather than the mere model weights in knowledge distillation.…”

Natural Facial and Head Behavior Recognition using Dictionary of Motion Primitives

“…On the other hand, Bousmalis et al [18], Jung et al [19], and Dapogny et al [20], use Hierarchical CRF, neural networks, and random forests for representing a temporal sequences of framewise static cues, respectively. Evangelos et al [21] generates a temporal BoW from framewise static cues. Unlike these state-of-the-art methods, our proposed method encodes the temporal sequences to be the BoMP.…”

Aff-Wild: Valence and Arousal ‘In-the-Wild’ Challenge

“…Some representative datasets, which are still used in many recent works [18], include the Cohn-Kanade database [33,24], the MMI database [29,35], the Multi-PIE database [17] and the BU-3D and BU-4D databases [41,40]. Nevertheless, it is now accepted by the community that the facial expressions of naturalistic behaviour could be radically different from the posed ones [9,32,44]. Hence, efforts have been made in order to collect subjects displaying naturalistic behaviour.…”