Video-based Facial Micro-Expression Analysis: A Survey of Datasets, Features and Algorithms

Abstract

Unlike the conventional facial expressions, micro-expressions are involuntary and transient facial expressions capable of revealing the genuine emotions that people attempt to hide. Therefore, they can provide important information in a broad range of applications such as lie detection, criminal detection, etc. Since micro-expressions are transient and of low intensity, however, their detection and recognition is difficult and relies heavily on expert experiences. Due to its intrinsic particularity and complexity, video-based micro-expression analysis is attractive but challenging, and has recently become an active area of research. Although there have been numerous developments in this area, thus far there has been no comprehensive survey that provides researchers with a systematic overview of these developments with a unified evaluation. Accordingly, in this survey paper, we first highlight the key differences between macro-and micro-expressions, then use these differences to guide our research survey of video-based micro-expression analysis in a cascaded structure, encompassing the neuropsychological basis, datasets, features, spotting algorithms, recognition algorithms, applications and evaluation of state-of-the-art approaches. For each aspect, the basic techniques, advanced developments and major challenges are addressed and discussed. Furthermore, after considering the limitations of existing micro-expression datasets, we present and release a new dataset -called micro-and-macro expression warehouse (MMEW)containing more video samples and more labeled emotion types. We then perform a unified comparison of representative methods on CAS(ME) 2 for spotting, and on MMEW and SAMM for recognition, respectively. Finally, some potential future research directions are explored and outlined.

An Overview of Facial Micro-Expression Analysis: Data, Methodology and Challenge

“…1. Several surveys on MER have been published in recent years [24], [25], [26], [27], [28], [29], [30], [23], [31]. However, most of them have been focused on traditional image-processing methods and a fresh overview…”

“…MEs are more likely to occur under high-arousal stimuli, so video clips with high emotional valence are proved to be effective materials for eliciting MEs [8], [9], [10], [20]. While recently proposed MMEW [23] adopted another strategy by constructing a high-stakes situation, i.e., poker games or TV interviews with difficult questions.…”

“…Considering real-world scenarios, macro-and microexpressions could co-occur, there are three public datasets combining macro and micro facial expressions, i.e., SAMM Long Videos [22], CAS(ME) 2 [21] and MMEW [23]. All of them can be empolyed for macro-expression and microexpression spotting from long videos.…”

Deep Learning Based Emotion Recognition and Visualization of Figural Representation

“…For example, Zheng et al (2020) they proposed a method for comparing skull similarity based on SPCA. Ben et al (2021) they proposed a video-based facial micro-expression analysis method. In summary, the deep learning algorithm has been studied in many aspects, such as speech-based image recognition, and it has great application value in speech and image recognition.…”

“…Some related research of emotion recognition or similarity measure methods of recognition have also gained some achievements. For example,Zheng et al (2020) they proposed a method for comparing skull similarity based on SPCA Ben et al (2021). they proposed a video-based facial microexpression analysis method.…”