Effective recognition of facial micro-expressions with video motion magnification

Wang, Yandan; See, John; Oh, Yee-Hui; Phan, Raphaël C.‐W.; Rahulamathavan, Yogachandran; Ling, Huo-Chong; Tan, Su Wei; Li, Xujie

doi:10.1007/s11042-016-4079-6

Cited by 79 publications

(26 citation statements)

References 51 publications

(79 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Accuracy LBP-TOP+SVM [26] 75.30% MDMO+SVM [19] 67.37% CNN+LSTM [12] 60.98% Proposed CNN method 75.57%…”

Section: Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets

Takalkar

2017

2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA)

View full text Add to dashboard Cite

Abstract-Facial micro-expression refers to split-second muscle changes in the face, indicating that a person is either consciously or unconsciously suppressing their true emotions and even mental health. Therefore, micro-expression recognition attracts increasing research efforts in both fields of psychology and computer vision. Existing research on micro-expression recognition has mainly used hand-crafted features, for example, Local Binary Pattern-Three Orthogonal Planes (LBP-TOP), Gabor filter and optical flow. Recently, Deep Convolutional neural systems have demonstrated a high degree effectiveness for difficult face recognition tasks. This paper explores the possible use of deep learning for micro-expression recognition. To develop a reliable deep neural network extensive training sets are required with a huge number of labeled image samples. However, micro-expression recognition is a challenging task due to the repressed facial appearance and short duration, which results in the lack of training data. In this paper, we propose to generate extensive training datasets of synthetic images using data augmentation on CASME and CASME II databases. Then, these datasets are combined to tune a satisfactory CNN-based microexpression recognizer. Experimental results demonstrate the effectiveness of the proposed CNN approach in image based micro-expression recognition and present comparable results with the best-related works.

show abstract

“…Accuracy LBP-TOP+SVM [26] 75.30% MDMO+SVM [19] 67.37% CNN+LSTM [12] 60.98% Proposed CNN method 75.57%…”

Section: Methodsmentioning

confidence: 99%

“…In [6,12,19,25,26], the researchers have considered the temporal factor from the video for recognition of microexpressions which have contributed an additional feature in the calculations. In case of our method, we tried to eliminate the temporal factor and simply exhibit the image based microexpression recognition approach.…”

Section: Methodsmentioning

confidence: 99%

Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets

Takalkar

2017

2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA)

View full text Add to dashboard Cite

show abstract

“…One solution to this problem is to exaggerate or magnify these facial micro-movements. In recent works (Park et al, 2015; Zarezadeh and Rezaeian, 2016; Li et al, 2017; Wang et al, 2017), the Eulerian Motion Magnification (EMM) (Wu et al, 2012) method was employed to magnify the subtle motions in the ME videos. The EMM method extracts the frequency bands of interest from the different spatial frequency bands obtained from the decomposition of an input video, by using band-pass filters; these extracted bandpass signals at different spatial level are amplified by a magnification factor α to magnify the motions.…”

Section: Recognition Of Facial Micro-expressionsmentioning

confidence: 99%

“…Among appearance-based feature extraction methods, local binary pattern on three orthogonal planes (LBP-TOP) is widely applied in many works (Li et al, 2013; Guo et al, 2014; Le Ngo et al, 2014, 2015, 2016a,b; Yan et al, 2014a; Adegun and Vadapalli, 2016; Zheng et al, 2016; Wang et al, 2017). Most existing datasets (SMIC, CASME II, SAMM) have all reported the LBP-TOP as their baseline evaluation method.…”

Section: Recognition Of Facial Micro-expressionsmentioning

confidence: 99%

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

et al. 2018

Self Cite

View full text Add to dashboard Cite

Over the last few years, automatic facial micro-expression analysis has garnered increasing attention from experts across different disciplines because of its potential applications in various fields such as clinical diagnosis, forensic investigation and security systems. Advances in computer algorithms and video acquisition technology have rendered machine analysis of facial micro-expressions possible today, in contrast to decades ago when it was primarily the domain of psychiatrists where analysis was largely manual. Indeed, although the study of facial micro-expressions is a well-established field in psychology, it is still relatively new from the computational perspective with many interesting problems. In this survey, we present a comprehensive review of state-of-the-art databases and methods for micro-expressions spotting and recognition. Individual stages involved in the automation of these tasks are also described and reviewed at length. In addition, we also deliberate on the challenges and future directions in this growing field of automatic facial micro-expression analysis.

show abstract

“…Motion Magnification: Since local intensity changes and facial movement changes in ME are subtle, effective ME characteristics are difficult to capture. To tackle these issues, following [47], [67] we use Eulerian Video Magnification (EVM) [68] to magnify the subtle motions in videos. The goal is to consider the time series of intensity values at any spatial location (pixel) and amplify variation in a given temporal frequency band of interest.…”

Section: Training Testingmentioning

confidence: 99%

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

et al. 2019

View full text Add to dashboard Cite

Facial Micro-Expressions (MEs) are spontaneous, involuntary facial movements when a person experiences an emotion but deliberately or unconsciously attempts to conceal his or her genuine emotions. Recently, ME recognition has attracted increasing attention due to its potential applications such as clinical diagnosis, business negotiation, interrogations, and security. However, it is expensive to build large scale ME datasets, mainly due to the difficulty of inducing spontaneous MEs. This limits the application of deep learning techniques which require lots of training data.In this paper, we propose a simple, efficient yet robust descriptor called Extended Local Binary Patterns on Three Orthogonal Planes (ELBPTOP) for ME recognition. ELBPTOP consists of three complementary binary descriptors: LBPTOP and two novel ones Radial Difference LBPTOP (RDLBPTOP) and Angular Difference LBPTOP (ADLBPTOP), which explore the local second order information along the radial and angular directions contained in ME video sequences. ELBPTOP is a novel ME descriptor inspired by unique and subtle facial movements. It is computationally efficient and only marginally increases the cost of computing LBPTOP, yet is extremely effective for ME recognition. In addition, by firstly introducing Whitened Principal Component Analysis (WPCA) to ME recognition, we can further obtain more compact and discriminative feature representations, then achieve significantly computational savings.Extensive experimental evaluation on three popular spontaneous ME datasets SMIC, CASME II and SAMM show that our proposed ELBPTOP approach significantly outperforms the previous state-of-the-art on all three single evaluated datasets and achieves promising results on cross-database recognition. Our code will be made available.Index Terms-Micro-expression recognition, local binary pattern, feature extraction. ADLBPTOP 2,4 + LBPTOP 3,4 + RDLBPXOT 3,4,1 Acc(%):73.94 ADLBPTOP 1,8 + LBPTOP 3,4 + RDLBPXOT 4,4,3 Acc(%):69.06 SMIC CASME II LBPTOP 3,4 Acc(%):71.55 ADLBPTOP 2,4 Acc(%):70.20 RDLBPXOT 3,4,1 Acc(%):69.24 LBPTOP 3,4 Acc(%):65.16 ADLBPTOP 1,8 Acc(%):63.73 RDLBPXOT 4,4,3 Acc(%):62.57

show abstract

Effective recognition of facial micro-expressions with video motion magnification

Cited by 79 publications

References 51 publications

Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets

Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets

A Survey of Automatic Facial Micro-Expression Analysis: Databases, Methods, and Challenges

Extended Local Binary Patterns for Efficient and Robust Spontaneous Facial Micro-Expression Recognition

Contact Info

Product

Resources

About