Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition

Abstract

Meaningful facial parts can convey key cues for both facial action unit detection and expression prediction. Textured 3D face scan can provide both detailed 3D geometric shape and 2D texture appearance cues of the face which are beneficial for Facial Expression Recognition (FER). However, accurate facial parts extraction as well as their fusion are challenging tasks. In this paper, a novel system for 3D FER is designed based on accurate facial parts extraction and deep feature fusion of facial parts. In particular, each textured 3D face scan is firstly represented as a 2D texture map and a depth map with one-to-one dense correspondence. Then, the facial parts of both texture map and depth map are extracted using a novel 4-stage process consists of facial landmark localization, facial rotation correction, facial resizing, facial parts bounding box extraction and post-processing procedures. Finally, deep fusion Convolutional Neural Networks (CNNs) features of all facial parts are learned from both texture maps and depth maps, respectively and nonlinear SVMs are used for expression prediction. Experiments are conducted on the BU-3DFE database, demonstrating the effectiveness of combing different facial parts, texture and depth cues and reporting the state-of-the-art results in comparison with all existing methods under the same setting.

3D Approaches and Challenges in Facial Expression Recognition Algorithms—A Literature Review

“…Features are usually calculated on the region surrounding principal facial landmarks or on the mouth and eyes that inherently contain essential information for emotion recognition. These key features that are considered closely related to expression categories, in order to perform FER are fed to various classifiers, as well as Support-Vector Machines (SVM) [47][48][49][50][51], Adaboost, k-Nearest Neighbors (k-NN), Linear Discriminant Analysis (LDA), Modified Principal Component Analysis (PCA), Hidden Markov Model (HMM) [44][45][46], Random Forest [52] or Neural Networks [51,53,54].…”

“…Savran et al [33] showed that in general 3D data perform better than 2D data, especially for lower face AUs, but with the fusion of two modalities higher detection rates are achieved (97.1 %). Depth maps (3D meshes) are fused with other 2D maps, such as texture maps [2,51,107], curvature maps [2,108] and normal maps [2,107].…”

“…Some of the last year's works are [112][113][114][115][116][117]. Others are those of Jan et al [51] and Zhu et al [118]. Jan et al [51] designed a novel system for 3D FER based on accurate facial parts extraction according to localized facial landmarks, and deep feature fusion of facial parts.…”

“…Others are those of Jan et al [51] and Zhu et al [118]. Jan et al [51] designed a novel system for 3D FER based on accurate facial parts extraction according to localized facial landmarks, and deep feature fusion of facial parts. With the use of facial parts, they achieved better performance than using the entire face.…”

An Separation of Different Facial Expression Recognition System

“…Jan et.al [4] suggested that an accurate location of facial components and profound teaching for identification of 3D facial expression using techniques. In this paper a new scheme for 3D facial recognition is intended based on precise facial components removal as well as their combination are difficult duties depending on parameters of texture, facial parts 3 dimensions and its outcomes in that the suggested method image 3D facial recognition offers an efficient identification of facial parts and textures and the benefit of this strategy or study that accurate recognition of facial parts can be done.…”