An efficient multimodal 2D + 3D feature-based approach to automatic facial expression recognition

Abstract

Highlights• We propose a feature-based 2D+3D multimodal facial expression recognition method.• It is fully automatic benefit from a large set of automatically detected landmarks.• The complementarities between 2D and 3D features are comprehensively demonstrated.• Our method achieves the best accuracy on the BU-3DFE database so far.• A good generalization ability is shown on the Bosphorus database. AbstractWe present a fully automatic multimodal 2D + 3D feature-based facial expression recognition approach and demonstrate its performance on the BU-3DFE database. Our approach combines multi-order gradient-based local texture and shape descriptors in order to achieve efficiency and robustness. First, a large set of fiducial facial landmarks of 2D face images along with their 3D face scans are localized using a novel algorithm namely incremental Parallel Cascade of Linear Regression (iPar-CLR). Then, a novel Histogram of Second Order Gradients (HSOG) based local image descriptor in conjunction with the widely used first-order gradient based SIFT descriptor are used to describe the local texture around each 2D landmark. Similarly, the local geometry around each 3D landmark is described by two novel local shape descriptors constructed using the first-order and the second-order surface differential geometry quantities, i.e., Histogram of mesh Gradients (meshHOG) and Histogram of mesh Shape index (curvature quantization, meshHOS). Fi- * Corresponding author A C C E P T E D M A N U S C R I P Tnally, the Support Vector Machine (SVM) based recognition results of all 2D and 3D descriptors are fused at both feature-level and score-level to further improve the accuracy. Comprehensive experimental results demonstrate that there exist impressive complementary characteristics between the 2D and 3D descriptors. We use the BU-3DFE benchmark to compare our approach to the state-of-the-art ones. Our multimodal feature-based approach outperforms the others by achieving an average recognition accuracy of 86.32%. Moreover, a good generalization ability is shown on the Bosphorus database.

Saliency-based framework for facial expression recognition

“…Most of the methods have been developed for facial expressions recognition from static 3D facial expression data [37][38][39][40][41]. However, recently community has proposed methods that employ dynamic 3D facial expression data for this purpose [42,43].…”

Accurate Facial Parts Localization and Deep Learning for 3D Facial Expression Recognition

“…3D FER has become an extensive field of research with many early attempts in [3], [4], [5], [6], [7] and most recent works in [8], [9], [10] that trend to use both 2D and 3D multi-modal data to further improve the accuracy. Huynh et al [11] proposed to use deep CNNs for classifying the six basic facial expressions.…”

“…1. Given textured 3D face scans, we first generate well-aligned texture and depth images as in [8]. Then, 2D facial landmarks are detected and used to correct facial pose and extract 4 key facial parts (i.e.…”

“…1) Facial Landmark localization: 2D texture face images are created by projecting each textured 3D facial scan into a 2D regular grid domain using the mapping and interpolation techniques in [8]. Following this, 49 2D facial landmarks are generated on the 2D texture face images using a protocol called Incremental Parallel Cascade of Linear Regressors (iPar-CLR [18]).…”

Experimental image and range scanner datasets fusion in SHM for displacement detection

“…Applications of this method can be seen in historical preservation, urban planning, digital mapping, structure, navigation, virtual reality, e‐commerce, digital cinematography, and computer games, just to name a few. Historically, fusion has been used for automation, improved accuracy, and reliability; related work addressing 2D and 3D integration has been performed for automatic reconstruction of urban scenes, large‐scale benchmark, face recognition, terrestrial mapping, medical imagining, and robot vision, to mention some. Now we are proposing a new method for 2D and 3D datasets fusion by an AI algorithm, which is a mathematical model that describes the integration of 2D vision‐based (camera with CCD sensor) and a 3D vision‐based (range scanner with the incoherent light emitter and photodiode sensor).…”