Development of a Robust Multi-Scale Featured Local Binary Pattern for Improved Facial Expression Recognition

Abstract

Compelling facial expression recognition (FER) processes have been utilized in very successful fields like computer vision, robotics, artificial intelligence, and dynamic texture recognition. However, the FER’s critical problem with traditional local binary pattern (LBP) is the loss of neighboring pixels related to different scales that can affect the texture of facial images. To overcome such limitations, this study describes a new extended LBP method to extract feature vectors from images, detecting each image from facial expressions. The proposed method is based on the bitwise AND operation of two rotational kernels applied on LBP(8,1) and LBP(8,2) and utilizes two accessible datasets. Firstly, the facial parts are detected and the essential components of a face are observed, such as eyes, nose, and lips. The portion of the face is then cropped to reduce the dimensions and an unsharp masking kernel is applied to sharpen the image. The filtered images then go through the feature extraction method and wait for the classification process. Four machine learning classifiers were used to verify the proposed method. This study shows that the proposed multi-scale featured local binary pattern (MSFLBP), together with Support Vector Machine (SVM), outperformed the recent LBP-based state-of-the-art approaches resulting in an accuracy of 99.12% for the Extended Cohn–Kanade (CK+) dataset and 89.08% for the Karolinska Directed Emotional Faces (KDEF) dataset.

Patch Attention Layer of Embedding Handcrafted Features in CNN for Facial Expression Recognition

“…Shan et al [ 15 ] empirically evaluated facial representation based on a statistical local feature called LBP, experiments had shown that the LBP feature has a better, stable, and robust performance when the input facial images have different forms. To overcome the limitation that traditional LBP can lose the neighboring pixels related to different scales that can affect the texture of facial images, Yasmin et al [ 30 ] proposed a new extended LBP method based on the bitwise “AND” operation of two rotational kernels to extract facial features. In view of satisfactory performance of the LBP operator, the CNNs that integrate advantages of the LBP have been developed [ 41 , 55 , 56 ].…”

“…Facial expressions can be divided into six basic emotions, namely, anger (An); disgust (Di); fear (Fe); happiness (Ha); sadness (Sa); surprise (Su); and one neutral (Ne) emotion [ 9 ], contempt (Co), was subsequently added as one of the basic emotions [ 10 ]. Recognition of these emotions can be categorized into image-based [ 11 , 12 , 13 , 14 , 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 , 25 , 26 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 ] and video-based [ 38 , 39 , 40 , 41 , 42 , 43 ] approaches. Image-based approaches only use information about the static input image to determine the category of facial expression; on the other hand, except when the spatial features extracted from a static image are available, video-based approaches can also use temporal information of a dynamic image sequence to capture the temporal changes of facial appearance when some facial expression occurs.…”

“…FER can also be divided into the traditional method [ 15 , 27 , 30 , 31 , 32 , 38 , 40 ], deep learning method [ 16 , 17 , 18 , 20 , 21 , 23 , 24 , 25 , 26 , 35 , 36 , 39 , 41 , 42 , 43 ], or a combination of the two [ 11 , 12 , 22 , 28 , 29 , 33 , 37 ]. Traditional FER systems usually involve facial representation and expression classification.…”

“…The majority of facial representations use handcrafted features, such as local binary patterns (LBP) [ 15 ], Gabor features [ 27 ], temporal patterns of oriented edge magnitudes (TPOEMs) [ 38 ], histogram of oriented gradients (HOG) [ 40 ], and bag-of-words (BoW) features [ 31 ] for FER. For expression classification, support vector machines (SVMs) are the most effective and common method, therefore, many studies [ 11 , 15 , 30 , 31 , 32 , 33 , 40 ] used SVM to build their classification algorithm. In the last few years, research on deep learning, especially on convolutional neural networks (CNNs), has made great progress in computer vision, including FER.…”

A Statistical Image Feature‐Based Deep Belief Network for Fire Detection

“…) is an operator that is used to describe the local texture features of an image [41]. It has the characteristics of multiresolution, invariant gray scale, and rotation; therefore, it is a good measure for image feature extraction, as shown in Figure 3.…”