Facial Expression Recognition Using a Hybrid CNN–SIFT Aggregator

Connie, Tee; Al-Shabi, Mundher; Cheah, Wooi Ping; Goh, Michael Kah Ong

doi:10.1007/978-3-319-69456-6_12

Cited by 81 publications

(40 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…With only a few exceptions [1,32,33], most of the recent works on facial expression recognition are based on deep learning [2,9,10,13,14,17,21,22,24,23,26,28,38,39,40]. Some of these recent works [14,17,21,38,39] proposed to train an ensemble of convolutional neural networks for improved performance, while others [6,16] combined deep features with handcrafted features such as SIFT [25] or Histograms of Oriented Gradients (HOG) [8]. While most works studied facial expression recognition from static images, some works tackled facial expression recognition in video [13,16].…”

Section: Related Workmentioning

confidence: 99%

“…Closer to our work are methods [6,16] that combine deep and handcrafted features or that employ local learning [15] for facial expression recognition. While Ionescu et al [15] used local learning to improve the performance of a handcrafted model, we show that local learning can also improve performance when deep features are used in combination with handcrafted features.…”

Section: Related Workmentioning

confidence: 99%

“…Works that combine deep and handcrafted features usually employ a single CNN model and various handcrafted features, e.g. Connie et al [6] employed SIFT and dense SIFT, while Kaya et al [16] employed SIFT, HOG and Local Gabor Binary Patterns (LGBP). On the other hand, we employ a single type of handcrafted features and we include various CNN architectures in the combination.…”

Section: Related Workmentioning

confidence: 99%

“…Another important difference from works [6,16] that combine deep and handcrafted features is that we employ local learning in the training stage. With these key changes, the empirical results indicate that our approach achieves better performance than the approach of Connie et al [6]. We do not compare with Kaya et al [16], since their approach is designed to work on video.…”

Section: Related Workmentioning

confidence: 99%

“…Table 1 includes the results of our combined models, one based on global SVM and another based on local SVM, on Table 1: Results on the FER 2013 [11], the FER+ [2] and the AffectNet [27] data sets. Our combination based on pre-trained, fine-tuned and handcrafted models, with and without data augmentation (aug.), are compared with several state-of-the-art approaches [2,6,14,15,17,21,23,27,34,39,40], which are listed in temporal order. The best result on each data set is highlighted in bold.…”

Section: Implementation Detailsmentioning

confidence: 99%

See 4 more Smart Citations

Local Learning With Deep and Handcrafted Features for Facial Expression Recognition

2019

View full text Add to dashboard Cite

We present an approach that combines automatic features learned by convolutional neural networks (CNN) and handcrafted features computed by the bag-of-visual-words (BOVW) model in order to achieve state-of-the-art results in facial expression recognition. To obtain automatic features, we experiment with multiple CNN architectures, pretrained models and training procedures, e.g. Dense-Sparse-Dense. After fusing the two types of features, we employ a local learning framework to predict the class label for each test image. The local learning framework is based on three steps. First, a k-nearest neighbors model is applied in order to select the nearest training samples for an input test image. Second, a one-versus-all Support Vector Machines (SVM) classifier is trained on the selected training samples. Finally, the SVM classifier is used to predict the class label only for the test image it was trained for. Although we have used local learning in combination with handcrafted features in our previous work, to the best of our knowledge, local learning has never been employed in combination with deep features. The experiments on the 2013 Facial Expression Recognition (FER) Challenge data set, the FER+ data set and the AffectNet data set demonstrate that our approach achieves state-of-the-art results. With a top accuracy of 75.42% on FER 2013, 87.76% on the FER+, 59.58% on AffectNet 8-way classification and 63.31% on AffectNet 7-way classification, we surpass the state-of-theart methods by more than 1% on all data sets.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Implementation Detailsmentioning

confidence: 99%

See 3 more Smart Citations

Local Learning With Deep and Handcrafted Features for Facial Expression Recognition

2019

View full text Add to dashboard Cite

show abstract

Fused deep learning based Facial Expression Recognition of students in online learning mode

Sumalakshmi

Vasuki²

2022

Concurrency and Computation

View full text Add to dashboard Cite

In this research work, Facial Expression Recognition (FER) is used in the analysis of facial expressions during the online learning sessions in the prevailing pandemic situation. An integrated geometric and appearance feature extraction is presented for the FER of the students participating in the online classes. The integrated features provided a low-dimensional significant feature area for better facial data representation. Feasible Weighted Squirrel Search Optimization (FW-SSO) algorithm is applied for selecting the optimal features due to its efficient exploration of the search space and enhancement of the dynamic search. The output of the FW-SSO algorithm is used for tuning the autoencoder. Autoencoder is used for combining the G&A features, for feature optimization process. Classification is done by using Long Short-Term Memory (LSTM) network with Attention Mechanism (ALSTM), as it is highly efficient in capturing the long-term dependency of the facial landmarks in the image/video sequences.The proposed fused deep learning method focuses on the fusion of the G&A features for high discrimination. Experimental analysis using FER-2013 and LIRIS datasets demonstrated that the proposed method achieved maximum accuracy of 85.96% than the existing architectures and maximum accuracy of 88.24% than the VGGNet-CNN architecture.

show abstract