Dual integrated convolutional neural network for real-time facial expression recognition in the wild

Saurav, Sumeet; Gidde, Prashant Sadashiv; Saini, Ramesh Kumar; Singh, Sanjay

doi:10.1007/s00371-021-02069-7

Cited by 35 publications

(30 citation statements)

References 43 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary interaction between teachers and students in classroom scenarios is in the form of discourse, and judging emotions through the lesson is also one of the most common ways. Therefore, this chapter studies speech emotion recognition using the speech data generated from teacher-student communication in classrooms [ 27 ]. At the same time, this chapter also references the effect of classroom speech as a single modality for the later multimodal fusion study and an optimal emotion recognition model structure for speech stream branching.…”

Section: Design Of Classroom Emotion Recognition Model Based On Convo...mentioning

confidence: 99%

A Classroom Emotion Recognition Model Based on a Convolutional Neural Network Speech Emotion Algorithm

Yuan

2022

Occupational Therapy International

View full text Add to dashboard Cite

In this paper, we construct a model of convolutional neural network speech emotion algorithm, analyze the classroom identified by the neural network with a certain degree of confidence together with the school used in the dataset, find the characteristics and rules of teachers’ control of classroom emotion nowadays using big data, find the parts of classroom emotion, and design a classroom emotion recognition model based on convolutional neural network speech emotion algorithm according to these characteristics. This paper will investigate the factors and patterns of teachers’ emotional control in the classroom. In this paper, the existing neural network is adapted and improved, and some preprocessing is performed on the current dataset to train the network. The network used in this paper is a combination of convolutional neural network (CNN) and recurrent neural network (RNN), which takes advantage of both CNN for feature extraction and RNN for memory capability in the sequence model. This network has a good effect on both object labeling and speech recognition. For the problem of extracting emotion features of whole-sentence speech, we propose an attention mechanism-based emotion recognition algorithm for variable-length speech and design a spatiotemporal attention module for the speech emotion algorithm and a convolutional channel attention module for the CNN network to reduce the contribution of the spatiotemporal data of the speech emotion algorithm and the unimportant parts of the CNN convolutional channel feature data in the subsequent recognition by the attention mechanism. In turn, the weight of core key data and features is increased to improve the model recognition accuracy.

show abstract

Section: Design Of Classroom Emotion Recognition Model Based On Convo...mentioning

confidence: 99%

A Classroom Emotion Recognition Model Based on a Convolutional Neural Network Speech Emotion Algorithm

Yuan

2022

Occupational Therapy International

View full text Add to dashboard Cite

show abstract

“…Saurav et al [20] constructed two custom CNNs using gradient weighted class activation mapping (Grad-CAM). They also proposed a dual-integrated CNN, which combines the features extracted from two CNNs and uses them for facial emotion recognition.…”

Section: A Facial Emotion Recognition Using a Lightweight Cnnmentioning

confidence: 99%

“…However, the previous works [14], [21] occupied a large amount of memory space and had a problem of high computational complexity. In earlier works [15], [20], the CNN was used with smaller parameters, but because two CNNs were used for inference, they still occupied a large amount of memory space and remain unsuitable for real-time processing. In other cases [16]- [19], CNN was used with small parameters and low computational complexity, but their facial emotion recognition accuracy is very poor.…”

Section: A Facial Emotion Recognition Using a Lightweight Cnnmentioning

confidence: 99%

A Resource Efficient Integer-Arithmetic-Only FPGA-Based CNN Accelerator for Real-Time Facial Emotion Recognition

2021

View full text Add to dashboard Cite

Recently, much research has been conducted on the recognition of facial emotion using convolutional neural networks (CNNs), which show excellent performance in computer vision. To obtain a high classification accuracy, a CNN architecture with many parameters and high computational complexity is required. However, this is not suitable for embedded systems where hardware resources are limited. In this paper, we present a lightweight CNN architecture optimized for embedded systems. The proposed CNN architecture has a small memory footprint and low computational complexity. Furthermore, a novel hardware-friendly quantization method that uses only integer-arithmetic is proposed. The proposed hardwarefriendly quantization method maps the scale factors to power-of-two terms and replaces multiplication and division operations using scale factors with shift operations. To improve the generalization and classification performance of the CNN, we create the FERPlus-A dataset. This is a new training dataset created using a variety of image processing algorithms. After training with FERPlus-A, quantization has been performed. The size of a quantized CNN parameter is about 0.39 MB, and the number of operations is about 28 M integer operations (IOPs). By evaluating the performance of the quantized CNN that uses only integer-arithmetic on the FERPlus test dataset, the classification accuracy is approximately 86.58%. It achieved higher accuracy than other lightweight CNNs in prior studies. The proposed CNN architecture that uses only integerarithmetic is implemented on the Xilinx ZC706 SoC platform for real-time facial emotion recognition by applying parallelism strategies and efficient data caching strategies. The FPGA-based CNN accelerator implemented for real-time facial emotion recognition achieves about 10 frames per second (FPS) at 250 MHz and consumes 2.3 W.

show abstract

“…Li et al [28] used reinforcement learning for selection of relevant images for expression classification. Saurav et al [44] proposed Dual Integrated Convolution Neural Network (DICNN) model for recognizing 'in the wild' facial expressions on embedded platform. Jeen et al [25] utilized subband selective multilevel stationary wavelet gradient transform features for recognizing facial expressions.…”

Section: Related Workmentioning

confidence: 99%

Modified chess patterns: handcrafted feature descriptors for facial expression recognition

Kartheek

Prasad

Bhukya

2021

Complex Intell. Syst.

View full text Add to dashboard Cite

Facial expressions are predominantly important in the social interaction as they convey the personal emotions of an individual. The main task in Facial Expression Recognition (FER) systems is to develop feature descriptors that could effectively classify the facial expressions into various categories. In this work, towards extracting distinctive features, Radial Cross Pattern (RCP), Chess Symmetric Pattern (CSP) and Radial Cross Symmetric Pattern (RCSP) feature descriptors have been proposed and are implemented in a 5 $$\times $$ × 5 overlapping neighborhood to overcome some of the limitations of the existing methods such as Chess Pattern (CP), Local Gradient Coding (LGC) and its variants. In a 5 $$\times $$ × 5 neighborhood, the 24 pixels surrounding the center pixel are arranged into two groups, namely Radial Cross Pattern (RCP), which extracts two feature values by comparing 16 pixels with the center pixel and Chess Symmetric Pattern (CSP) extracts one feature value from the remaining 8 pixels. The experiments are conducted using RCP and CSP independently and also with their fusion RCSP using different weights, on a variety of facial expression datasets to demonstrate the efficiency of the proposed methods. The results obtained from the experimental analysis demonstrate the efficiency of the proposed methods.

show abstract

Dual integrated convolutional neural network for real-time facial expression recognition in the wild

Cited by 35 publications

References 43 publications

A Classroom Emotion Recognition Model Based on a Convolutional Neural Network Speech Emotion Algorithm

A Classroom Emotion Recognition Model Based on a Convolutional Neural Network Speech Emotion Algorithm

A Resource Efficient Integer-Arithmetic-Only FPGA-Based CNN Accelerator for Real-Time Facial Emotion Recognition

Modified chess patterns: handcrafted feature descriptors for facial expression recognition

Contact Info

Product

Resources

About