Real-time video based emotion recognition using convolutional neural network and transfer learning

Abstract

Background/Objectives: The deep learning approaches have paved their way to construct various artificial intelligence products and the proposed system uses a convolutional neural network for detecting real-time emotions of mankind. The objective of the study is to develop a real-time application for emotion recognition using convolutional neural network and transfer learning methods. Methods/Statistical analysis: The proposed system considers happy, normal and surprised categories of emotions. The system consists of four major steps: dataset collection, training, validation, and real-time testing. The dataset is comprised of face images containing emotions such as happy, normal and surprised in the form of video frames. The face and mouth regions are detected using the Haar-Based Cascade classifier at 20 frames per second. Findings: The convolutional neural network (CNN) is trained using mouth images and the pre-trained models VGG16 and VGG19 are trained with face images. The trained model is used to detect the emotions in the live webcam video. The experimental results show that the CNN model trained using mouth images gives an accuracy of 85.71% and the pre-trained models trained with face images using transfer learning method achieves an accuracy of 77.78%. The proposed system using CNN outperforms the pre-trained models for recognizing the emotions in real-time video. Novelty/Applications: The proposed system is entirely based on the mouth region video frames and the real-time emotion recognition system is developed. This work can detect the three emotions in an unconstrained laboratory environment.

Application of Transfer Learning and Feature Fusion Algorithms to Improve the Identification and Prediction Efficiency of Premature Ovarian Failure

“…When comparing samples, the source domain sample γ s and the target domain sample γ t can be mapped to the corresponding subspace respectively, and then, the subspace can be aligned by the transformation matrix M ∗ , and the similarity function can be defined [ 13 , 14 ]: where A = X S X S T X t X t T represents the importance of each part of the feature vector in the original space. Sim( γ s , γ t ) compares the similarity between the source domain sample γ s and the target domain sample γ t on the aligned subspace, so the K -nearest neighbor algorithm can be used directly for classification [ 15 ]. In addition, it is also possible to map the source domain sample γ s from X a to the aligned subspace and map the target domain sample γ t to its corresponding subspace through X t and then use the SVM algorithm for classification [ 16 ].…”

Emotion Recognition With Audio, Video, EEG, and EMG: A Dataset and Baseline Approaches

“…Convolution Neural Network (CNN) is commonly applied in areas related to analyzing visual images, such as object detection, image recognition and classification, and facial recognition. CNN contains convolution layers that extract the input's significant features while preserving the relationship between the 2D spatial domain and extracted features, and has been used for emotion recognition [53,55]. We can also convert temporal data into two-dimensions of time-frequency data, then use CNN to find the relationship between the time domain and the frequency/spatial domain to determine the corresponding emotion.…”

“…Pre-trained CNN Models are usually better in retrieving meaningful generic features, especially from images. The VGG neural network (VGG16, VGG19) [53] has been widely used in image classification and extracts the image data features for emotion recognition [54,55]. Even though the VGG neural network is pre-trained for object classification of various objects rather than human images, since the ImageNet contains vast data samples, the convolution filters have been trained to extract the key features of the images of faces.…”

Patient Monitoring Using Emotion Recognition

“…According to Palanivel S. and Sujanaa J. [10] a straightforward preprocessing phase where the RGB photos are changed into grayscale images is first applied to the photographs. The input frames were pre-processed using the Scikit-library.…”

“…As a result, the positions of the face, left and right eyes, nose, and the left and right corners of the mouth can thus be obtained. Using the cascade classifier, an algorithm devised by Paul Viola and Michael Jones that employs machine learning techniques, [10] note that these cascade classifiers are trained with samples containing facial and non-facial images. With the aid of the Haar-based cascade classifier, the mouth and face of each video frame is detected.…”