Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Minaee, Shervin; Minaei, Mehdi; Abdolrashidi, AmirAli

doi:10.3390/s21093046

Cited by 360 publications

(168 citation statements)

References 54 publications

(71 reference statements)

Supporting

Mentioning

119

Contrasting

Order By: Relevance

“…However, we note that formulating a competitive FER algorithm is outside the purview of this study. Instead, we adopt a recent, astute FER deep learning model (i.e., the Deep-emotion from [ 17 ]) as the algorithm of our FER layer. Further reasoning in support of this choice is presented later in Section 3 .…”

Section: General Framework Of the Proposed Aole Systemmentioning

confidence: 99%

“…Subsequent outcomes of the localisation network, which regresses the transformation parameters, are transformed to the sampling grid τ ( 𝜃 ) that produces warped data. For this, images in the FER 2013 [ 17 ] dataset are used as training model for expression recognition. This dataset is widely used in studies on facial emotion recognition, such as its use to analyse the psychological condition of patients in [ 21 ].…”

Section: General Framework Of the Proposed Aole Systemmentioning

confidence: 99%

See 1 more Smart Citation

Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 Era

Yan

Iliyasu

et al. 2022

Appl Intell

View full text Add to dashboard Cite

In addition to the almost five million lives lost and millions more than that in hospitalisations, efforts to mitigate the spread of the COVID-19 pandemic, which that has disrupted every aspect of human life deserves the contributions of all and sundry. Education is one of the areas most affected by the COVID-imposed abhorrence to physical (i.e., face-to-face (F2F)) communication. Consequently, schools, colleges, and universities worldwide have been forced to transition to different forms of online and virtual learning. Unlike F2F classes where the instructors could monitor and adjust lessons and content in tandem with the learners’ perceived emotions and engagement, in online learning environments (OLE), such tasks are daunting to undertake. In our modest contribution to ameliorate disruptions to education caused by the pandemic, this study presents an intuitive model to monitor the concentration, understanding, and engagement expected of a productive classroom environment. The proposed apposite OLE (i.e., AOLE) provides an intelligent 3D visualisation of the classroom atmosphere (CA), which could assist instructors adjust and tailor both content and instruction for maximum delivery. Furthermore, individual learner status could be tracked via visualisation of his/her emotion curve at any stage of the lesson or learning cycle. Considering the enormous emotional and psychological toll caused by COVID and the attendant shift to OLE, the emotion curves could be progressively compared through the duration of the learning cycle and the semester to track learners’ performance through to the final examinations. In terms of learning within the CA, our proposed AOLE is assessed within a class of 15 students and three instructors. Correlation of the outcomes reported with those from administered questionnaires validate the potential of our proposed model as a support for learning and counselling during these unprecedentedtimes that we find ourselves.

show abstract

Section: General Framework Of the Proposed Aole Systemmentioning

confidence: 99%

Section: General Framework Of the Proposed Aole Systemmentioning

confidence: 99%

Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 Era

Yan

Iliyasu

et al. 2022

Appl Intell

View full text Add to dashboard Cite

show abstract

“…However, CNNs lack an attention mechanism that can identify the most appropriate parts of an image from which to learn. Spatial Transformer Networks (STN) [58] aim to detect the principal regions that appear on an image and correct spatial variations by transforming the input data, as happens in Deep-Emotion [59]. Deep-Emotion uses an STN architecture to address emotion recognition, emphasizing that these models are appropriate to solve this task.…”

Section: Facial Emotion Recognitionmentioning

confidence: 99%

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Jiménez

Griol

Callejas

et al. 2021

Sensors

View full text Add to dashboard Cite

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

show abstract

“…In [25], the authors presented a deep learning method based on the attentional convolutional network capable of focusing on critical areas of the face and outperforms prior models on various datasets, including FER-2013, CK+, FERG and JAFFE. The suggested scheme encodes shape, appearance and extensive dynamic information.…”

Section: Related Workmentioning

confidence: 99%

Racial Identity-Aware Facial Expression Recognition Using Deep Convolutional Neural Networks

et al. 2021

View full text Add to dashboard Cite

Multi-culture facial expression recognition remains challenging due to cross cultural variations in facial expressions representation, caused by facial structure variations and culture specific facial characteristics. In this research, a joint deep learning approach called racial identity aware deep convolution neural network is developed to recognize the multicultural facial expressions. In the proposed model, a pre-trained racial identity network learns the racial features. Then, the racial identity aware network and racial identity network jointly learn the racial identity aware facial expressions. By enforcing the marginal independence of facial expression and racial identity, the proposed joint learning approach is expected to be purer for the expression and be robust to facial structure and culture specific facial characteristics variations. For the reliability of the proposed joint learning technique, extensive experiments were performed with racial identity features and without racial identity features. Moreover, culture wise facial expression recognition was performed to analyze the effect of inter-culture variations in facial expression representation. A large scale multi-culture dataset is developed by combining the four facial expression datasets including JAFFE, TFEID, CK+ and RaFD. It contains facial expression images of Japanese, Taiwanese, American, Caucasian and Moroccan cultures. We achieved 96% accuracy with racial identity features and 93% accuracy without racial identity features.

show abstract

Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network

Cited by 360 publications

References 54 publications

Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 Era

Framework for identifying and visualising emotional atmosphere in online learning environments in the COVID-19 Era

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning

Racial Identity-Aware Facial Expression Recognition Using Deep Convolutional Neural Networks

Contact Info

Product

Resources

About