Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information

Xu, Ruyi; Chen, Jingying; Han, Jiaxu; Tan, Lei; Xu, Luhui

doi:10.1007/s00607-019-00722-7

Cited by 26 publications

(15 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“… Xu et al (2020) proposed a multitasking learning system using a cascaded CNN, and the objectives tend towards incorporating students attentiveness and students emotion recognition and intensity estimation into an intelligent class system. The first module of the cascaded network handled the preprocessing stages that involve face detection, face alignment and head pose estimation through which attentiveness is determined.…”

Section: Related Workmentioning

confidence: 99%

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

Ekundayo¹,

Viriri²

2021

PeerJ Computer Science

View full text Add to dashboard Cite

Facial Expression Recognition (FER) has gained considerable attention in affective computing due to its vast area of applications. Diverse approaches and methods have been considered for a robust FER in the field, but only a few works considered the intensity of emotion embedded in the expression. Even the available studies on expression intensity estimation successfully assigned a nominal/regression value or classified emotion in a range of intervals. Most of the available works on facial expression intensity estimation successfully present only the emotion intensity estimation. At the same time, others proposed methods that predict emotion and its intensity in different channels. These multiclass approaches and extensions do not conform to man heuristic manner of recognising emotion and its intensity estimation. This work presents a Multilabel Convolution Neural Network (ML-CNN)-based model, which could simultaneously recognise emotion and provide ordinal metrics as the intensity estimation of the emotion. The proposed ML-CNN is enhanced with the aggregation of Binary Cross-Entropy (BCE) loss and Island Loss (IL) functions to minimise intraclass and interclass variations. Also, ML-CNN model is pre-trained with Visual Geometric Group (VGG-16) to control overfitting. In the experiments conducted on Binghampton University 3D Facial Expression (BU-3DFE) and Cohn Kanade extension (CK+) datasets, we evaluate ML-CNN’s performance based on accuracy and loss. We also carried out a comparative study of our model with some popularly used multilabel algorithms using standard multilabel metrics. ML-CNN model simultaneously predicts emotion and intensity estimation using ordinal metrics. The model also shows appreciable and superior performance over four standard multilabel algorithms: Chain Classifier (CC), distinct Random K label set (RAKEL), Multilabel K Nearest Neighbour (MLKNN) and Multilabel ARAM (MLARAM).

show abstract

Section: Related Workmentioning

confidence: 99%

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

Ekundayo¹,

Viriri²

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Big data analysis has rendered a chance of analyzing the psychology of a leaner. The authors in Reference 28 propose a novel emotion‐sensitive method to determine the learners' interest based on the position of the head and facial emotions. The major challenge of online learning is the retention rate.…”

Section: Deep Learning For Smart Educationmentioning

confidence: 99%

A review on deep learning for future smart cities

Bhattacharya

Somayaji

Gadekallu

et al. 2020

Internet Technology Letters

View full text Add to dashboard Cite

The advancements in Information and Communication Technologies (ICT) made the concept of Smart Cities into reality. In a smart city several Internet of Things (IoT) sensors are deployed across several locations to collect the data about traffic, drainage, mobility of citizens etc. and the insights gained from these data are used to manage resources, assets etc. effectively. Deep Learning has been used extensively on the data generated by IoT sensors in a smart city by several researchers. In this article an attempt is made to survey several state-of-the art on usage of Deep Learning on Smart City data. Several future research directions are suggested at the end of the article.

show abstract

“…Como trabajos unimodales, se pueden citar a: (van der Haar, 2019) donde se captura el rostro a través de videos (Zatarain Cabada, Barron Estrada, Halor-Hernandez, & Reyes-García, 2014), donde se captura la imagen del rostro (Wei-Long & Bao-Liang, 2015). Como trabajos multimodales, se pueden citar como ejemplos a: (Xu, Chen, Han, Tan, & Xu, 2019) donde agregan a la captura del rostro la posición de la cabeza, (Alepis & Virvou, 2006) donde utilizan cuestionarios, tecleo y micrófono, (Salmeron-Majadas, Santos, & Boticario, 2014) donde utilizan movimientos del mouse y teclado y, (Calot, Ierache, & Hasperué, 2019) donde se realiza la captura del tecleo. En la utilización de estados afectivos utilizando enfoques multimodales, existen distintos desafíos, como ser: decidir qué modalidades combinar, recopilar datos de entrenamiento, manejar datos faltantes, manejar diferentes tasas de muestreo e interdependencia de modalidades al construir modelos, decidir cómo fusionar datos de diferentes modalidades, decidir cómo evaluar los estados emocionales.…”

Section: Métodosunclassified

Framework multimodal emocional en el contexto de ambientes dinámicos

Ierache

Sattolo²,

Chapperón³

2020

risti

View full text Add to dashboard Cite

En este trabajo, se presentan los antecedentes entorno a instrumentos, métodos y modelos en la temática de computación afectiva. En particular se exploran los modelos emocionales deductivos y categóricos. Se describe el Framework multimodal (MM) emocional desarrollado y su arquitectura conceptual. Finalmente se exhiben y se discuten los resultados generales de las experimentaciones realizadas, en particular las que se realizaron con un simulador de vuelo. El resultado de las experiencias permitió verificar la correlación entre los registros SAM y los valores del BCI (excitación y relajación, EEG) registrados por el Framework para el caso de prueba presentado.

show abstract

Towards emotion-sensitive learning cognitive state analysis of big data in education: deep learning-based facial expression analysis using ordinal information

Cited by 26 publications

References 22 publications

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

Multilabel convolution neural network for facial expression recognition and ordinal intensity estimation

A review on deep learning for future smart cities

Framework multimodal emocional en el contexto de ambientes dinámicos

Contact Info

Product

Resources

About