A Survey on Multimodal Data Stream Mining for e-Learner’s Emotion Recognition

Nandi, Arijit; Xhafa, Fatos; Subirats, Laia; Fort, Santi

doi:10.1109/coins49042.2020.9191370

Cited by 12 publications

(6 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently in [ 16 ], we have introduced data stream processing approaches for emotion classification, and we also reviewed various emotional states, which are mostly studied for the e-Learning context. Furthermore, we have identified the most appropriate data channels for emotion recognition.…”

Section: Related Workmentioning

confidence: 99%

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Nandi

Xhafa

Subirats

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

In face-to-face and online learning, emotions and emotional intelligence have an influence and play an essential role. Learners’ emotions are crucial for e-learning system because they promote or restrain the learning. Many researchers have investigated the impacts of emotions in enhancing and maximizing e-learning outcomes. Several machine learning and deep learning approaches have also been proposed to achieve this goal. All such approaches are suitable for an offline mode, where the data for emotion classification are stored and can be accessed infinitely. However, these offline mode approaches are inappropriate for real-time emotion classification when the data are coming in a continuous stream and data can be seen to the model at once only. We also need real-time responses according to the emotional state. For this, we propose a real-time emotion classification system (RECS)-based Logistic Regression (LR) trained in an online fashion using the Stochastic Gradient Descent (SGD) algorithm. The proposed RECS is capable of classifying emotions in real-time by training the model in an online fashion using an EEG signal stream. To validate the performance of RECS, we have used the DEAP data set, which is the most widely used benchmark data set for emotion classification. The results show that the proposed approach can effectively classify emotions in real-time from the EEG data stream, which achieved a better accuracy and F1-score than other offline and online approaches. The developed real-time emotion classification system is analyzed in an e-learning context scenario.

show abstract

Section: Related Workmentioning

confidence: 99%

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Nandi

Xhafa

Subirats

et al. 2021

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…These works examined the application of EDM/LA in multimodal educational data, but which barely touched on data fusion, focusing instead on complex learning tasks (Blikstein & Worsley, 2016), the study of LA architectures (Shankar et al, 2018), and the study of learning environments where multimodal LA is usually applied (Ochoa, 2017). There are also a few review papers more focused in the specific application of data fusion in EDM/LA (Dewan et al, 2019; Han et al, 2020; Nandi et al, 2020). However, they only focused on some specific aspects, including emotion recognition (Nandi et al, 2020), engagement detection (Dewan et al, 2019), or sentiment analysis (Han et al, 2020).…”

Section: Introductionmentioning

confidence: 99%

“…There are also a few review papers more focused in the specific application of data fusion in EDM/LA (Dewan et al, 2019; Han et al, 2020; Nandi et al, 2020). However, they only focused on some specific aspects, including emotion recognition (Nandi et al, 2020), engagement detection (Dewan et al, 2019), or sentiment analysis (Han et al, 2020). Finally, the survey that is most closely related with our current review is from Mu et al (2020), which focused only on LA, without examining EDM bibliography.…”

Section: Introductionmentioning

confidence: 99%

A review on data fusion in multimodal learning analytics and educational data mining

Chango

Lara²,

Cerezo

et al. 2022

WIREs Data Min & Knowl

View full text Add to dashboard Cite

The new educational models such as smart learning environments use of digital and context-aware devices to facilitate the learning process. In this new educational scenario, a huge quantity of multimodal students' data from a variety of different sources can be captured, fused, and analyze. It offers to researchers and educators a unique opportunity of being able to discover new knowledge to better understand the learning process and to intervene if necessary. However, it is necessary to apply correctly data fusion approaches and techniques in order to combine various sources of multimodal learning analytics (MLA). These sources or modalities in MLA include audio, video, electrodermal activity data, eye-tracking, user logs, and click-stream data, but also learning artifacts and more natural human signals such as gestures, gaze, speech, or writing. This survey introduces data fusion in learning analytics (LA) and educational data mining (EDM) and how these data fusion techniques have been applied in smart learning. It shows the current state of the art by reviewing the main publications, the main type of fused educational data, and the data fusion approaches and techniques used in EDM/LA, as well as the main open problems, trends, and challenges in this specific research area.

show abstract

“…Human emotions are recognizable using various measures such as vocal, visual, or physiological signals. 4 It should be noted that visual and vocal measures are maskable, which means the subjects can purposely mask their actual emotions, which may therefore result in false emotions classification 4 . In contrast, physiological signals are more reliable, robust, and objective to emotional state recognition of the subject 5 .…”

Section: Introductionmentioning

confidence: 99%

Reward-Penalty Weighted Ensemble for Emotion State Classification from Multi-Modal Data Streams

Nandi

Xhafa

Subirats

et al. 2022

Int. J. Neur. Syst.

View full text Add to dashboard Cite

Researchers have shown the limitations of using the single-modal data stream for emotion classification. Multi-modal data streams are therefore deemed necessary to improve the accuracy and performance of online emotion classifiers. An online decision ensemble is a widely used approach to classify emotions in real-time using multi-modal data streams. There is a plethora of online ensemble approaches; these approaches use a fixed parameter ([Formula: see text]) to adjust the weights of each classifier (called penalty) in case of wrong classification and no reward for a good performing classifier. Also, the performance of the ensemble depends on the [Formula: see text], which is set using trial and error. This paper presents a new Reward-Penalty-based Weighted Ensemble (RPWE) for real-time multi-modal emotion classification using multi-modal physiological data streams. The proposed RPWE is thoroughly tested using two prevalent benchmark data sets, DEAP and AMIGOS. The first experiment confirms the impact of the base stream classifier with RPWE for emotion classification in real-time. The RPWE is compared with different popular and widely used online ensemble approaches using multi-modal data streams in the second experiment. The average balanced accuracy, F1-score results showed the usefulness and robustness of RPWE in emotion classification in real-time from the multi-modal data stream.

show abstract

A Survey on Multimodal Data Stream Mining for e-Learner’s Emotion Recognition

Cited by 12 publications

References 36 publications

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

Real-Time Emotion Classification Using EEG Data Stream in E-Learning Contexts

A review on data fusion in multimodal learning analytics and educational data mining

Reward-Penalty Weighted Ensemble for Emotion State Classification from Multi-Modal Data Streams

Contact Info

Product

Resources

About