Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate

Monkaresi, Hamed; Bosch, Nigel; Calvo, Rafael A.; D’Mello, Sidney K.

doi:10.1109/taffc.2016.2515084

Cited by 256 publications

(159 citation statements)

References 62 publications

Supporting

Mentioning

158

Contrasting

Order By: Relevance

“…[8]. Automated measures are based on the timing and participants' physiological information produced by heart, brain and skin [19,23]. Another kind of automatic engagement recognition is based on computer vision, which provides an automatic estimation of engagement by analysing cues from the face and gestures [5,7,19,23].…”

Section: Methodsmentioning

confidence: 99%

Evaluating engagement in digital narratives from facial data

Huan

2017

Proceedings of the 19th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

Engagement researchers indicate that the engagement level of people in a narrative has an influence on people's subsequent storyrelated attitudes and beliefs [6], which helps psychologists understand people's social behaviours and personal experience. With the arrival of multimedia, the digital narrative combines multimedia features (e.g. varying images, music and voiceover) with traditional storytelling. Research on digital narratives has been widely used in helping students gain problem-solving and presentation skills [4] as well as supporting child psychologists investigating children's social understanding such as family/peer relationships [18] through completing their digital narratives. However, there is little study on the effect of multimedia features in digital narratives on the engagement level of people.This research focuses on measuring the levels of engagement of people in digital narratives and specifically on understanding the media effect of digital narratives on people's engagement levels. Measurement tools are developed and validated through analyses of facial data from different age groups (children and young adults) in watching stories with different media features of digital narratives. Data sources used in this research include a questionnaire with Smileyometer scale [10] and the observation of each participant's facial behaviours.

show abstract

Section: Methodsmentioning

confidence: 99%

Evaluating engagement in digital narratives from facial data

Huan

2017

Proceedings of the 19th ACM International Conference on Multimodal Interaction

View full text Add to dashboard Cite

show abstract

“…Non-intrusive visual observation and estimation of affective parameters is commonly using recorded video (RGB) signal, for example, to estimate student engagement from facial expressions [3,17], to estimate mood of children during one-to-one tutoring by using facial analysis [2], and to estimate driver's vigilance from his head pose [18]. A survey of automatic affect detection methods [4] identified various types of signals (video, EKG, EMG...) used in affect analysis.…”

Section: Automated Measurement Of Affective Parametersmentioning

confidence: 99%

“…Monkaresi et al [17] use combination of geometrical facial features (detected by Kinect sensor), texture description features (local binary patterns), and physiological features (heart rate) to estimate two-level engagement of students. Whitehill et al [3] use computer vision methods to register faces and extract Box Filter features (Haar wavelets) and then train binary classifiers to estimate four states of engagement.…”

Section: Automated Measurement Of Affective Parametersmentioning

confidence: 99%

“…Two recent studies by Whitehill et al [3] and Monkaresi et al [17] propose systems for automatic estimation of student engagement from face and other features in the context of computerized learning. The experiments involve cameras which observe subjects during playing cognitive skills training game on iPad [3] and during writing an essay on a computer.…”

Section: Comparison To State-of-the-artmentioning

confidence: 99%

“…The first study [3] utilized human labelers to estimate how engaged the subject appear to be on a four-level scale from either 60-or 10-s video clips, while our time resolution of annotations and predictions was 1 s. Their best model, SVM classifier with Gabor features, achieved subject-independent engagement recognition accuracy of 0.729 for detecting each level from all the other levels. The second study [17] uses concurrent self-reports on a two-level scale every 2 min, and retrospective self-reports after observing 10-s clips as a reference data. They achieved an accuracy of 0.758 for a person-independent two-level engagement detection.…”

Section: Comparison To State-of-the-artmentioning

confidence: 99%

See 2 more Smart Citations

Predicting students’ attention in the classroom from Kinect facial and body features

Zaletelj

Košir

2017

J Image Video Proc.

123

View full text Add to dashboard Cite

This paper proposes a novel approach to automatic estimation of attention of students during lectures in the classroom. The approach uses 2D and 3D data obtained by the Kinect One sensor to build a feature set characterizing both facial and body properties of a student, including gaze point and body posture. Machine learning algorithms are used to train classifiers which estimate time-varying attention levels of individual students. Human observers' estimation of attention level is used as a reference. The comparison of attention prediction accuracy of seven classifiers is done on a data set comprising 18 subjects. Our best person-independent three-level attention classifier achieved moderate accuracy of 0.753, comparable to results of other studies in the field of student engagement. The results indicate that Kinect-based attention monitoring system is able to predict both students' attention over time as well as average attention levels and could be applied as a tool for non-intrusive automated analytics of the learning process.

show abstract

Engagement estimation of the elderly from wild multiparty human–robot interaction

Zhang

Zheng

Thalmann

2022

Computer Animation & Virtual

View full text Add to dashboard Cite

The use of social robots in healthcare systems or nursing homes to assist the elderly and their caregivers will be becoming common, where robots' understanding of engagement of the elderly is important. Traditional engagement estimation (EE) often requires expert involvement in a controlled dyadic interaction environment. In this article, we propose a supervised machine learning method to estimate the engagement state of the elderly in a multiparty human-robot interaction (HRI) scenario from the real-world video recording as input. The method is built upon the basic concept of engagement in geriatric psychiatry and HRI video representations. It adapts pretrained models to extract behavior, affective, and visual signals to form the multi-modal features. These features are then fed into a neural network made of a self-attention mechanism and average pooling for individual learning, a graph attention network for group learning and a fully connected layer to estimate the engagement. We tested the proposed method using 43 wild multiparty elderly robot interaction (ERI) videos. The experimental results show that our method is capable of detecting the key participants and estimating the engagement state of the elderly effectively. Also our study demonstrates the signals from side-participants in the main interaction group considerably contribute to the EE of the elderly in the multiparty ERI.

show abstract

Automated Detection of Engagement Using Video-Based Estimation of Facial Expressions and Heart Rate

Cited by 256 publications

References 62 publications

Evaluating engagement in digital narratives from facial data

Evaluating engagement in digital narratives from facial data

Predicting students’ attention in the classroom from Kinect facial and body features

Engagement estimation of the elderly from wild multiparty human–robot interaction

Contact Info

Product

Resources

About