Automatically Recognizing Facial Indicators of Frustration: A Learning-centric Analysis

Abstract

Affective and cognitive processes form a rich substrate on which learning plays out. Affective states often influence progress on learning tasks, resulting in positive or negative cycles of affect that impact learning outcomes. Developing a detailed account of the occurrence and timing of cognitive-affective states during learning can inform the design of affective tutorial interventions. In order to advance understanding of learning-centered affect, this paper reports on a study to analyze a video corpus of computer-mediated human tutoring using an automated facial expression recognition tool that detects fine-grained facial movements. The results reveal three significant relationships between facial expression, frustration, and learning: 1) Action Unit 2 (outer brow raise) was negatively correlated with learning gain, 2) Action Unit 4 (brow lowering) was positively correlated with frustration, and 3) Action Unit 14 (mouth dimpling) was positively correlated with both frustration and learning gain. Additionally, early prediction models demonstrated that facial actions during the first five minutes were significantly predictive of frustration and learning at the end of the tutoring session. The results represent a step toward a deeper understanding of learning-centered affective states, which will form the foundation for data-driven design of affective tutoring systems.

Exploring Causes of Frustration for Software Developers

“…However, failure to overcome obstacles in these situations can introduce a sense of mounting frustration in developers that can negatively impact learning outcomes [1] and influence retention in a field [2].…”

The Additive Value of Multimodal Features for Predicting Engagement, Frustration, and Learning during Tutoring

“…Open questions from the tutor (QO-TUTOR) with no student brow lowering (NoAU4) were positively predictive of engagement in the model with the combined bimodal feature set (Bimodal Union). Brow lowering has been previously associated with confusion, frustration, and mental effort [5,8,9,14]. Thus, this may highlight moments when the student was not perplexed by a tutor question.…”

EmotionBike: A Study of Provoking Emotions in Cycling Exergames

“…CERT recognition is based on a support vector learning approach of frontal face images and includes pose correction. Grafsgaard et al [16] suggests that facial expression probabilities may vary significantly between individuals. In order to access this variation phenomenon we compared the machine facial expression analysis with the observer-and self-assessment.…”

“…For the CERT assessment of emotion for computer-mediated tutorial dialogues Grafsgaard et al [16] reported the importance to perform a bias adjustment, due to inter-person variation. For this study we do not adjust this bias, since we intend to show the ability of our physical exergame to provoke emotions and show differences.…”