The Additive Value of Multimodal Features for Predicting Engagement, Frustration, and Learning during Tutoring

Abstract

Detecting learning-centered affective states is difficult, yet crucial for adapting most effectively to users. Within tutoring in particular, the combined context of student task actions and tutorial dialogue shape the student's affective experience. As we move toward detecting affect, we may also supplement the task and dialogue streams with rich sensor data. In a study of introductory computer programming tutoring, human tutors communicated with students through a text-based interface. Automated approaches were leveraged to annotate dialogue, task actions, facial movements, postural positions, and hand-to-face gestures. These dialogue, nonverbal behavior, and task action input streams were then used to predict retrospective student self-reports of engagement and frustration, as well as pretest/posttest learning gains. The results show that the combined set of multimodal features is most predictive, indicating an additive effect. Additionally, the findings demonstrate that the role of nonverbal behavior may depend on the dialogue and task context in which it occurs. This line of research identifies contextual and behavioral cues that may be leveraged in future adaptive multimodal systems.

Detecting and Addressing Frustration in a Serious Game for Military Training

“…In fact, the posture-based detectors performed only slightly better than chance, and in the case of some algorithms and D'Mello and Graesser 2009;Grafsgaard et al 2014). There are several possible explanations for why the posture-based predictors were not more effective.…”

“…Several research labs have investigated sensor-based affect recognition in learning environments over the past decade, including work with facial expression , eye tracking (Jaques et al 2014), and posture Grafsgaard et al 2014). In this work, we focus primarily on posture sensor-based models of affect recognition.…”

“…In this work, we focus primarily on posture sensor-based models of affect recognition. To date, posture-based affect recognition models have been induced with data from pressure-sensitive chairs D'Mello and Graesser 2010), as well as motion sensors, such as Microsoft Kinect (Grafsgaard et al 2014). These two data streams, drawing from distinct types of sensors, are superficially different, but can be distilled into analogous predictor features that have similar relationships with affective states such as engagement, boredom, frustration, and confusion.…”

“…Results indicated that posture-based features were significantly predictive of self-reported excitement during learning, although they were not part of the best-performing models for other emotional states. Grafsgaard et al (2014) have investigated postured-based affect prediction using Microsoft Kinect sensors with an intelligent tutoring system for introductory programming. Posture features were distilled from depth image recordings by tracking the distance between the depth camera and the participant's head, upper torso, and lower torso.…”

“…The feature set was designed to emulate the posture-related features that had previously been utilized in the aforementioned posture-based affect detection work (D'Mello and Graesser 2009;Grafsgaard et al 2014). For each of three retained skeletal vertices tracked by the Kinect (head, center_shoulder, and top_skull), we calculated 18 features based on multiple time window durations.…”