Detecting Naturalistic Expressions of Nonbasic Affect Using Physiological Signals

Abstract

Signals from peripheral physiology (e.g., ECG, EMG, and GSR) in conjunction with machine learning techniques can be used for the automatic detection of affective states. The affect detector can be user-independent, where it is expected to generalize to novel users, or user-dependent, where it is tailored to a specific user. Previous studies have reported some success in detecting affect from physiological signals, but much of the work has focused on induced affect or acted expressions instead of contextually constrained spontaneous expressions of affect. This study addresses these issues by developing and evaluating user-independent and user-dependent physiology-based detectors of nonbasic affective states (e.g., boredom, confusion, curiosity) that were trained and validated on naturalistic data collected during interactions between 27 students and AutoTutor, an intelligent tutoring system with conversational dialogues. There is also no consensus on which techniques (i.e., feature selection or classification methods) work best for this type of data. Therefore, this study also evaluates the efficacy of affect detection using a host of feature selection and classification techniques on three physiological signals (ECG, EMG, and GSR) and their combinations. Two feature selection methods and nine classifiers were applied to the problem of recognizing eight affective states (boredom, confusion, curiosity, delight, flow/engagement, surprise, and neutral). The results indicated that the user-independent modeling approach was not feasible; however, a mean kappa score of 0.25 was obtained for user-dependent models that discriminated among the most frequent emotions. The results also indicated that k-nearest neighbor and Linear Bayes Normal Classifier (LBNC) classifiers yielded the best affect detection rates. Single channel ECG, EMG, and GSR and three-channel multimodal models were generally more diagnostic than two-channel models.

Towards affective touch interaction: predicting mobile user emotion from finger strokes

“…Koelstra et al [20] recorded multiple physiological signals such as EOG, HR, GSR, EBR and EEG to detect emotions. In Alzoubi et al [1], three physiological signals, namely EMG (Electromyography), ECG and GSR were recorded and used for detection of user's affective states. Hazlett [13] reported the use of EMG signal to measure positive and negative emotional valence during interactive experience.…”

“…[1]Positive emotion range: [50,105] [2]Negative emotion range: [1,12] [3]Neutral emotion range: [25,36] In order to obtain the other parameters (A's, B's and C's of Eqs. 12, 13 and 14), we used an elaborate assignment based approach.…”

What cognitive and affective states should technology monitor to support learning?

“…progression towards goal) were used to automatically classify 5 levels of self-efficacy with performance of 0.82 R 2 . In [8], facial muscle activity, skin conductance, and electrocardiography signals were used to automatically differentiate curiosity from engagement, confusion, frustration, delight, boredom, and neutral with F1 score of 0.36.…”

“…Further, unlike, the traditional affective computing modalities (face and voice), bodily gesture and body movement encapsulate information about the action tendency of the learner towards coping with or addressing the experienced state and so offer unique insight into subjective experiences [24]. The role of bodily gestures and movement cues may be enhanced in the weDRAW system where learning activities will be designed to involve this channel unlike the sedentary scenarios considered in previous studies [4][5][6] [8].…”

“…[ [8]), a gap still exists in understanding which of these states is of benefit to monitor. This paper is a first step in providing such understanding through discussion of three relevant states: selfefficacy, curiosity, and reflectivity.…”