Expressure: Detect Expressions Related to Emotional and Cognitive Activities Using Forehead Textile Pressure Mechanomyography

Abstract

We investigate how pressure-sensitive smart textiles, in the form of a headband, can detect changes in facial expressions that are indicative of emotions and cognitive activities. Specifically, we present the Expressure system that performs surface pressure mechanomyography on the forehead using an array of textile pressure sensors that is not dependent on specific placement or attachment to the skin. Our approach is evaluated in systematic psychological experiments. First, through a mimicking expression experiment with 20 participants, we demonstrate the system’s ability to detect well-defined facial expressions. We achieved accuracies of 0.824 to classify among three eyebrow movements (0.333 chance-level) and 0.381 among seven full-face expressions (0.143 chance-level). A second experiment was conducted with 20 participants to induce cognitive loads with N-back tasks. Statistical analysis has shown significant correlations between the Expressure features on a fine time granularity and the cognitive activity. The results have also shown significant correlations between the Expressure features and the N-back score. From the 10 most facially expressive participants, our approach can predict whether the N-back score is above or below the average with 0.767 accuracy.

Facial Muscle Activity Recognition with Reconfigurable Differential Stethoscope-Microphones

“…In Table 2 , we have summarized key previously published approaches to non-vision based analysis of facial actions according to employed sensors, number of participants (average of 15.25), the total number of samples (average of 1885.25), number of experiment repetitions (average 2), placement of the sensors (typical glasses frame positions), set of expressions (average 6.8), and performance results (a direct comparison is not possible). The number of volunteers (20 with five repetitions) used in [ 17 ] is because, in this work, an additional cognitive-load experiment was designed (minimal 20 participants [ 72 ]), in our research, we are only evaluating the facial muscular movements with sound. It is imperative to highlight that our experimental design was never intended to be a psychology experiment, and it is a hardware sensing feasibility evaluation.…”

“…This will allow for us to go from mimicked expressions in a lab setting to recognizing real emotions under realistic circumstances. We will also investigate the fusion of differential sound information with other sensing modalities in particular with EMG, (simple) EEG, and our textile pressure sensor arrays based mechanomyography [ 17 ].…”

“…Key limitations of EMG are that the signal magnitude tends to be inversely proportional to the age; it is strongly influenced by the person’s weight and has problems with discrimination between adjacent muscles [ 16 ]. Another myography (TPM) modality is evaluated by Zhou in [ 17 ], achieving high accuracy results (82%) on classifying eyebrows’ movements and 38% for the same set of expressions as in [ 12 ] without including contempt. Unlike our approach, the TPM system is mostly limited to detecting local changes of muscle shape, and we see it as a potential candidate for fusion with our sound-based approach in future work.…”

On the Use of Movement-Based Interaction with Smart Textiles for Emotion Regulation

“…Likewise, Zhou et al [ 47 ] explored how pressure-sensitive smart textiles can help monitor people’s emotional states through changes in facial expressions. They proposed the use of textile pressure mapping arrays integrated into a headband to capture the forehead muscle movements.…”

Physiological Sensors Based Emotion Recognition While Experiencing Tactile Enhanced Multimedia

“…These cues are used in associating the emotional state of an individual with an external stimulus. Emotion recognition using speech [ 11 , 12 , 13 ], facial expressions [ 14 , 15 , 16 ] and their fusion [ 17 , 18 ] has been explored. These conventional methods for emotion recognition have limitations such as privacy and camera positioning [ 19 ].…”