Automatic Coding of Facial Expressions of Pain: Are We There Yet?

Abstract

Introduction. The experience of pain is regularly accompanied by facial expressions. The gold standard for analyzing these facial expressions is the Facial Action Coding System (FACS), which provides so-called action units (AUs) as parametrical indicators of facial muscular activity. Particular combinations of AUs have appeared to be pain-indicative. The manual coding of AUs is, however, too time- and labor-intensive in clinical practice. New developments in automatic facial expression analysis have promised to enable automatic detection of AUs, which might be used for pain detection. Objective. Our aim is to compare manual with automatic AU coding of facial expressions of pain. Methods. FaceReader7 was used for automatic AU detection. We compared the performance of FaceReader7 using videos of 40 participants (20 younger with a mean age of 25.7 years and 20 older with a mean age of 52.1 years) undergoing experimentally induced heat pain to manually coded AUs as gold standard labeling. Percentages of correctly and falsely classified AUs were calculated, and we computed as indicators of congruency, “sensitivity/recall,” “precision,” and “overall agreement (F1).” Results. The automatic coding of AUs only showed poor to moderate outcomes regarding sensitivity/recall, precision, and F1. The congruency was better for younger compared to older faces and was better for pain-indicative AUs compared to other AUs. Conclusion. At the moment, automatic analyses of genuine facial expressions of pain may qualify at best as semiautomatic systems, which require further validation by human observers before they can be used to validly assess facial expressions of pain.

Classification of elderly pain severity from automated video clip facial action unit analysis: A study from a Thai data repository

“…The best model is SVM for average activities of AU1, 2, 4, 7, 9, 10, 12, 20, 25, and 45, and gender, which had the best accuracy, at 58%. This result was expected and is consistent with previous studies (Lautenbacher et al, 2022 ). However, this real-world study provided insight into the interpretation and expression issues that continue to pose challenges for automated facial pain ratings.…”

“…According to Lautenbacher et al, the currently available automated facial recognition algorithms, that is, Facereader7 © , OpenFace © , and Affdex SDK © , have comparable outcomes with a lack of robustness (0.3-0.4%) and inconsistency between manual and automatic AU detection. In addition, the discrepancy between laboratory-based eliciting of responses and automatic AU coding significantly increases when the facial expression occurs during spontaneous (emotional) eliciting (Lautenbacher et al, 2022).…”

Classification of elderly pain severity from automated video clip facial action units analysis: A study from a Thai data repository

“…The best model is SVM for average activities of AU1, 2, 4, 7, 9, 10, 12, 20, 25, and 45, and gender, which had the best accuracy, at 58%. This result was expected and is consistent with previous studies [13]. However, this real-world study provided insight into the interpretation and expression issues that continue to pose challenges for automated facial pain ratings.…”

“…According to Lautenbacher et al, the currently available automated facial recognition algorithms, that is, Facereader7©, OpenFace©, and Affdex SDK © have comparable outcomes with a lack of robustness (0.3%-0.4%) and inconsistency between manual and automatic AU detection. Additionally, the discrepancy between laboratory-based eliciting of responses and automatic AU coding signi cantly increases when the facial expression occurs during spontaneous (emotional) eliciting [13].…”

Assessing Pain Research: A Narrative Review of Emerging Pain Methods, Their Technosocial Implications, and Opportunities for Multidisciplinary Approaches

“…Sleep sensors can take many forms ( 237 , 238 ), including portable polysomnography sensor systems embedded into head caps, smart apps that capture movements or noise, infrared cameras that capture body position, or pressure platforms put under or into mattresses, pillows, or cushions ( 239 – 244 ). Finally, facial expression monitoring of pain intensity is also an active area of inquiry ( 245 – 249 ), part of a growing subarea of IoT and digital patient monitoring ( 237 , 250 ) that utilizes multi-camera systems and video recordings to measure “emitted” facial expressions in real-time or retrospectively ( 251 – 253 ). These measures have been primarily based on automatic recognition of facial action coding system units (FACS) ( 254 ) and associated sentiment ( 255 ), although some analyze audio and visual signals together ( 256 – 259 ).…”