Automatically Detecting Pain in Video Through Facial Action Units

Abstract

In a clinical setting, pain is reported either through patient self-report or via an observer. Such measures are problematic as they are: 1) subjective, and 2) give no specific timing information. Coding pain as a series of facial action units (AUs) can avoid these issues as it can be used to gain an objective measure of pain on a frame-by-frame basis. Using video data from patients with shoulder injuries, in this paper, we describe an active appearance model (AAM)-based system that can automatically detect the frames in video in which a patient is in pain. This pain data set highlights the many challenges associated with spontaneous emotion detection, particularly that of expression and head movement due to the patient's reaction to pain. In this paper, we show that the AAM can deal with these movements and can achieve significant improvements in both the AU and pain detection performance compared to the current-state-of-the-art approaches which utilize similarity-normalized appearance features only.

Joint Facial Action Unit Detection and Feature Fusion: A Multi-Conditional Learning Approach

“…The majority of the existing works attempt to recognize AUs or certain AU combinations independently [4], [9], [7], [8], [11], [5], [24], [25]. A common limitation of these methods is that they construct independent AU classifiers that ignore the relations among the AUs.…”

“…Based on how the AUspecific classifiers are designed, they can be divided into two main categories: (a) static modeling approaches, where each frame is evaluated independently [4], [9], [7], [8], [11], and (b) temporal modeling approaches, where temporal dynamics are explored within a video sequence [5], [24], [26]. Representatives of the first group commonly apply independent classifiers, e.g., support vector machine (SVM) [4], [9], and Adaboost [7] on the collected features, or use the notion of domain adaptation to develop personalized AU-classifiers [8]. Alternatively, in [11] sparse representations are employed to create a dictionary of facial images with certain AU combinations.…”

“…[28] provides an extensive overview on how vocal, gestural and facial features extracted from both audio and visual modalities can be used to identify particular human behaviors. As we have already seen, the most evident way towards feature fusion is to concatenate the individual modalities and apply a single classifier for the target task [4], [9]. An orthogonal approach is to first train individual classifiers per modality and then fuse the predictions, e.g., by feeding their outputs in another classifier as in [29].…”

“…Others do not exhibit strong cooccurrence patterns (e.g., AU5 in DISFA). Hence, we selected the following subsets of highly correlated AUs: AUs (1,2,4,6,7,12,15,17) for CK+, AUs (1,2,4,6,12,15,17) for DISFA and AUs (4,6,7,9,10,43) for Shoulder-pain. The selected AUs occur jointly in the context of recorded expressions (e.g., pain expression, see [58]).…”

“…Despite all this, most of the existing approaches to AU detection model each AU independently, using either a single type of facial features [7], [8], or combining multiple features by means of naive approach (i.e., simple feature concatenation) [4], [9] or multiple-kernel learning (MKL) [10]. Furthermore, some methods treat different combinations of AUs as new independent classes [11]; however, this is impractical given the number of possible AU combinations.…”