Social signal processing for pain monitoring using a hidden conditional random field

Abstract

Automatic pain monitoring has the potential to greatly improve patient diagnosis and outcomes by providing a continuous objective measure. One of the most promising methods is to do this via automatically detecting facial expressions. However, current approaches have failed due to their inability to: 1) integrate the rigid and non-rigid head motion into a single feature representation, and 2) incorporate the salient temporal patterns into the classification stage. In this paper, we tackle the first problem by developing a "histogram of facial action units" representation using Active Appearance Model (AAM) face features, and then utilize a Hidden Conditional Random Field (HCRF) to overcome the second issue. We show that both of these methods improve the performance on the task of pain detection in sequence level compared to current state-of-the-art-methods on the UNBC-McMaster Shoulder Pain Archive.

Automatic Detection of Pain from Facial Expressions: A Survey

“…Tsai et al [28] textural HOG from Three Orthogonal Planes (HOG-TOP) Chen et al [75] LBP from Three Orthogonal Planes (LBP-TOP) Kaltwang et al [113] combinations of LBP-TOP, LPQ-TOP, BSIF-TOP Yang et al [104] energy from optical flow Ghasemi et al [74] time-integral of histogram of oriented energies Irani et al [122], Irani et al [66] Hankel matrices from time series of Haar and/or Gabor Almost all classification and regression tasks were supervised. Ground truth in the form of pain or AU labels, and discrete or continuous-valued pain or AU intensities, were used to train the machine learning models.…”

“…The metrics used to quantify the performance of automatic pain detection from facial expressions depend on the learning task. For classification tasks, metrics such as accuracy, F1 score, and area under Receiver Operating Rudovic et al [121] hidden conditional random field Lopez-Martinez et al [118] regularized multi-task learning Romera-Paredes et al [108] two-step SVM step1-AU: Lucey et al [83], Lucey et al [128] step2-pain: Bartlett et al [131] both steps: Littlewort et al [72], Littlewort et al [56], Ghasemi et al [74] logistical linear regression step2-pain: Lucey et al [83], Lucey et al [128] k-nearest neighbors step1-AU: Zafar and Khan [129] logistic regression step2-pain: Sikka et al [59] alignment-based learning step2-pain: Schmid et al [77], Siebers et al [78] hidden conditional random field step2-pain: Ghasemi et al [74] latent-dynamic conditional random field step1-AU: Zhang et al [76] regression one-step support vector regression Florea et al [111], Lopez-Martinez et al [118] ordinal support vector regression Zhao et al [114] relevance vector regression or its variants Kaltwang et al [107], Kaltwang et al [113], Egede et al [116], Egede and Valstar [117] random forest Kächele et al [103] linear regression Neshov and Manolova [94] ordinal support vector regression Zhao et al [114] NN Lopez-Martinez et al [118] Convolutional Neural Network (CNN) Wang et al [109] 3D CNN with kernels of varying temporal lengths Tavakolian and Hadid [119] recurrent CNN Zhou et al [112] LSTM recurrent neural network Rodriguez et al [115], Lopez-Martinez et al [118] two-step support vector regression step1-AU: B...…”

“…The two-step approaches use or require an intermediate learning stage for describing the facial expression in terms of AUs or AU intensities. While the one-step approaches correspond to the classical way of learning the target from the input features, the two-step approaches are motivated by the way in which human observers detect and code pain [74]-on the basis of specific facial expression elements (cf. [13]).…”

“…Summary of spatial representations extracted directly from facial images for automatic pain detection. Berthouze[102], Ghasemi et al[74], Aung et al[27], Rupenga and Vadapalli[98], Liu et al[110], Lopez-Martinez et al[118] facial landmark distances Romera-Paredes et al[108], Meawad et al[99] facial landmark distances and angles Niese et al[54], Siebers et al[84] …”

Exploiting Domain-specific Knowledge for Classifier Learning - AU-based Facial Expression Analysis and Emotion Recognition

“…[62]). Two-stage approaches are inspired by the way in which human coders would rate pain [30], namely based on specific facial expression elements [14].…”