Pain Intensity Estimation Using Deep Spatiotemporal and Handcrafted Features

Abstract

Automatically recognizing pain and estimating pain intensity is an emerging research area that has promising applications in the medical and healthcare field, and this task possesses a crucial role in the diagnosis and treatment of patients who have limited ability to communicate verbally and remains a challenge in pattern recognition. Recently, deep learning has achieved impressive results in many domains. However, deep architectures require a significant amount of labeled data for training, and they may fail to outperform conventional handcrafted features due to insufficient data, which is also the problem faced by pain detection. Furthermore, the latest studies show that handcrafted features may provide complementary information to deep-learned features; hence, combining these features may result in improved performance. Motived by the above considerations, in this paper, we propose an innovative method based on the combination of deep spatiotemporal and handcrafted features for pain intensity estimation. We use C3D, a deep 3-dimensional convolutional network that takes a continuous sequence of video frames as input, to extract spatiotemporal facial features. C3D models the appearance and motion of videos simultaneously. For handcrafted features, we propose extracting the geometric information by computing the distance between normalized facial landmarks per frame and the ones of the mean face shape, and we extract the appearance information using the histogram of oriented gradients (HOG) features around normalized facial landmarks per frame. Two levels of SVRs are trained using spatiotemporal, geometric and appearance features to obtain estimation results. We tested our proposed method on the UNBC-McMaster shoulder pain expression archive database and obtained experimental results that outperform the current state-of-the-art.

Pain expression assessment based on a locality and identity aware network

“…Some researches leveraged a set of hand‐crafted features, for example, [10, 28, 49, 50, 61, 62]. And some researchers combine traditional and deep features for pain assessment [19, 58, 59 63]. And we obtain the best results on UNBC and the good enough results on BioVid.…”

“…Besides, the comparison with deep learning methods on two databases is illustrated in Tables 5 and 6. As can be seen from Tables 3–6, as for pain intensity assessment, the conventional methods with hand‐crafted features plus classifiers [4, 5, 23, 31, 49–51, 57] in the early years gradually converted into deep learning methods [14–17, 19, 55, 56] recently. It is obvious that with the development of deep learning, the pain assessment method shows better performance.…”

“…Then, these features were aggregated to exploit statistics information. Wang et al [19] employed an SVR to estimate pain from three kinds of features: 3D CNN features, histogram of oriented gradients features and geometric features. And, in [36], the authors used two RNNs to exploit the dynamic information from the VGGface features.…”

“…Besides, in recent years, owing to the excellent performance of deep learning, the application of deep networks in pain assessment also spread. Specifically, convolutional neural networks (CNNs) [14,15], recurrent convolutional neural network (RCNN) [16], the combination of CNNs and recurrent neural networks (RNNs) [17], and 3D CNNs [18,19] had been used for this issue.…”

Image-Based Pain Intensity Estimation Using Parallel CNNs with Regional Attention

“…Tavakolian et al [ 20 ] used several convolutional layers with diverse temporal depths to form a spatiotemporal convolutional neural network (SCN) for automatic pain analysis. Wang et al [ 21 ] proposed a C3D network to extract spatiotemporal facial features and combined the HOG (Histogram of Oriented Gradient) feature histogram in 2D images as geometric information to distinguish the degree of pain in facial expressions. Mohammad et al [ 22 ] proposed statistical spatiotemporal distillation (SSD) to encode the spatiotemporal variations underlying the facial video into a single RGB image and then used 2D models to process video data.…”