HybNet: a hybrid network structure for pain intensity estimation

Huang, Yibo; Qing, Linbo; Xu, Shuangxi; Wang, Lu; Peng, Yonghong

doi:10.1007/s00371-021-02056-y

Cited by 26 publications

(26 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Siamese network [9] 1.03 Joint deep neural network [14] 0.95 HybNet [15] 0.76 LSTM [12] 0.74 CNN [13] 0.69 SCN [16] 0.32 EJH-CVV-BiLSTM * [10] 0. To gain more insights into the behavior of different models, Figure 6 shows the performance of different models at frame level for one video.…”

Section: Methods Msementioning

confidence: 99%

“…In Reference [14], the authors used two different Recurrent Neural Networks (RNN), which were pre-trained with VGGFace-CNN and then joined together as a network for pain assessment. Recent work of Huang et al [15] proposed a hybrid network to estimate pain. In this paper, the authors proposed to extract multidimensional features from images.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures

et al. 2021

View full text Add to dashboard Cite

Automatic pain recognition from facial expressions is a challenging problem that has attracted a significant attention from the research community. This article provides a comprehensive analysis on the topic by comparing some popular and Off-the-Shell CNN (Convolutional Neural Network) architectures, including MobileNet, GoogleNet, ResNeXt-50, ResNet18, and DenseNet-161. We use these networks in two distinct modes: stand alone mode or feature extractor mode. In stand alone mode, the models (i.e., the networks) are used for directly estimating the pain. In feature extractor mode, the “values” of the middle layers are extracted and used as inputs to classifiers, such as SVR (Support Vector Regression) and RFR (Random Forest Regression). We perform extensive experiments on the benchmarking and publicly available database called UNBC-McMaster Shoulder Pain. The obtained results are interesting as they give valuable insights into the usefulness of the hidden CNN layers for automatic pain estimation.

show abstract

Section: Methods Msementioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Facial expression is the most well-applied indicator for pain assessment in practice as it is non-invasive and easily acquired by video recording techniques [6]- [10]. Extracting the best set of facial features is critical to obtain accurate pain assessment.…”

Section: Facial Expressionmentioning

confidence: 99%

Multimodal-Based Stream Integrated Neural Networks for Pain Assessment

Zhi

Zhou

et al. 2021

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Pain is an essential physiological phenomenon of human beings. Accurate assessment of pain is important to develop proper treatment. Although self-report method is the gold standard in pain assessment, it is not applicable to individuals with communicative impairment. Nonverbal pain indicators such as pain related facial expressions and changes in physiological parameters could provide valuable insights for pain assessment. In this paper, we propose a multimodal-based Stream Integrated Neural Network with Different Frame Rates (SINN) that combines facial expression and biomedical signals for automatic pain assessment. The main contributions of this research are threefold. (1) There are four-stream inputs of the SINN for facial expression feature extraction. The variant facial features are integrated with biomedical features, and the joint features are utilized for pain assessment. (2) The dynamic facial features are learned in both implicit and explicit manners to better represent the facial changes that occur during pain experience. (3) Multiple modalities are utilized to identify various pain states, including facial expression and biomedical signals. The experiments are conducted on publicly available pain datasets, and the performance is compared with several deep learning models. The experimental results illustrate the superiority of the proposed model, and it achieves the highest accuracy of 68.2%, which is up to 5% higher than the basic deep learning models on pain assessment with binary classification.

show abstract

“…A number of automated recognition of facial expressions of pain and emotion has been developed through using distinct approaches ( 5 – 9 ). Pedersen et al used Support Vector Machine (SVM) as a facial expression-based pain classifier in UNBC-McMaster Shoulder Pain Expression Archive Database, consisting of 200 video sequences obtained from 25 patients with shoulder pain, and reported that the accuracy of the leave-one-subject-out 25-fold cross was 0.861 ( 7 ).…”

Section: Introductionmentioning

confidence: 99%

“…Pedersen et al used Support Vector Machine (SVM) as a facial expression-based pain classifier in UNBC-McMaster Shoulder Pain Expression Archive Database, consisting of 200 video sequences obtained from 25 patients with shoulder pain, and reported that the accuracy of the leave-one-subject-out 25-fold cross was 0.861 ( 7 ). Given that video sequences contain temporal information with respect to pain, two studies were used Recurrent Neural Network (RNN) and hybrid network to extract the time-frame feature among images and reported an improved performance ( 8 , 9 ). Furthermore, recent studies have employed fusion network architectures and further improved the F1 score to ~0.94 ( 10 , 11 ).…”

Section: Introductionmentioning

confidence: 99%

Deep Learning-Based Pain Classifier Based on the Facial Expression in Critically Ill Patients

Liu

et al. 2022

Front. Med.

View full text Add to dashboard Cite

ObjectivePain assessment based on facial expressions is an essential issue in critically ill patients, but an automated assessment tool is still lacking. We conducted this prospective study to establish the deep learning-based pain classifier based on facial expressions.MethodsWe enrolled critically ill patients during 2020–2021 at a tertiary hospital in central Taiwan and recorded video clips with labeled pain scores based on facial expressions, such as relaxed (0), tense (1), and grimacing (2). We established both image- and video-based pain classifiers through using convolutional neural network (CNN) models, such as Resnet34, VGG16, and InceptionV1 and bidirectional long short-term memory networks (BiLSTM). The performance of classifiers in the test dataset was determined by accuracy, sensitivity, and F1-score.ResultsA total of 63 participants with 746 video clips were eligible for analysis. The accuracy of using Resnet34 in the polychromous image-based classifier for pain scores 0, 1, 2 was merely 0.5589, and the accuracy of dichotomous pain classifiers between 0 vs. 1/2 and 0 vs. 2 were 0.7668 and 0.8593, respectively. Similar accuracy of image-based pain classifier was found using VGG16 and InceptionV1. The accuracy of the video-based pain classifier to classify 0 vs. 1/2 and 0 vs. 2 was approximately 0.81 and 0.88, respectively. We further tested the performance of established classifiers without reference, mimicking clinical scenarios with a new patient, and found the performance remained high.ConclusionsThe present study demonstrates the practical application of deep learning-based automated pain assessment in critically ill patients, and more studies are warranted to validate our findings.

show abstract

HybNet: a hybrid network structure for pain intensity estimation

Cited by 26 publications

References 35 publications

Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures

Automatic Pain Estimation from Facial Expressions: A Comparative Analysis Using Off-the-Shelf CNN Architectures

Multimodal-Based Stream Integrated Neural Networks for Pain Assessment

Deep Learning-Based Pain Classifier Based on the Facial Expression in Critically Ill Patients

Contact Info

Product

Resources

About