Automatic Detection of Pain from Facial Expressions: A Survey

Abstract

Pain sensation is essential for survival, since it draws attention to physical threat to the body. Pain assessment is usually done through self-reports. However, self-assessment of pain is not available in the case of noncommunicative patients, and therefore, observer reports should be relied upon. Observer reports of pain could be prone to errors due to subjective biases of observers. Moreover, continuous monitoring by humans is impractical. Therefore, automatic pain detection technology could be deployed to assist human caregivers and complement their service, thereby improving the quality of pain management, especially for noncommunicative patients. Facial expressions are a reliable indicator of pain, and are used in all observer-based pain assessment tools. Following the advancements in automatic facial expression analysis, computer vision researchers have tried to use this technology for developing approaches for automatically detecting pain from facial expressions. This paper surveys the literature published in this field over the past decade, categorizes it, and identifies future research directions. The survey covers the pain datasets used in the reviewed literature, the learning tasks targeted by the approaches, the features extracted from images and image sequences to represent pain-related information, and finally, the machine learning methods used.

Towards Robust and Interpretable Practical Applications of Automatic Mental State Analysis Using a Dynamic and Hybrid Facial Action Estimation Approach

“…In the process, several datasets consisting of facial expressions of pain have been created. A survey of these automatic pain detection methods and the pain datasets is provided in Hassan et al [68] (Publication B.1.1). The key findings are summarised in this section.…”

“…BioVid Heat Pain Database [190], WESAD [165] 1 , a multimodal distracted driving dataset [177]). However, the stimuli used to induce a mental state, the recorded modalities, the devices used to acquire multimodal data, and the self or observer-reporting methods used for annotating data vary between datasets [68,166,170,198], making models developed on one dataset not comparable with those developed on another dataset. The use cases and settings that are considered for data collection also vary between datasets.…”

“…Stress datasets have been collected for application contexts such as driving, computerbased work, or public speaking [123]. Pain datasets have been collected mostly under experimental conditions and occassionally under clinical settings [68]. Since there can be use case-specific opportunities and challenges in the acquisition of data, and use case-specific differences in the features and methods used for mental state detection, separate datasets for each use case is necessary.…”

“…[179]). However, automatic pain detection approaches have seldom tried to distinguish pain from other emotions such as disgust and happiness [68]. Therefore, we trained a CNN model to discriminate between pain, happiness and disgust, and to apply explainable AI methods to make the predictions made by this CNN model transparent.…”

“…Further, automatic mental state analysis could contribute to preventive healthcare, by monitoring and detecting unhealthy stress levels at workplaces or in everyday life [15]. In the medical field, mental state analysis could assist professionals in the diagnosis of psychological conditions such as depression [151], or assist caregivers in efficient detection and subsequent treatment of pain [68,198]. In the field of market research, automatic mental state analysis could be used to study a subject's emotional response to advertisements (e.g.…”