Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level

Khan, Masood Mehmood; Ward, Robert; Ingleby, Michael

doi:10.1109/taffc.2016.2535291

Cited by 25 publications

(18 citation statements)

References 70 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thermal imaging is a key non-contact method to study heat patterns of materials and organisms. Despite thermographic channels being still little explored in affective computing, various studies have explored the possible thermal signatures appearing in association with a person's psychological affective states (e.g., [17]- [21]). For instance, Engert et al [18] identified that a decrease of temperature of the nose tip and perioral areas could be a barometer of mentally stressful states.…”

Section: Thermal Imaging and Affect Detectionmentioning

confidence: 99%

“…For instance, Engert et al [18] identified that a decrease of temperature of the nose tip and perioral areas could be a barometer of mentally stressful states. While most works (e.g., [17]- [19]) focused on attempting to confirm the relationship between directional changes in temperature of facial areas and affective states, a few works investigated how such thermal information can be used for automatic affect recognition (e.g., [20], [21]). For example, Nhan and Chau [20] used temperature changes on left, right supraorbital, periorbital, and nasal areas as features for distinguishing high arousal from a baseline.…”

Section: Thermal Imaging and Affect Detectionmentioning

confidence: 99%

See 1 more Smart Citation

DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

Cho

Bianchi-Berthouze

Julier

2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

102

113

View full text Add to dashboard Cite

We propose DeepBreath, a deep learning model which automatically recognises people's psychological stress level (mental overload) from their breathing patterns. Using a low cost thermal camera, we track a person's breathing patterns as temperature changes around his/her nostril. The paper's technical contribution is threefold. First of all, instead of creating handcrafted features to capture aspects of the breathing patterns, we transform the uni-dimensional breathing signals into two dimensional respiration variability spectrogram (RVS) sequences. The spectrograms easily capture the complexity of the breathing dynamics. Second, a spatial pattern analysis based on a deep Convolutional Neural Network (CNN) is directly applied to the spectrogram sequences without the need of hand-crafting features. Finally, a data augmentation technique, inspired from solutions for over-fitting problems in deep learning, is applied to allow the CNN to learn with a small-scale dataset from short-term measurements (e.g., up to a few hours). The model is trained and tested with data collected from people exposed to two types of cognitive tasks (Stroop Colour Word Test, Mental Computation test) with sessions of different difficulty levels. Using normalised self-report as ground truth, the CNN reaches 84.59% accuracy in discriminating between two levels of stress and 56.52% in discriminating between three levels. In addition, the CNN outperformed powerful shallow learning methods based on a single layer neural network. Finally, the dataset of labelled thermal images will be open to the community.

show abstract

Section: Thermal Imaging and Affect Detectionmentioning

confidence: 99%

Section: Thermal Imaging and Affect Detectionmentioning

confidence: 99%

DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

Cho

Bianchi-Berthouze

Julier

2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

102

113

View full text Add to dashboard Cite

show abstract

“…These cues also help humans in observing and synthesizing others' expressions and assessing affective experiences [16]. To imitate the human model of ASA, algorithm-based learning and classification approaches rely on one or more of visual, vocal, psychophysiological and neural cues [13,[17][18][19]. Emerging sophisticated ASA systems use different combinations of affective state models, learning techniques and classification methods [20][21][22] as their capabilities and accuracies progressively improve [23][24][25][26].…”

Section: Introductionmentioning

confidence: 99%

“…Emerging sophisticated ASA systems use different combinations of affective state models, learning techniques and classification methods [20][21][22] as their capabilities and accuracies progressively improve [23][24][25][26]. Modern ASA capabilities include the dynamic assessment of affect-arousal and, multimodal and contextual assessment of affective states [18,[26][27][28]. Since ASA systems rely on algorithm-based analysis, they inherit algorithmic biases while making inferences.…”

Section: Introductionmentioning

confidence: 99%

Toward Accountable and Explainable Artificial Intelligence Part Two: The Framework Implementation

Khan¹,

Vice²

2022

Preprint

View full text Add to dashboard Cite

<div>This paper builds upon the theoretical foundations of the Accountable explainable Artificial Intelligence (AXAI) capability framework presented in part one of this paper. This part demonstrates the incorporation of the AXAI capability in the real time Affective State Assessment Module (ASAM) of a robotic system. The paper argues that adhering to the extreme Programming (XP) practices would help in understanding user behavior while systematically incorporating the AXAI capability in AI systems. Issues pertaining to identification of ethical, technical, functional, and domain-specific system requirements were resolved using an appropriate software design process, thus establishing confidence in the system. The presented ASAM synthesizes discrete and continuous models of affective state expressions and classifies them in real-time. Input, processed and output data are continuously shared with users via a graphical user interface (GUI) for providing reasons behind decisions. The GUI also disseminates information about local reasoning, data handling and decision-making. We hope this work will initiate further investigations on selection of suitable software design practices for incorporating the AXAI capability in AI systems, enhancing AI system acceptability and utility and, establishing a chain of responsibility.</div>

show abstract

“…Although earlier approaches provided initial evidences of the relation between affect and temperature, a variety of body regions and their thermal signatures have been much less investigated than thermal directionality of facial and palm regions [4], [12]- [17]. Little is indeed known about how such areas respond to stress level.…”

Section: Introductionmentioning

confidence: 99%

Automated mental stress recognition through mobile thermal imaging

Cho

2017

2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)

View full text Add to dashboard Cite

Abstract-Mental stress is a critical problem in our modern society. This form of stress strongly affects our well being, and technology is needed to help us to manage health problems. The ability to automatically recognize a person's mental stress can be fundamental in supporting stress and health management. This research focuses on the use of mobile thermal imaging, a new and less explored sensor, to merge the measurement of multiple physiological signatures into one sensor and to build a reliable mental stress automatic recognition model. Mobile thermal imaging has greater potentials for real-world applications given that it is small and light weight, and requires low computation cost. To make mobile thermal imaging a robust multimodal stress sensor, we have so far contributed: i) a new robust respiration tracking method; and ii) a novel respiration-based automatic stress recognition model that works in ubiquitous settings. We are currently investigating new thermal signatures from underexplored body regions (i.e. trapezius muscle) and formulating a research framework to fuse multiple thermal signatures for more reliable stress recognition outcomes.

show abstract

Toward Use of Facial Thermal Features in Dynamic Assessment of Affect and Arousal Level

Cited by 25 publications

References 70 publications

DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

DeepBreath: Deep learning of breathing patterns for automatic stress recognition using low-cost thermal imaging in unconstrained settings

Toward Accountable and Explainable Artificial Intelligence Part Two: The Framework Implementation

Automated mental stress recognition through mobile thermal imaging

Contact Info

Product

Resources

About