Thermal spatio-temporal data for stress recognition

Abstract

Stress is a serious concern facing our world today, motivating the development of a better objective understanding through the use of non-intrusive means for stress recognition by reducing restrictions to natural human behavior. As an initial step in computer vision-based stress detection, this paper proposes a temporal thermal spectrum (TS) and visible spectrum (VS) video database ANUStressDB -a major contribution to stress research. The database contains videos of 35 subjects watching stressed and not-stressed film clips validated by the subjects. We present the experiment and the process conducted to acquire videos of subjects' faces while they watched the films for the ANUStressDB. Further, a baseline model based on computing local binary patterns on three orthogonal planes (LBP-TOP) descriptor on VS and TS videos for stress detection is presented. A LBP-TOP-inspired descriptor was used to capture dynamic thermal patterns in histograms (HDTP) which exploited spatio-temporal characteristics in TS videos. Support vector machines were used for our stress detection model. A genetic algorithm was used to select salient facial block divisions for stress classification and to determine whether certain regions of the face of subjects showed better stress patterns. Results showed that a fusion of facial patterns from VS and TS videos produced statistically significantly better stress recognition rates than patterns from VS or TS videos used in isolation. Moreover, the genetic algorithm selection method led to statistically significantly better stress detection rates than classifiers that used all the facial block divisions. In addition, the best stress recognition rate was obtained from HDTP features fused with LBP-TOP features for TS and VS videos using a hybrid of a genetic algorithm and a support vector machine stress detection model. The model produced an accuracy of 86%.

Thermal super-pixels for bimodal stress recognition

“…This method not only groups the adjacent pixels with high correlation but also increases the speed of processing. Our experimental results show promising outcomes and are in agreement with the stateof-the-art method of [3]. a b Super-pixels are the results of perceptual grouping of pixels and involve more information and provide better image alignment compared to using a single pixel alone [23].…”

“…In this work, similar to [3] in each frame the facial region is segmented into a grid of 3 × 3 blocks. Next, LBP features are computed for each block [28].…”

“…To employ the opportunities of fusing the two modalities, a recent literature [3] presented a computational model using the information from both thermal and RGB imaging. They proposed a new descriptor named Histogram of Dynamic Thermal Patterns (HDTP).…”

“…The features in [3] are extracted directly from pixel values. In this paper we show that extracting the features from super-pixels, followed by decision level fusion results in a system outperforming [3]. The experimental results on ANUstressDB database show that our system achieves 89% classification accuracy.…”

“…To the best of our knowledge, the only work on fusion of these two modalities for stress recognition is [3] which uses a feature level fusion of the two modalities. The features in [3] are extracted directly from pixel values. In this paper we show that extracting the features from super-pixels, followed by decision level fusion results in a system outperforming [3].…”