Spatial–Spectral–Temporal Framework for Emotion Recognition

Abstract

An emotion recognition method based on multispectral imaging technology and tissue oxygen saturation (StO2) is proposed in this study. This method is called spatial-spectral-temporal adjustment convolutional neural network (SACNN). First, we use the algorithm to extract the StO2 content of an emotionally sensitive nose area through real-time multispectral imaging technology. Compared with facial expression data, StO2 data are more objective and cannot be controlled and changed artificially. Second, we construct a clustering algorithm based on the emotional state by extracting the spectral, StO2, and spatial features of the nose image to obtain accurate signals of emotionally sensitive areas. To utilize the correlation between spectral and spatial signals, we propose an adjustment-based CNN module, which reorganizes the relationship between all previous layers of the feature map, thereby making the relationship among layers close and highly quantitative. The features extracted through this method are consistent with spatial-spectral features. Third, we incorporate the extracted temporal feature signal into the long short-term memory module and finally complete the correlation between the spatial-spectral-temporal features. Experimental results show that the accuracy of the SACNN algorithm in emotional recognition reaches 90%, and the proposed method is more competitive than state-of-the-art approaches. To the best of our knowledge, this study is the first to use time-series StO2 signals for emotion recognition. INDEX TERMS Multispectral imaging, oxygen saturation, spatial-spectral-temporal adjustment convolutional neural network. I.

Classification of emotional stress and physical stress using a multispectral based deep feature extraction model

“…The four main chromophores of facial skin absorption spectrum are deoxyhemoglobin (Hb), oxyhemoglobin (HbO 2 ), scattering effect, and melanin. Given that the light absorption of each chromophore varies in different wavelengths, the commonly used StO 2 extraction algorithms are mostly based on the Beer–Lambert (BL) model 101 , 102 , 104 – 109 . The BL model describes the relationship between the light absorption and the thickness of the absorbing medium, which can be expressed as follows: where A is the rate of absorption; ε is the molar absorptivity of HbO 2 , Hb, and melanin; and c is the effective concentration ( 10 −3 mol/cm 2 ).…”

“…After obtaining the wavelet decomposed image, the obtained subblocks are amplified. Eulerian magnification (EM) has been used for signal magnification 44 , 83 , 101 , 102 . For practical application, signal amplification has been troubled by noise, especially in the process of amplifying high-frequency signals.…”

“…Then, the correlation between feature points and adjacent dictionaries needs to be taken into account. To this end, a multidimensional spatial relationship map (MSM) in the dictionary is initially defined 83 , 101 , 102 . The MSM is designed into five layers, and the adjacent points are then set based on the cluster center KNNs in the MSM; its high-order neighbors are used as the scanning points for MSM.…”