Attention-Block Deep Learning Based Features Fusion in Wearable Social Sensor for Mental Wellbeing Evaluations

Abstract

With the progressive increase of stress, anxiety and depression in working and living environment, mental health assessment becomes an important social interaction research topic. Generally, clinicians evaluate the psychology of participants through an effective psychological evaluation and questionnaires. However, these methods suffer from subjectivity and memory effects. In this paper, a new multi-sensing wearable device has been developed and applied in self-designed psychological tests. Speech under different emotions as well as behavior signals are captured and analyzed. The mental state of the participants is objectively assessed through a group of psychological questionnaires. In particular, we propose an attention-based block deep learning architecture within the device for multi-feature classification and fusion analysis. This enables the deep learning architecture to autonomously train to obtain the optimum fusion weights of different domain features. The proposed attention-based architecture has led to improving performance compared with direct connecting fusion method. Experimental studies have been carried out in order to verify the effectiveness and robustness of the proposed architecture. The obtained results have shown that the wearable multi-sensing devices equipped with the attention-based block deep learning architecture can effectively classify mental state with better performance. INDEX TERMS Mental health assessment, wearable device, attention-based feature fusion.

The Concept of Advanced Multi-Sensor Monitoring of Human Stress

“…Statistically, more than 300 million people suffer from depression and, along with anxiety, are the most common mental disorders. The annual global cost is estimated at $2.5 trillion and is very likely to increase in the coming years [ 35 , 36 , 37 , 39 , 40 , 41 , 42 ].…”

“…There are currently several studies that attempt to identify stress patterns of voice using neural networks and machine learning [ 57 , 58 , 59 ]. Some even combine speech signals with electrodermal activity [ 60 ] or use wearable devices with multi-sensors combining audio and physiological sensors together with deep neural learning networks to monitoring an individual’s well-being in a naturalistic environment [ 42 , 61 , 62 ].…”

“…These systems can be easily incorporated into modern mobile phones [ 210 ]. In the research of Jin et al [ 42 ], in addition to the speech analysis itself, they included behavioral signals (3D accelerometer and 3D angle sensor) and propose an attention-based deep-learning architecture for a more accurate classification of mental state. Nowadays, there are even systems that perform the analysis of biomarkers from sweat and saliva directly during sports activities [ 211 ].…”

Analysis of Data from Wearable Sensors for Sleep Quality Estimation and Prediction Using Deep Learning

“…The intervention of machine learning and deep learning algorithms with wearable technology for healthcare monitoring has been seen recently in a number of studies. Jin et al [48] propose an attention-based deep learning framework with a multi sensing wearable device for mental wellbeing evaluation in order to do multi-feature classification and fusion analysis. The attention-based deep learning model shows improvement of performance.…”

Deep Temporal Convolution Network for Time Series Classification

“…Besides time series classification, deep neural networks such as pyramid recurrent neural network [ 33 ] can be used for change point detection to detect abrupt or gradual changes in the signal characteristics, achieved by transforming the time series data into a pyramid of multiscale feature maps in a trainable wavelet layer. In addition, an ensemble of neural networks can be used to boost the performance of time series classification [ 34 , 35 , 36 ]. InceptionTime [ 37 ] is one where a set of five different models formed by cascading multiple deep convolution neural networks, called the Inception module [ 38 ], are used.…”