Bi-LSTM Network for Multimodal Continuous Human Activity Recognition and Fall Detection

Abstract

This paper presents a framework based on multi-layer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities' patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic "leaving one participant out" (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.

Sequential Human Gait Classification With Distributed Radar Sensor Fusion

“…For sequential gaits, the network is able to learn the dependencies related to the order of the gaits, where the transition between two different gait styles is the key to reinforce the inter-connections among the cells on two Bi-LSTM layers. A dual layer architecture has higher capabilities than a single layer one; however, there is a trade off between the number of layers and the computational complexity to achieve a boost in accuracy with feasible network training time, as shown in previous work [27], [28].…”

“…Typical hard fusion methods include majority voting (MV), weighted majority voting (WMV), Recall Combiner (RC) and Naive Bayes Combiner (NBC) [39]. In our previous work [27], it was shown that NBC outperformed other hard fusion methods, and this was chosen to be the hard fusion approach also in this paper. Compared to soft fusion, NBC attempts to gather the results from all the N classifiers to build a classifier ensemble.…”

“…This procedure can help correct events misclassified by single radar and ultimately yield higher confidence level for the correct class after fusion. Hard decision fusion [27], [39] utilizes the prediction results of the classifier in the confusion matrices, rather than focusing on the posterior probabilities. Typical hard fusion methods include majority voting (MV), weighted majority voting (WMV), Recall Combiner (RC) and Naive Bayes Combiner (NBC) [39].…”

“…For radar data of human activities or walking gaits in a sequence, Bi-LSTM can capture the kinematic constraints and correlations that link each action or gait to the previous and the following actions in the sequence. In previous work [27], [28], Bi-LSTMs were shown to be beneficial for classification of human activities and fall detection. However, in this paper we modify that approach.…”

Human Motion Serialization Recognition With Through-the-Wall Radar

“…It can utilize the sequentiality to learn and extract sequential features from a sequential input data stream. In [25], H. Li et al presented a framework based on multilayer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities' patterns and high-risk events such as falls. In [26], J. Zhu et al proposed a deep learning model composed of 1-D convolutional neural networks (1D-CNNs) and long shortterm memory (LSTM).…”