Feature Fusion of a Deep-Learning Algorithm into Wearable Sensor Devices for Human Activity Recognition

Abstract

This paper presents a wearable device, fitted on the waist of a participant that recognizes six activities of daily living (walking, walking upstairs, walking downstairs, sitting, standing, and laying) through a deep-learning algorithm, human activity recognition (HAR). The wearable device comprises a single-board computer (SBC) and six-axis sensors. The deep-learning algorithm employs three parallel convolutional neural networks for local feature extraction and for subsequent concatenation to establish feature fusion models of varying kernel size. By using kernels of different sizes, relevant local features of varying lengths were identified, thereby increasing the accuracy of human activity recognition. Regarding experimental data, the database of University of California, Irvine (UCI) and self-recorded data were used separately. The self-recorded data were obtained by having 21 participants wear the device on their waist and perform six common activities in the laboratory. These data were used to verify the proposed deep-learning algorithm on the performance of the wearable device. The accuracy of these six activities in the UCI dataset and in the self-recorded data were 97.49% and 96.27%, respectively. The accuracies in tenfold cross-validation were 99.56% and 97.46%, respectively. The experimental results have successfully verified the proposed convolutional neural network (CNN) architecture, which can be used in rehabilitation assessment for people unable to exercise vigorously.

Merging-Squeeze-Excitation Feature Fusion for Human Activity Recognition Using Wearable Sensors

“…Therefore, the performance results of the investigated model will highly depend on the data in the testing set. In order to avoid this problem, similar to [18,22,25,27], we applied the k-fold cross validation, where k is set to 10, to the experiments. The 10-fold cross validation will divide a dataset into 10 parts.…”

“…In the second category, we apply a set of sensor data to a multi-branch DL architecture where each branch uses a different DL model and results in a different set of features. Three-branch DL architectures were proposed in [20,21,[24][25][26][27][28], where a CNN model [20,24,27,28], a hybrid of a CNN model, and a bidirectional long short-term memory (LSTM) layer [25], as well as a CNN model with an SE block [21], and a hybrid of convolutional layers and gated recurrent unit (GRU) layers [26] were used on each branch to extract a set of features. The differences among these branches are the kernel sizes of the convolutional layers [20,21,[24][25][26][27][28] and the number of layers [20].…”

“…Conventional feature fusion methods [19][20][21][22][23][24][25][26][27][28] combine all feature maps from all branches equally without considering which feature maps are useful. However, some feature maps in A (n) may be unhelpful for the classification and they should be suppressed while the informative feature maps in A (n) should be emphasized.…”

Feature Fusion-Based Deep Learning Network to Recognize Table Tennis Actions

“…Pardo et al placed sixaxis sensors on participant wrists and waists for data collection and used a CNN to classify four shots in tennis and seven non-tennis activities with a mean accuracy of 99% [24]. Yen et al used deep learning with feature fusion method into wearable sensor devices for human activity recognition and the accuracies in tenfold cross-validation were 99.56% and 97.46%, respectively [25].…”

Wearable Sensor Feature Fusion for Human Activity Recognition (HAR): A Proposed Classification Framework

“…It has recently become possible to extract features from raw sensory data using deep learning methods. On the other hand, the most recent research in HAR has shown that one cannot disregard the significance of handcrafted features because this information is derived from the domain expertise of specialists [3]. Conventional ways of machine learning extract hand-crafted features that need to be chosen manually.…”