Harmonic Loss Function for Sensor-Based Human Activity Recognition Based on LSTM Recurrent Neural Networks

Abstract

Human activity recognition (HAR) has been a very popular field in both real practice and theoretical research. Over the years, a number of many-vs-one Long Short-Term Memory (LSTM) models have been proposed for the sensor-based HAR problem. However, how to utilize sequence outputs of them to improve the HAR performance has not been studied seriously. To solve this problem, we present a novel loss function named harmonic loss, which is utilized to improve the overall classification performance of HAR based on baseline LSTM networks. First, label replication method is presented to duplicate true labels at each sequence step in many-vs-one LSTM networks, thus each sequence step can generate a local error and a local output. Then, considering the importance of different local errors and inspired by the Ebbinghaus memory curve, the harmonic loss is proposed to give unequal weights to different local errors based on harmonic series equation. Additionally, to improve the overall classification performance of HAR, integrated methods are utilized to exploit the sequence outputs of LSTM models based on harmonic loss and ensemble learning strategy. Finally, based on the LSTM model construction and hyper-parameter setting, extensive experiments are conducted. A series of experimental results demonstrate that our harmonic loss significantly achieves higher macro-F1 and accuracy than strong baselines on two public HAR benchmarks. Compared with previous state-of-art methods, our proposed methods can achieve competitive classification performance.

Deep Learning in Human Activity Recognition with Wearable Sensors: A Review on Advances

“…In an effort to address these challenges, long short-term memory (LSTM)based RNNs [195], and Gated Recurrent Units (GRUs) [196] are introduced to model temporal sequences and their broad dependencies. The GRU introduces a reset and update gate to control the flow of inputs to a cell [197][198][199][200][201]. The LSTM has been shown capable of memorizing and modelling the long-term dependency in data.…”

“…Residual connections do not impede gradients and could help to refine the output of layers. For example, [200] proposes a harmonic loss function and [207] combines LSTM with batch normalization to achieve 92% accuracy with raw accelerometer and gyroscope data. Ref.…”

Upper Body Posture Recognition Using Inertial Sensors and Recurrent Neural Networks

“…RNNs have been widely used in language modeling, objective detection, and speech recognition and have received increased attention in biomechanics. RNNs are often used to construct sequence-to-sequence models and have been applied to HAR tasks because of RNNs' competence in temporal domain processing [24,[33][34][35][36]. RNNs can transition between hidden states of previous time steps to subsequent time steps, processing and passing down time-dependent sequences in chronological order.…”

“…Sci. 2021, 11, x FOR PEER REVIEW 6 of 13 because of RNNs' competence in temporal domain processing [24,[33][34][35][36]. RNNs can transition between hidden states of previous time steps to subsequent time steps, processing and passing down time-dependent sequences in chronological order.…”

Inertial Sensor Data to Image Encoding for Human Action Recognition

“…To enhance the recognition process, feature level and late fusions are also investigated using two datasets. In [31], a novel loss function called harmonic loss, which is based on the label replication method to replicate true labels at each sequence step of LSTM models, is presented to improve the overall classification performance of sensor based HAR. This loss function not only takes all local sequence errors into accounts but also considers the relative importance of different local errors in the training.…”