Understanding and improving recurrent networks for human activity recognition by continuous attention

Abstract

Deep neural networks, including recurrent networks, have been successfully applied to human activity recognition. Unfortunately, the final representation learned by recurrent networks might encode some noise (irrelevant signal components, unimportant sensor modalities, etc.). Besides, it is difficult to interpret the recurrent networks to gain insight into the models' behavior. To address these issues, we propose two attention models for human activity recognition: temporal attention and sensor attention. These two mechanisms adaptively focus on important signals and sensor modalities. To further improve the understandability and mean F1 score, we add continuity constraints, considering that continuous sensor signals are more robust than discrete ones. We evaluate the approaches on three datasets and obtain state-of-theart results. Furthermore, qualitative analysis shows that the attention learned by the models agree well with human intuition.

Improving Cross-Subject Activity Recognition via Adversarial Learning

“…The Daphnet data was recorded during various walking tasks of 10 different participants and have three different annotations: 1) transient activities (which are discarded here), 2) freezing of gait, and 3) normal movements. The data were recorded with a sampling rate of 64Hz, however, we downsample it to 32Hz by decimation and discard the transient activities, following [40].…”

“…We choose a sliding window of approximately 2.5 seconds for the HARD, HARD2 and HARD3 datasets, with 50% of overlapping. Following other works [8], [26], [40], the PAMAP2 and Daphnet datasets have, respectively, a window size of approximately In these networks, the second FC layer contains the same number of neurons as the number of classes and is followed by the softmax function. The convolutional kernel for all CNN layers was set to 3 × 3, whereas the max-pooling kernel size was 2 × 2.…”

“…Subjects 5 and 6 form the validation and test sets, respectively, for the PAMAP2 dataset. This is a common choice of subjects in the literature [7], [8], [26], [37], [40]. Decimation was used as the downsampling method.…”

Learning Temporal and Bodily Attention in Protective Movement Behavior Detection

“…Targeting the spatial and temporal subset selection within such data, attention mechanisms have been recently explored to improve HAR performances on MoCap data. To enable understanding of the relevance of each sensor in such scenario, Zeng et al [19] proposed an attentionbased LSTM framework, where a sensor attention module was used at the input level and for each timestep, with an additional temporal attention module at a later layer. Their sensor attention module was implemented with input from different sensors at single timestep, while temporal attention was computed based on the output of the LSTM layer.…”

“…We also compare with a variant of the BANet with a fullyconnected layer used in the temporal attention computation (BANet-dense) instead of a 1 × 1 convolution layer. In addition, we compare our work with the approach used in related HAR studies [19] [20] [21], where the sensor attention was computed before the extraction of temporal information. As such, we create a variant (BANet-compat, for BANet compatibility version) where the computation of body attention was done at input level instead of at feature fusion level with the same attention algorithms presented in last section.…”