Two-Stream Convolution Augmented Transformer for Human Activity Recognition

Abstract

Recognition of human activities is an important task due to its far-reaching applications such as healthcare system, context-aware applications, and security monitoring. Recently, WiFi based human activity recognition (HAR) is becoming ubiquitous due to its non-invasiveness. Existing WiFi-based HAR methods regard WiFi signals as a temporal sequence of channel state information (CSI), and employ deep sequential models (e.g., RNN, LSTM) to automatically capture channel-over-time features. Although being remarkably effective, they suffer from two major drawbacks. Firstly, the granularity of a single temporal point is blindly elementary for representing meaningful CSI patterns. Secondly, the time-over-channel features are also important, and could be a natural data augmentation. To address the drawbacks, we propose a novel Two-stream Convolution Augmented Human Activity Transformer (THAT) model. Our model proposes to utilize a two-stream structure to capture both time-over-channel and channel-over-time features, and use the multi-scale convolution augmented transformer to capture range-based patterns. Extensive experiments on four real experiment datasets demonstrate that our model outperforms state-of-the-art models in terms of both effectiveness and efficiency.

Dual-Stream Contrastive Learning for Channel State Information Based Human Activity Recognition

“…On this basis, Chen et al argued that traditional LSTM networks ignored the backward direction's information during prediction and proposed ABLSTM for HAR with WiFi CSI [13]. Li et al [19] showed that existing methods treated each frame as a single temporal point which ignores the informative context and only considered channel-over-time features while disregarding the time-over-channel features. Hence, they proposed THAT and gained SOTA performance.…”

“…There are two encoders in our model. Both of them are built up with Multiscale Convolution Augmented Transformer (MCAT) layers proposed in [19] and 1D convolution layers. Fig.…”

“…The temporal encoder f t (•) extracts feature along temporal axis. Starting with a position encoding module [19], f t (•) consists of five MCAT layers whose multi-head attention sublayer has 5 heads and dimensionality equals 90. The Channel encoder f c (•) extracts feature along channel axis.…”

“…: It randomly initializes parameters and freeze the model except for the last layer classifier. • Supervised: It trains THAT [19] model in a fully supervised learning manner with default setting. • AutoEncoder: It is trained the using reconstruction loss (mean squared error).…”

“…We choose the subset that was collected in the home environment for the experiment. For all datasets, we adopt preprocessing method mentioned in [19].…”