Multi-task Self-Supervised Learning for Human Activity Detection

Abstract

Deep learning methods are successfully used in applications pertaining to ubiquitous computing, pervasive intelligence, health, and well-being. Speci cally, the area of human activity recognition (HAR) is primarily transformed by the convolutional and recurrent neural networks, thanks to their ability to learn semantic representations directly from raw input. However, in order to extract generalizable features massive amounts of well-curated data are required, which is a notoriously challenging task; hindered by privacy issues and annotation costs. erefore, unsupervised representation learning (i.e., learning without manually labeling the instances) is of prime importance to leverage the vast amount of unlabeled data produced by smart devices. In this work, we propose a novel self-supervised technique for feature learning from sensory data that does not require access to any form of semantic labels, i.e., activity classes. We learn a multi-task temporal convolutional network to recognize transformations applied on an input signal. By exploiting these transformations, we demonstrate that simple auxiliary tasks of the binary classi cation result in a strong supervisory signal for extracting useful features for the down-stream task. We extensively evaluate the proposed approach on several publicly available datasets for smartphone-based HAR in unsupervised, semi-supervised and transfer learning se ings. Our method achieves performance levels superior to or comparable with fully-supervised networks trained directly with activity labels, and it performs signi cantly be er than unsupervised learning through autoencoders. Notably, for the semi-supervised case, the self-supervised features substantially boost the detection rate by a aining a kappa score between 0.7 − 0.8 with only 10 labeled examples per class. We get similar impressive performance even if the features are transferred from a di erent data source. Self-supervision drastically reduces the requirement of labeled activity data, e ectively narrowing the gap between supervised and unsupervised techniques for learning meaningful representations. While this paper focuses on HAR as the application domain, the proposed approach is general and could be applied to a wide variety of problems in other areas. 000:2 • A. Saeed et al.activity recognition (HAR), 1D convolutional and recurrent neural networks trained on raw labeled signals signi cantly improve the detection rate over traditional methods [20,44,68,72,73]. Despite the recent advances in the eld of HAR, learning representations from a massive amount of unlabeled data still presents a signi cant challenge. Obtaining large, well-curated activity recognition datasets is problematic due to a number of issues. First, smartphone data are privacy sensitive, which makes it hard to collect su cient amounts of user-activity instances in a real-life se ing. Second, the annotation cost and the time it takes to generate a large volume of labeled instances are prohibitive. Finally, the diversity of devices, types of embedd...

Idioms

“…A possible solution is to use self-supervised learning architecture that can be used to increase the generalizability of a classification model with a small number of labeled samples while utilizing unlabeled images that are more easily available. Self-supervised learning in the form of rotation augmentation has shown to improve the model generalizability in a number of computer vision [3,4,8,11] and human activity recognition [2,9] tasks. Inspired by our success with AugToAct [2], where we modeled the self-supervision task as regression (reconstruction of a rotated/scaled input feature space), instead of classification (predicting whether rotation was done) with higher performance gain, we propose to augment self-supervision with a reinforcement learning loop [1,6] to learn the optimum augmentation policy for self-supervision.…”

Federated Self-Supervised Learning of Multisensor Representations for Embedded Intelligence

“…L EARNING representations with deep neural networks have made tremendous improvements in the last few years on challenging real-world tasks [1]- [4], thanks to the emergence of massive datasets. In particular, the wealth of sensory data from the Internet of Things (IoT) devices are only recently being leveraged for tackling important problems in understanding context, user monitoring, health, and other predictive analytics tasks, e.g., for emotional well-being [5], [6], sleep tracking [7], and physical activity detection [8]. The success is mainly attributed to the supervised methods that utilize labeled datasets for training models in a central environment.…”

SelfHAR

“…Traditionally, HAR through wearable and mobile sensing relied on sliding window segmentation and manual feature extraction, followed by a variety of supervised learning techniques to recognize simple and complex activities like walking, running, cycling and cooking. While in simple scenarios hand-crafted features may suffice, deep learning methods have proven to be more effective in complex HAR tasks [12,21,29,37,39]. Indeed, deep learning has shown promise in HAR by automatically extracting useful features for the target task [42,61].…”

“…It is particularly important to consider limitations due to their size, diversity and ability to capture and represent the richness, noise and complexity that can be found in free-living, unconstrained data [25,48,49]. These limitations on mobile sensing datasets are only exacerbated by the difficulty in collecting labeled data outside of laboratory settings [39].…”

“…Most importantly, the intermediate representations capture semantic and structural meaning that can then be exploited for a variety of downstream tasks. This new paradigm has been successfully applied in filling the blanks for image datasets [14], next word prediction [7], video frame order validation [27], and, more recently, small-scale activity data [39].…”