Contrastive Predictive Coding for Human Activity Recognition

Haresamudram, Harish; Essa, Irfan; Plötz, Thomas

doi:10.1145/3463506

Cited by 77 publications

(77 citation statements)

References 75 publications

(79 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has demonstrated strong recognition when data from both accelerometer and gyroscope are available. Contrastive Predictive Coding (CPC) [31]: adopted and applied the contrastive predictive coding framework to wearable data, and makes use of long term temporal properties for effective representation learning by predicting multiple future timesteps. It performs successfully for activity recognition and semi-supervised learning, especially when multiple sensors are available.…”

Section: Self-supervised Har Methodsmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Haresamudram¹,

Essa²,

Plötz³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems from only small amounts of labeled training samples. Furthermore, self-supervised methods enable a host of new application domains such as, for example, domain adaptation and transfer across sensor positions, activities etc. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone the classic activity recognition chain with explicit focus on designing feature representations for sensor data. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, and utilize it to assess state-of-the-art self-supervised learning methods in a large empirical study on a curated set of nine diverse benchmarks. This exploration leads us to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios. Based on our findings we call upon the community to join our efforts and to contribute towards shaping the evaluation of the ongoing paradigm change in modeling human activities from body-worn sensor data.CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing; Empirical studies in ubiquitous and mobile computing; • Computing methodologies → Machine learning.

show abstract

Section: Self-supervised Har Methodsmentioning

confidence: 99%

“…The Contrastive Predictive Coding (CPC) framework was adopted and applied to human activity recognition in [31]. Windows of accelerometer and gyroscope data from mobile phones are encoded with a 1D convolutional encoder.…”

Section: Self-supervised Learning In Human Activity Recognitionmentioning

confidence: 99%

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Haresamudram¹,

Essa²,

Plötz³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…For a federated learning framework involving multiple sensors, such as WiFi, IMU, electroencephalogram, and blood volume pulse, Saeed et al [43] applied contrastive learning with wavelet transformations. Combining contrastive learning with predictive coding, Haresamudram et al [44] showed that the performance of downstream classification tasks could be significantly improved.…”

Section: Transfer Learning and Self-supervised Learningmentioning

confidence: 99%

“…Recently, semi-supervised and especially self-supervised learning methods [101][102][103] have become popular in many machine learning application domains including human activity recognition using wearables [13,43,44]. Here the idea is to enhance small amounts of labeled data through specific modification, formalized through so-called pretext learning tasks, such that, through solving the auxiliary task, a meta-learning procedure is forced to learn higher-level concepts that eventually lead to improved activity recognition performance.…”

Section: Virtual Imu Data As Basis For Alternatives To Supervised Learningmentioning

confidence: 99%

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

Kwon

Abowd

Plötz

2021

Sensors

Self Cite

View full text Add to dashboard Cite

Supervised training of human activity recognition (HAR) systems based on body-worn inertial measurement units (IMUs) is often constrained by the typically rather small amounts of labeled sample data. Systems like IMUTube have been introduced that employ cross-modality transfer approaches to convert videos of activities of interest into virtual IMU data. We demonstrate for the first time how such large-scale virtual IMU datasets can be used to train HAR systems that are substantially more complex than the state-of-the-art. Complexity is thereby represented by the number of model parameters that can be trained robustly. Our models contain components that are dedicated to capture the essentials of IMU data as they are of relevance for activity recognition, which increased the number of trainable parameters by a factor of 1100 compared to state-of-the-art model architectures. We evaluate the new model architecture on the challenging task of analyzing free-weight gym exercises, specifically on classifying 13 dumbbell execises. We have collected around 41 h of virtual IMU data using IMUTube from exercise videos available from YouTube. The proposed model is trained with the large amount of virtual IMU data and calibrated with a mere 36 min of real IMU data. The trained model was evaluated on a real IMU dataset and we demonstrate the substantial performance improvements of 20% absolute F1 score compared to the state-of-the-art convolutional models in HAR.

show abstract

“…Its key idea is to conduct a discriminative learning approach to learn encoded feature representations, in which similar sample pairs remain close together, whereas different sample pairs remain widely apart. It has been successfully verified in many computer vision tasks such as image classification [ 17 ] and human activity recognition [ 18 , 19 ].…”

Section: Introductionmentioning

confidence: 99%

A Computer Vision-Based Yoga Pose Grading Approach Using Contrastive Skeleton Feature Representations

Lin

Yang

et al. 2021

Healthcare

View full text Add to dashboard Cite

The main objective of yoga pose grading is to assess the input yoga pose and compare it to a standard pose in order to provide a quantitative evaluation as a grade. In this paper, a computer vision-based yoga pose grading approach is proposed using contrastive skeleton feature representations. First, the proposed approach extracts human body skeleton keypoints from the input yoga pose image and then feeds their coordinates into a pose feature encoder, which is trained using contrastive triplet examples; finally, a comparison of similar encoded pose features is made. Furthermore, to tackle the inherent challenge of composing contrastive examples in pose feature encoding, this paper proposes a new strategy to use both a coarse triplet example—comprised of an anchor, a positive example from the same category, and a negative example from a different category, and a fine triplet example—comprised of an anchor, a positive example, and a negative example from the same category with different pose qualities. Extensive experiments are conducted using two benchmark datasets to demonstrate the superior performance of the proposed approach.

show abstract

Contrastive Predictive Coding for Human Activity Recognition

Cited by 77 publications

References 75 publications

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Complex Deep Neural Networks from Large Scale Virtual IMU Data for Effective Human Activity Recognition Using Wearables

A Computer Vision-Based Yoga Pose Grading Approach Using Contrastive Skeleton Feature Representations

Contact Info

Product

Resources

About