IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

Kwon, Hyeokhyen; Tong, Catherine; Haresamudram, Harish; Gao, Yan; Abowd, Gregory D.; Lane, Nicholas D.; Ploetz, Thomas

doi:10.48550/arxiv.2006.05675

Cited by 3 publications

(4 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent approaches addressing the lack of annotated datasets involve extraction of virtual movement data from modalities such as mocap [37,72] and videos [44], which contain a large number of participants and diverse set of actions and environments. Such techniques have the advantage of being capable of extracting large quantities of virtual movement data at arbitrary positions on the body.…”

Section: Research Agenda: Representation Learning For Human Activity ...mentioning

confidence: 99%

Contrastive Predictive Coding for Human Activity Recognition

Haresamudram¹,

Essa²,

Ploetz³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Feature extraction is crucial for human activity recognition (HAR) using body-worn movement sensors. Recently, learned representations have been used successfully, offering promising alternatives to manually engineered features. Our work focuses on effective use of small amounts of labeled data and the opportunistic exploitation of unlabeled data that are straightforward to collect in mobile and ubiquitous computing scenarios. We hypothesize and demonstrate that explicitly considering the temporality of sensor data at representation level plays an important role for effective HAR in challenging scenarios. We introduce the Contrastive Predictive Coding (CPC) framework to human activity recognition, which captures the long-term temporal structure of sensor data streams. Through a range of experimental evaluations on real-life recognition tasks, we demonstrate its effectiveness for improved HAR. CPC-based pre-training is self-supervised, and the resulting learned representations can be integrated into standard activity chains. It leads to significantly improved recognition performance when only small amounts of labeled training data are available, thereby demonstrating the practical value of our approach.

show abstract

Section: Research Agenda: Representation Learning For Human Activity ...mentioning

confidence: 99%

Contrastive Predictive Coding for Human Activity Recognition

Haresamudram¹,

Essa²,

Ploetz³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Similar to BMTL, the loss function of TMTL is also calculated on each individual task. We then take the summation of losses over all source domains as the final objective function (1).…”

Section: Invariant Feature Learningmentioning

confidence: 99%

“…Human activity recognition (HAR) is the foundation to realize remote health services and in-home mobility monitoring. Although deep learning has seen many successes in this field, training deep models often requires a large amount of sensory data that is not always available [1]. For research ethics compliance, it often takes months to design study protocols, recruit volunteers and collect customized sensory datasets.…”

Section: Introductionmentioning

confidence: 99%

Invariant Feature Learning for Sensor-based Human Activity Recognition

Hao¹,

Wang²,

Zheng³

2020

Preprint

View full text Add to dashboard Cite

Wearable sensor-based human activity recognition (HAR) has been a research focus in the field of ubiquitous and mobile computing for years. In recent years, many deep models have been applied to HAR problems. However, deep learning methods typically require a large amount of data for models to generalize well. Significant variances caused by different participants or diverse sensor devices limit the direct application of a pre-trained model to a subject or device that has not been seen before. To address these problems, we present an invariant feature learning framework (IFLF) that extracts common information shared across subjects and devices. IFLF incorporates two learning paradigms: 1) meta-learning to capture robust features across seen domains and adapt to an unseen one with similarity-based data selection; 2) multi-task learning to deal with data shortage and enhance overall performance via knowledge sharing among different subjects. Experiments demonstrated that IFLF is effective in handling both subject and device diversion across popular open datasets and an in-house dataset. It outperforms a baseline model of up to 40% in test accuracy.

show abstract

“…For example, the smartphonebased speech assistant (e.g. ProxiTalk [57]), and wearable sensor-enabled activity recognition (IMUTube [34], MITIER [8]). Notably, there is a growing trend to bring deep learning (e.g.…”

Section: Introductionmentioning

confidence: 99%

AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications

Liu,

Guo,

et al. 2021

Preprint

View full text Add to dashboard Cite

There are many deep learning (e.g. DNN) powered mobile and wearable applications today continuously and unobtrusively sensing the ambient surroundings to enhance all aspects of human lives. To enable robust and private mobile sensing, DNN tends to be deployed locally on the resource-constrained mobile devices via model compression. The current practice either hand-crafted DNN compression techniques, i.e., for optimizing DNN-relative performance (e.g. parameter size), or on-demand DNN compression methods, i.e., for optimizing hardware-dependent metrics (e.g. latency), cannot be locally online because they require offline retraining to ensure accuracy. Also, none of them have correlated their efforts with runtime adaptive compression to consider the dynamic nature of deployment context of mobile applications. To address those challenges, we present AdaSpring, a context-adaptive and self-evolutionary DNN compression framework. It enables the runtime adaptive DNN compression locally online. Specifically, it presents the ensemble training of a retraining-free and self-evolutionary network to integrate multiple alternative DNN compression configurations (i.e., compressed architectures and weights). It then introduces the runtime search strategy to quickly search for the most suitable compression configurations and evolve the corresponding weights. With evaluation on five tasks across three platforms and a real-world case study, experiment outcomes show that AdaSpring obtains up to 3.1× latency reduction, 4.2× energy efficiency improvement in DNNs, compared to hand-crafted compression techniques, while only incurring ≤ 6.2𝑚𝑠 runtime-evolution latency.CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing systems and tools.

show abstract

IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition

Cited by 3 publications

References 64 publications

Contrastive Predictive Coding for Human Activity Recognition

Contrastive Predictive Coding for Human Activity Recognition

Invariant Feature Learning for Sensor-based Human Activity Recognition

AdaSpring: Context-adaptive and Runtime-evolutionary Deep Model Compression for Mobile Applications

Contact Info

Product

Resources

About