Sense and Learn: Self-supervision for omnipresent sensors

Saeed, Aaqib; Ungureanu, Victor; Gfeller, Beat

doi:10.1016/j.mlwa.2021.100152

Cited by 22 publications

(32 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The objective is to solve multiple binary classification problems to detect if the sample has been transformed or is a clean example. The sense and learn framework [120] proposed a range of pretext tasks to teach models based on a wide variety of inputs, as in some cases utilising transformations without domain knowledge can lead to unintended biases. Specifically, the tasks of blend detection, odd segment recognition, modality denoising and feature prediction from a masked window provide an effective, scalable and inexpensive way to acquire pseudo labels from the signal itself.…”

Section: Pretext Taskmentioning

confidence: 99%

See 1 more Smart Citation

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Deldari¹,

Xue²,

Saeed³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, Self-Supervised Representation Learning (SSRL) has attracted much attention in the field of computer vision, speech, natural language processing (NLP), and recently, with other types of modalities, including time series from sensors. The popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training. Acquiring annotated data can be a difficult and costly process. Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models using supervisory signals that have been freely obtained from the raw data. Unlike existing reviews of SSRL that have pre-dominately focused upon methods in the fields of CV or NLP for a single modality, we aim to provide the first comprehensive review of multimodal self-supervised learning methods for temporal data.To this end, we 1) provide a comprehensive categorization of existing SSRL methods, 2) introduce a generic pipeline by defining the key components of a SSRL framework, 3) compare existing models in terms of their objective function, network architecture and potential applications, and 4) review existing multimodal techniques in each category and various modalities. Finally, we present existing weaknesses and future opportunities. We believe our work develops a perspective on the requirements of SSRL in domains that utilise multimodal and/or temporal data.

show abstract

Section: Pretext Taskmentioning

confidence: 99%

“…In the time-series realm, [120] used symmetric triple-based objectives for learning representations from various sensor input modalities. [67] applied contrastive predictive coding [105] for HAR.…”

Section: Multi-task Lossesmentioning

confidence: 99%

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Deldari¹,

Xue²,

Saeed³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Contrastive learning methods mostly have been proposed for single modality data across a variety of applications including computer vision [10,12,32,37], audio processing [32,62,90], natural language processing [23,28], sensor data analytics [13,18,22,24,47,64,65,78]. Recently [19] has reviewed existing self-supervised learning approaches for multimodal and temporal data including contrastive learning models.…”

Section: Multimodal Contrastive Learningmentioning

confidence: 99%

“…In this regard, several recent self-supervised deep learning models are proposed to learn from two or three modalities together, such as audio-video [51,53], audio-text [40], vision-text [41,49,58], audio-vision-text [1,2,34]. Despite the undeniable need of considering data from many different sensor sources [7] and the difficulty of annotation of fast-generating time-series data in ubiquitous applications, applications of self-supervised contrastive learning in cross-modal time-series data is still not well-studied yet and they vastly focus on surrogate tasks [65], multimodal learning [22,36], or generative modeling [92].…”

Section: Introductionmentioning

confidence: 99%

COCOA: Cross Modality Contrastive Learning for Sensor Data

Deldari,

Xue,

Saeed

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labeled data, and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of different sensor modalities. While existing CL methods are limited to learning from one or two data sources, we propose COCOA (Cross mOdality COntrastive leArning), a self-supervised model that employs a novel objective function to learn quality representations from multisensor data by computing the cross-correlation between different data modalities and minimizing the similarity between irrelevant instances. We evaluate the effectiveness of COCOA against eight recently introduced stateof-the-art self-supervised models, and two supervised baselines across five public datasets. We show that COCOA achieves superior classification performance to all other approaches. Also, COCOA is far more label-efficient than the other baselines including the fully supervised model using only one-tenth of available labeled data.

show abstract

“…Other self-supervised approaches in activity recognition include signal correspondence learning with the wavelet transform in a federated learning framework [59], utilizing eight auxiliary tasks for learning high-level features [60], contrastive learning for activity recognition in healthcare-based applications [66]. In most of the unsupervised and self-supervised approaches including Autoencoders, Multi-task self-supervision, Masked reconstruction, and CPC, the source and target datasets are typically the same for most experiments, albeit the pre-training does not use label information.…”

Section: Self-supervised Learning In Human Activity Recognitionmentioning

confidence: 99%

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Haresamudram¹,

Essa²,

Plötz³

2022

Preprint

View full text Add to dashboard Cite

The emergence of self-supervised learning in the field of wearables-based human activity recognition (HAR) has opened up opportunities to tackle the most pressing challenges in the field, namely to exploit unlabeled data to derive reliable recognition systems from only small amounts of labeled training samples. Furthermore, self-supervised methods enable a host of new application domains such as, for example, domain adaptation and transfer across sensor positions, activities etc. As such, self-supervision, i.e., the paradigm of 'pretrain-then-finetune' has the potential to become a strong alternative to the predominant end-to-end training approaches, let alone the classic activity recognition chain with explicit focus on designing feature representations for sensor data. Recently a number of contributions have been made that introduced self-supervised learning into the field of HAR, including, Multi-task self-supervision, Masked Reconstruction, CPC to name but a few. With the initial success of these methods, the time has come for a systematic inventory and analysis of the potential self-supervised learning has for the field. This paper provides exactly that. We assess the progress of self-supervised HAR research by introducing a framework that performs a multi-faceted exploration of model performance. We organize the framework into three dimensions, each containing three constituent criteria, and utilize it to assess state-of-the-art self-supervised learning methods in a large empirical study on a curated set of nine diverse benchmarks. This exploration leads us to the formulation of insights into the properties of these techniques and to establish their value towards learning representations for diverse scenarios. Based on our findings we call upon the community to join our efforts and to contribute towards shaping the evaluation of the ongoing paradigm change in modeling human activities from body-worn sensor data.CCS Concepts: • Human-centered computing → Ubiquitous and mobile computing; Empirical studies in ubiquitous and mobile computing; • Computing methodologies → Machine learning.

show abstract

Sense and Learn: Self-supervision for omnipresent sensors

Cited by 22 publications

References 23 publications

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

COCOA: Cross Modality Contrastive Learning for Sensor Data

Assessing the State of Self-Supervised Human Activity Recognition using Wearables

Contact Info

Product

Resources

About