Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Abstract

Recently, Self-Supervised Representation Learning (SSRL) has attracted much attention in the field of computer vision, speech, natural language processing (NLP), and recently, with other types of modalities, including time series from sensors. The popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training. Acquiring annotated data can be a difficult and costly process. Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models using supervisory signals that have been freely obtained from the raw data. Unlike existing reviews of SSRL that have pre-dominately focused upon methods in the fields of CV or NLP for a single modality, we aim to provide the first comprehensive review of multimodal self-supervised learning methods for temporal data.To this end, we 1) provide a comprehensive categorization of existing SSRL methods, 2) introduce a generic pipeline by defining the key components of a SSRL framework, 3) compare existing models in terms of their objective function, network architecture and potential applications, and 4) review existing multimodal techniques in each category and various modalities. Finally, we present existing weaknesses and future opportunities. We believe our work develops a perspective on the requirements of SSRL in domains that utilise multimodal and/or temporal data.

Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding

“…In the field of healthcare and physiological applications, SSL faces additional challenges due to the heterogeneous nature of data acquired from different sensors with varying characteristics, sampling rates, and resolutions (Deldari et al, 2022a). The dynamic nature of real-world situations further complicates the aggregation and compression of multimodal sensor data into a coherent global embedding suitable for downstream tasks.…”

Applications of Self-Supervised Learning to Biomedical Signals: where are we now

“…This section focuses on approaches that are compatible with biosignals and that were encountered during the survey. Various classification schemes have been proposed for pretext tasks, depending on the domain of application [33]. Here, methodologies will be grouped in the following categories: predictive pretext tasks, generative pretext tasks, contrastive learning pretext tasks.…”

Self-supervised deep representation learning of a foundation transformer model enabling efficient ECG-based assessment of cardiac and coronary function with limited labels

“…We have demonstrated the feasibility of predicting impaired MFR using a relatively simple CNN with only 72,929 trainable parameters. We expect that CNN model performance may be further improved by additional machine learning technologies such as larger CNN architectures (41), self-supervised model pre-training (62), and transformer-based frameworks (40,63). A further possible extension would be to train the CNN using only rest ECG waveforms or reduced leads which could enable use of ambulatory ECG monitoring data.…”