A Survey on Contrastive Self-Supervised Learning

Abstract

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating large-scale datasets. It is capable of adopting self-defined pseudolabels as supervision and use the learned representations for several downstream tasks. Specifically, contrastive learning has recently become a dominant component in self-supervised learning for computer vision, natural language processing (NLP), and other domains. It aims at embedding augmented versions of the same sample close to each other while trying to push away embeddings from different samples. This paper provides an extensive review of self-supervised methods that follow the contrastive approach. The work explains commonly used pretext tasks in a contrastive learning setup, followed by different architectures that have been proposed so far. Next, we present a performance comparison of different methods for multiple downstream tasks such as image classification, object detection, and action recognition. Finally, we conclude with the limitations of the current methods and the need for further techniques and future directions to make meaningful progress.

FragNet, a Contrastive Learning-Based Transformer Model for Clustering, Interpreting, Visualizing, and Navigating Chemical Space

“…A second point is that in the previous work [5], we made no real attempt to manipulate the latent space so as to "disentangle" the input representations, and if one is to begin to understand the working of such "deep" neural networks it is necessary to do so. Of the various strategies available, those using contrastive learning [11,62,66,[169][170][171] seem to be the most apposite. In contrastive learning, one informs the learning algorithm whether two (or more) individual examples come from the same of different classes.…”

“…However, a third development is the recognition that training with such unlabeled data can also be used to optimize the (self-) organization of the latent space itself. A particular objective of one kind of self-organization is one in which individual inputs are used to create a structure in which similar input examples are also closer to each other in the latent space; this is commonly referred to as self-supervised [12,[55][56][57], or contrastive [58][59][60][61][62][63][64][65][66], learning. In image processing this is often performed by augmenting training data with rotated or otherwise distorted versions of a given image, which then retain the same class membership or "similarity" despite appearing very different [61,[67][68][69].…”

Data-Driven Remaining Useful Life Estimation for Milling Process: Sensors, Algorithms, Datasets, and Future Directions

“…The authors of [180] propose a novel transfer learning technique based on multiple layer perceptron (MLP) for dissimilar data distribution problems in RUL prediction of bearing machinery. However, many scopes to use selfsupervised learning [199] and self-supervised contrastive learning [200] algorithms are fine-tuned on limited data. It is proved that self-supervised algorithms work better in data scarcity situations and where data labeling is timeconsuming and costly.…”

Contrastive Learning for Fault Detection and Diagnostics in the Context of Changing Operating Conditions and Novel Fault Types

“…Contrastive learning is a discriminative approach that aims to group semantically similar samples close to each other in the feature space while pushing semantically dissimilar samples far apart from each other [7,8]. To achieve this, a contrastive loss is formulated based on a similarity metric quantifying how close different features are [9].…”

“…The accelerometer measurements are recorded under four different loads 0, 1, 2, 3, which correspond to different operating conditions in our case studies. Ten different health conditions of the bearing are represented in the dataset (see Table 1): Healthy condition (N), three different fault types (inner race faults [IR], outer race faults [OR], and ball faults [B]), and three different fault severities for each of the fault types (7,14,21). The sample dataset was collected from the CWRU dataset with sampling frequency of 48 kHz.…”