Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

Nozawa, Kento; Sato, Issei

doi:10.48550/arxiv.2102.06866

Cited by 4 publications

(5 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Along with various subject matter improvements [26-29, 8, 9], contrastive learners now provide comprehensive solutions for self-supervised learning. Despite encouraging progress, there are still many unresolved issues with contrastive learning, with the following three particularly relevant to this investigation: (i) contrastive learners need a very large number of negative samples to work well; (ii) the bias, variance, and performance tradeoffs are in debate [13,21,30]; and, crucially, (iii) there is a lack of training diagnostic tools for contrastive learning. Among these three, (i) is most concerning, as it implies training can be very expensive, while the needed massive computational resources may not be widely available.…”

Section: Introductionmentioning

confidence: 99%

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

Chen¹,

Gan²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

InfoNCE-based contrastive representation learners, such as SimCLR [1], have been tremendously successful in recent years. However, these contrastive schemes are notoriously resource demanding, as their effectiveness breaks down with smallbatch training (i.e., the log-K curse, whereas K is the batch-size). In this work, we reveal mathematically why contrastive learners fail in the small-batch-size regime, and present a novel simple, non-trivial contrastive objective named FlatNCE, which fixes this issue. Unlike InfoNCE, our FlatNCE no longer explicitly appeals to a discriminative classification goal for contrastive learning. Theoretically, we show FlatNCE is the mathematical dual formulation of InfoNCE, thus bridging the classical literature on energy modeling; and empirically, we demonstrate that, with minimal modification of code, FlatNCE enables immediate performance boost independent of the subject-matter engineering efforts. The significance of this work is furthered by the powerful generalization of contrastive learning techniques, and the introduction of new tools to monitor and diagnose contrastive training. We substantiate our claims with empirical evidence on CIFAR10 and ImageNet datasets, where FlatNCE consistently outperforms InfoNCE.Preprint. Under review.

show abstract

Section: Introductionmentioning

confidence: 99%

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

Chen¹,

Gan²,

Li³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…InfoNCE loss was identified to have the hardnessaware property, which is critical for optimization [64] and preventing collapse by instance de-correlation [1]. [15], [34], [36], [49], [62], [66], [69] have demonstrated that hard negative samples mining strategies can be beneficial for better performance over the baselines. Notably, [65] identified CL form alignment and uniformity of feature space which benefits downstream tasks.…”

Section: Related Workmentioning

confidence: 99%

“…In the view of optimization, the hardness-aware property puts more weight into optimizing negative pairs that have high similarities. This way is influenced by hard examples mining and has proven to be effective [4], [36], [49], [62], [66], [72].…”

Section: B Hardness-aware Property In Dimclmentioning

confidence: 99%

DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning

et al. 2023

View full text Add to dashboard Cite

Self-supervised learning (SSL) has gained remarkable success, for which contrastive learning (CL) plays a key role. However, the recent development of new non-CL frameworks has achieved comparable or better performance, prompting researchers to enhance these frameworks further. Assimilating CL into non-CL frameworks has been thought to be beneficial, but empirical evidence indicates no visible improvements. In view of that, this paper proposes a strategy of performing CL along the dimensional direction instead of along the batch direction as done in convention contrastive learning, named Dimensional Contrastive Learning (DimCL). DimCL aims to enhance the feature diversity, and it can serve as a regularizer to prior SSL frameworks. DimCL has been found to be effective, and the hardness-aware property is identified as a critical reason for its success. Extensive experimental results reveal that assimilating DimCL into SSL frameworks leads to performance improvement by a non-trivial margin on various datasets and backbone architectures.

show abstract

“…Given its empirical success, there has been significant interest in the theory of contrastive learning, from various perspectives. Most relevant to us are learning theoretic analyses [Arora et al, 2019, Tosh et al, 2021b,a, HaoChen et al, 2021, Wang et al, 2022 and their follow ups [Nozawa andSato, 2021, Ash et al, 2021]. These study the downstream linear classification performance of learned representation, by making assumptions about the data and augmentation distributions; we discuss these in more detail in Section 2.…”

Section: Related Workmentioning

confidence: 99%

Understanding Contrastive Learning Requires Incorporating Inductive Biases

Saunshi¹,

Ash²,

Goel³

et al. 2022

Preprint

View full text Add to dashboard Cite

Contrastive learning is a popular form of self-supervised learning that encourages augmentations (views) of the same input to have more similar representations compared to augmentations of different inputs. Recent attempts to theoretically explain the success of contrastive learning on downstream classification tasks prove guarantees depending on properties of augmentations and the value of contrastive loss of representations. We demonstrate that such analyses, that ignore inductive biases of the function class and training algorithm, cannot adequately explain the success of contrastive learning, even provably leading to vacuous guarantees in some settings. Extensive experiments on image and text domains highlight the ubiquity of this problem -different function classes and algorithms behave very differently on downstream tasks, despite having the same augmentations and contrastive losses. Theoretical analysis is presented for the class of linear representations, where incorporating inductive biases of the function class allows contrastive learning to work with less stringent conditions compared to prior analyses.

show abstract

Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning

Cited by 4 publications

References 39 publications

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

Simpler, Faster, Stronger: Breaking The log-K Curse On Contrastive Learners With FlatNCE

DimCL: Dimensional Contrastive Learning for Improving Self-Supervised Learning

Understanding Contrastive Learning Requires Incorporating Inductive Biases

Contact Info

Product

Resources

About