Probing Negative Sampling for Contrastive Learning to Learn Graph Representations

Chen, Shiyi; Wang, Ziao; Zhang, Xinni; Zhang, Xiaofeng; Peng, Dan

doi:10.1007/978-3-030-86520-7_27

Cited by 14 publications

(35 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the generalization bounds there enjoy a linear dependency on k, which would not be effective if k is large. Moreover, this is not consistent with many studies which show a large number of negative examples [11,21,23,38] is necessary for good generalization performance. For example, the work [20] used 65536 negative examples in unsupervised visual representation learning, for which the existing analysis requires n ≥ (65536) 2 d training examples to get non-vacuous bounds [3].…”

Section: Introductioncontrasting

confidence: 72%

“…The performance of machine learning (ML) models often depends largely on the representation of data, which motivates a resurgence of contrastive representation learning (CRL) to learn a representation function f : X → R d from unsupervised data [11,20,23]. The basic idea is to pull together similar pairs (x, x + ) and push apart disimilar pairs (x, x − ) in an embedding space, which can be formulated as minimizing the following objective [11,32]…”

Section: Introductionmentioning

confidence: 99%

“…The hope is that the learned representation f (x) would capture the latent structure and be beneficial to other downstream learning tasks [3,40]. CRL has achieved impressive empirical performance in advancing the state-of-the-art performance in various domains such as computer vision [10,10,11,20] and natural language processing [8,15,33].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Generalization Analysis for Contrastive Representation Learning

Lei¹,

Yang²,

Ying³

2023

Preprint

View full text Add to dashboard Cite

Recently, contrastive learning has found impressive success in advancing the state of the art in solving various machine learning tasks. However, the existing generalization analysis is very limited or even not meaningful. In particular, the existing generalization error bounds depend linearly on the number k of negative examples while it was widely shown in practice that choosing a large k is necessary to guarantee good generalization of contrastive learning in downstream tasks. In this paper, we establish novel generalization bounds for contrastive learning which do not depend on k, up to logarithmic terms. Our analysis uses structural results on empirical covering numbers and Rademacher complexities to exploit the Lipschitz continuity of loss functions. For self-bounding Lipschitz loss functions, we further improve our results by developing optimistic bounds which imply fast rates in a low noise condition. We apply our results to learning with both linear representation and nonlinear representation by deep neural networks, for both of which we derive Rademacher complexity bounds to get improved generalization bounds.

show abstract

Section: Introductioncontrasting

confidence: 72%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Generalization Analysis for Contrastive Representation Learning

Lei¹,

Yang²,

Ying³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Our CL and SCL losses enforce consistency of the network across subsequent training epochs, which favors invariance of the network outputs to the data augmentation. This can be connected to recent trends of self-supervised learning for instance discrimination, ensuring that the embeddings of data-augmented versions of an instance are closer in embedding space than the embeddings of different instances [48,36,18,20,6]. In the fully annotated multilabel image classification setting, [15] encourages consistency of the spatial activations of the network among two data augmentations of an image, akin to a spatial extension of the Π-model [29].…”

Section: Related Workmentioning

confidence: 99%

Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations

Verelst

Rubenstein

Eichner

et al. 2023

2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

Multi-label image classification is more applicable "in the wild" than single-label classification, as natural images usually contain multiple objects. However, exhaustively annotating images with every object of interest is costly and time-consuming. We train multi-label classifiers from datasets where each image is annotated with a single positive label only. As the presence of all other classes is unknown, we propose an Expected Negative loss that builds a set of expected negative labels in addition to the annotated positives. This set is determined based on prediction consistency, by averaging predictions over consecutive training epochs to build robust targets. Moreover, the 'crop' data augmentation leads to additional label noise by cropping out the single annotated object. Our novel spatial consistency loss improves supervision and ensures consistency of the spatial feature maps by maintaining per-class running-average heatmaps for each training image. We use MS-COCO, Pascal VOC, NUS-WIDE and CUB-Birds datasets to demonstrate the gains of the Expected Negative loss in combination with consistency and spatial consistency losses. We also demonstrate improved multi-label classification mAP on ImageNet-1K using the ReaL multilabel validation set.

show abstract

“…Self-supervised learning (SSL) is a learning paradigm that presents itself as a more scalable approach to pretrain models for transfer learning with less human labeled data [2,3,9,11]. SSL can be understood as a two-step approach: a) learning data representation from solving a proxy or pretext task using automatically generated pseudolabels from raw-unlabeled data, followed by b) fine-tuning of the learned features on the actual downstream task, i.e.…”

Section: Related Workmentioning

confidence: 99%

Self-supervised Learning for Sonar Image Classification

Preciado-Grijalva

Wehbe

Firvida

et al. 2022

2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)

View full text Add to dashboard Cite

Self-supervised learning has proved to be a powerful approach to learn image representations without the need of large labeled datasets. For underwater robotics, it is of great interest to design computer vision algorithms to improve perception capabilities such as sonar image classification. Due to the confidential nature of sonar imaging and the difficulty to interpret sonar images, it is challenging to create public large labeled sonar datasets to train supervised learning algorithms. In this work, we investigate the potential of three self-supervised learning methods (RotNet, Denoising Autoencoders, and Jigsaw) to learn high-quality sonar image representation without the need of human labels. We present pre-training and transfer learning results on real-life sonar image datasets. Our results indicate that self-supervised pre-training yields classification performance comparable to supervised pre-training in a few-shot transfer learning setup across all three methods. Code and self-supervised pre-trained models are be available at agrija9/ssl-sonar-images.

show abstract

Probing Negative Sampling for Contrastive Learning to Learn Graph Representations

Cited by 14 publications

References 9 publications

Generalization Analysis for Contrastive Representation Learning

Generalization Analysis for Contrastive Representation Learning

Spatial Consistency Loss for Training Multi-Label Classifiers from Single-Label Annotations

Self-supervised Learning for Sonar Image Classification

Contact Info

Product

Resources

About