Representation Learning with Contrastive Predictive Coding

Oord, Aäron van den; Li, Yazhe; Vinyals, Oriol

doi:10.48550/arxiv.1807.03748

Cited by 869 publications

(1,456 citation statements)

References 0 publications

Supporting

Mentioning

1,445

Contrasting

Order By: Relevance

“…Training. The model is trained by minimizing the canonical β-VQ-VAE loss without the auxiliary codebook loss (Oord et al, 2019) with β = 0.02 in all of our experiments. The codebook is updated using the EMA update step as proposed in the original paper.…”

Section: Vq-vae Modelmentioning

confidence: 99%

FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control

Dimitri¹,

Biggio²,

Kilcher³

et al. 2022

Preprint

View full text Add to dashboard Cite

Generating music with deep neural networks has been an area of active research in recent years. While the quality of generated samples has been steadily increasing, most methods are only able to exert minimal control over the generated sequence, if any. We propose the self-supervised description-to-sequence task, which allows for fine-grained controllable generation on a global level. We do so by extracting high-level features about the target sequence and learning the conditional distribution of sequences given the corresponding high-level description in a sequence-tosequence modelling setup. We train FIGARO (FIne-grained music Generation via Attentionbased, RObust control) by applying descriptionto-sequence modelling to symbolic music. By combining learned high level features with domain knowledge, which acts as a strong inductive bias, the model achieves state-of-the-art results in controllable symbolic music generation and generalizes well beyond the training distribution.

show abstract

Section: Vq-vae Modelmentioning

confidence: 99%

FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control

Dimitri¹,

Biggio²,

Kilcher³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Contrastive learning is a representation learning method that could refinement the distribution of the feature space to acquire a better semantic representation. It has been outstandingly successful in practice [3][4] [10][32] [64][65] [66] and involves learning transformation-invariant feature representations from unlabeled image data. The common strategy among those works is to pull an anchor towards a "positive" sample in the embedding space and to push the anchor away from many "negative" samples.…”

Section: Online Confusion Category Miningmentioning

confidence: 99%

“…Contrastive learning is a typical discriminative selfsupervised learning [3][4] [9][32] [34][34] [43] method that aims to learn useful representations of the input data without relying on task-specific manual annotations. Recent advances in self-supervised visual representation learning based on contrastive methods show that self-supervised representations outperform their supervised counterparts [5][34] [35][36] [37] on several downstream transfer learning benchmarks.…”

Section: B Contrastive Learningmentioning

confidence: 99%

See 1 more Smart Citation

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Feng¹,

Cao²,

Xue³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning a powerful representation from point clouds is a fundamental and challenging problem in the field of computer vision. Different from images where RGB pixels are stored in the regular grid, for point clouds, the underlying semantic and structural information of point clouds is the spatial layout of the points. Moreover, the properties of challenging incontext and background noise pose more challenges to point cloud analysis. One assumption is that the poor performance of the classification model can be attributed to the indistinguishable embedding feature that impedes the search for the optimal classifier. This work offers a new strategy for learning powerful representations via a contrastive learning approach that can be embedded into any point cloud classification network. First, we propose a supervised contrastive classification (SCC) method to implement embedding feature distribution refinement by improving the intra-class compactness and inter-class separability. Second, to solve the confusion problem caused by small inter-class variations between some similar-looking categories, we propose a confusion-prone class mining (CPCM) strategy to alleviate the confusion effect. Finally, considering that outliers of the sample clusters in the embedding space may cause performance degradation, we design an entropy-aware attention (EAA) module with information entropy theory to identify the outlier cases and the unstable samples by measuring the uncertainty of predicted probability. We assume that the outlier and unstable samples need to be ignored and enhanced, respectively, to enable the model to learn more robust decision boundaries. The results of extensive experiments demonstrate that our method outperforms the state-of-the-art approaches by achieving 82.9% accuracy on the real-world ScanObjectNN dataset and substantial performance gains up to 2.9% in DCGNN, 3.1% in PointNet++, and 2.4% in GBNet. The code will be available at https://github.com/YangFengSEU/CEDR.

show abstract

“…where f (x x x) ∈ R p is the neural network representation for an input x x x, and x x x, x x x + are drawn from the augmentations of the same natural data point, and x x x, x x x − are two augmentations generated independently either from the same data point or two different data points. The above loss function is similar to many standard contrastive loss functions (Oord et al, 2018;Sohn, 2016;Wu et al, 2018), including SimCLR (Chen et al, 2020) that we will use in our experiments. Minimizing this objective leads to representations with provable accuracy guarantees under linear probe evaluation.…”

Section: Self-supervised Contrastive Learningmentioning

confidence: 99%

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

Xue¹,

Whitecross²,

Mirzasoleiman³

2022

Preprint

View full text Add to dashboard Cite

Self-supervised contrastive learning has recently been shown to be very effective in preventing deep networks from overfitting noisy labels. Despite its empirical success, the theoretical understanding of the effect of contrastive learning on boosting robustness is very limited. In this work, we rigorously prove that the representation matrix learned by contrastive learning boosts robustness, by having: (i) one prominent singular value corresponding to every sub-class in the data, and remaining significantly smaller singular values; and (ii) a large alignment between the prominent singular vector and the clean labels of each subclass. The above properties allow a linear layer trained on the representations to quickly learn the clean labels, and prevent it from overfitting the noise for a large number of training iterations. We further show that the low-rank structure of the Jacobian of deep networks pre-trained with contrastive learning allows them to achieve a superior performance initially, when fine-tuned on noisy labels. Finally, we demonstrate that the initial robustness provided by contrastive learning enables robust training methods to achieve stateof-the-art performance under extreme noise levels, e.g., an average of 27.18% and 15.58% increase in accuracy on CIFAR-10 and CIFAR-100 with 80% symmetric noisy labels, and 4.11% increase in accuracy on WebVision. Noise Type Sym Asym Sym Asym Noise Ratio 20 50 80 40 20 50 80 40 F-correction 85.1 ± 0.4 76.0 ± 0.2 34.8 ± 4.5 83.6 ± 2.2 55.8 ± 0.5 43.3 ± 0.7 − 42.3 ± 0.7 Decoupling 86.7 ± 0.3 79.3 ± 0.6 36.9 ± 4.6 75.3 ± 0.8 57.6 ± 0.5 45.7 ± 0.4 − 43.1 ± 0.4 Co-teaching 89.1 ± 0.3 82.1 ± 0.6 16.2 ± 3.2 84.6 ± 2.8 64.0 ± 0.3 52.3 ± 0.4 − 47.7 ± 1.2 MentorNet 88.4 ± 0.5 77.1 ± 0.4 28.9 ± 2.3 77.3 ± 0.8 63.0 ± 0.4 46.4 ± 0.4 − 42.4 ± 0.5 D2L 86.1 ± 0.4 67.4 ± 3.6 10.0 ± 0.1 85.6 ± 1.2 12.5 ± 4.2 5.6 ± 5.4 − 14.1 ± 5.8 INCV 89.7 ± 0.2 84.8 ± 0.3 52.3 ± 3.5 86.0 ± 0.5 60.2 ± 0.2 53.1 ± 0.4 − 50.7 ± 0.2 T-Revision 79.3 ± 0.5 78.5 ± 0.6 36.2 ± 1.6 76.3 ± 0.8 52.4 ± 0.3 37.6 ± 0.3 − 32.3 ± 0.4 L DMI 84.3 ± 0.4 78.8 ± 0.5 20.9 ± 2.2 84.8 ± 0.7 56.8 ± 0.4 42.2 ± 0.5 − 39.5 ± 0.4 Crust * 85.3 ± 0.5 86.8 ± 0.3 33.8 ± 1.3 76.7 ± 3.4 62.9 ± 0.3 55.5 ± 1.1 18.5 ± 0.8 52.5 ± 0.4 Mixup 89.7 ± 0.7 84.5 ± 0.3 40.7 ± 1.1 86.3 ± 0.1 64.0 ± 0.4 53.4 ± 0.5 15.1 ± 0.1 54.4 ± 2.0 ELR * 90.6 ± 0.6 87.7 ± 1.0 69.5 ± 5.0 86.6 ± 2.9 63.6 ± 1.7 52.5 ± 4.2 23.4 ± 1.9 59.7 ± 0.1 CL+E2E * 88.8 ± 0.5 82.8 ± 0.2 72.0 ± 0.3 83.5 ± 0.5 63.5 ± 0.2 56.1 ± 0.3 36.7 ± 0.3 52.4 ± 0.2 CL+Crust * 86.5 ± 0.7 87.6 ± 0.3 77.9 ± 0.3 85.9 ± 0.4 63.0 ± 0.8 58.3 ± 0.1 34.8 ± 1.5 53.3 ± 0.7 CL+Mixup * 90.8 ± 0.2 84.6 ± 0.4 74.8 ± 0.3 87.5 ± 1.3 64.4 ± 0.4 55.5 ± 0.1 30.3 ± 0.4 55.5 ± 0.8 CL+ELR * 91.3 ± 0.0 89.1 ± 0.1 77.7 ± 0.2 89.7 ± 0.3 64.7 ± 0.2 55.6 ± 0.2 35.9 ± 0.3 63.6 ± 0.1

show abstract

Representation Learning with Contrastive Predictive Coding

Cited by 869 publications

References 0 publications

FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control

FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control

Contrastive Embedding Distribution Refinement and Entropy-Aware Attention for 3D Point Cloud Classification

Investigating Why Contrastive Learning Benefits Robustness Against Label Noise

Contact Info

Product

Resources

About