Contrastive Multiview Coding

Tian, Yonglong; Krishnan, Dilip; Isola, Phillip

doi:10.1007/978-3-030-58621-8_45

Cited by 1,288 publications

(1,044 citation statements)

References 66 publications

Supporting

Mentioning

915

Contrasting

Order By: Relevance

“…To encourage the encoder to learn a richer embedding, and to mitigate the need to train separate critics for each position of k , we modify the bilinear critic used in Oord et al (2018), and instead use a parameterless dot product critic for f (Chen et al, 2020). Rather than using a memory bank (Wu et al, 2018; Tian et al, 2019; He et al, 2019), we draw “fake” samples from p ( z ) and p ( c ) using other z t + k and c t from other samples in the same batch (Chen et al, 2020). That is, the diagonal of the output of the dot-product critic is the “correct pairing” of and at a given t and k and the softmax is computed using all off-diagonal entries to draw the N −1 “fake” samples from the noise distribution p ( z ′).…”

Section: Methodsmentioning

confidence: 99%

“…It is possible to use a variational approach to estimate a bound on the mutual information between continuous, high-dimensional quantities (Donsker & Varadhan, 1983; Nguyen et al, 2010; Alemi et al, 2016; Belghazi et al, 2018; Oord et al, 2018; Poole et al, 2019). Recent works capture this intuition to yield self-supervised embeddings in the modalities of imaging (Oord et al, 2018; Hjelm et al, 2018; Bachman et al, 2019; Tian et al, 2019; Hénaff et al, 2019; Löwe et al, 2019; He et al, 2019; Chen et al, 2020; Tian et al, 2020; Wang & Isola, 2020), text (Rivière et al, 2020; Oord et al, 2018; Kong et al, 2019), and audio (Löwe et al, 2019; Oord et al, 2018), with high empirical downstream performance.…”

Section: Background and Related Workmentioning

confidence: 99%

“…Following previous work in self-supervised learning, we also assess embeddings using a linear model (Oord et al, 2018; Hénaff et al, 2019; He et al, 2019; Bachman et al, 2019; Tian et al, 2019; 2020). Note that as compared to the neural network finetuning head evaluation, we use static embeddings extracted from the model without end-to-end optimization.…”

Section: Data and Downstream Evaluationmentioning

confidence: 99%

See 2 more Smart Citations

Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Zhang

Ghassemi

et al. 2020

Preprint

View full text Add to dashboard Cite

Pretrained embedding representations of biological sequences which capture meaningful properties can alleviate many problems associated with supervised learning in biology. We apply the principle of mutual information maximization between local and global information as a self-supervised pretraining signal for protein embeddings. To do so, we divide protein sequences into fixed size fragments, and train an autoregressive model to distinguish between subsequent fragments from the same protein and fragments from random proteins. Our model, CPCProt, achieves comparable performance to state-of-the-art self-supervised models for protein sequence embeddings on various downstream tasks, but reduces the number of parameters down to 0.9% to 8.9% of benchmarked models. Further, we explore how downstream assessment protocols affect embedding evaluation, and the effect of contrastive learning hyperparameters on empirical performance. We hope that these results will inform the development of contrastive learning methods in protein biology and other modalities.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Background and Related Workmentioning

confidence: 99%

Section: Data and Downstream Evaluationmentioning

confidence: 99%

See 1 more Smart Citation

Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Zhang

Ghassemi

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Different contrastive methods differ from each other in terms of the approach to resolving this intractability, the definitions of what different views are, and the exact implementations of the contrastive loss form. Such methods include instance recognition (IR) ( 40 ), contrastive multiview coding (CMC) ( 39 ), momentum contrast (MoCo) ( 42 ), simple contrastive learning of representation (SimCLR) ( 43 ), and local aggregation (LA) ( 41 ). For example, the IR method involves maintaining running averages of embeddings for all inputs (called the “memory bank”) across the training time and replacing

and

with the corresponding running-average embeddings

and

.…”

Section: Unsupervised Learning Algorithmsmentioning

confidence: 99%

Unsupervised neural network models of the ventral visual stream

Zhuang

Yan

Nayebi

et al. 2021

Proc. Natl. Acad. Sci. U.S.A.

236

197

View full text Add to dashboard Cite

Deep neural networks currently provide the best quantitative models of the response patterns of neurons throughout the primate ventral visual stream. However, such networks have remained implausible as a model of the development of the ventral stream, in part because they are trained with supervised methods requiring many more labels than are accessible to infants during development. Here, we report that recent rapid progress in unsupervised learning has largely closed this gap. We find that neural network models learned with deep unsupervised contrastive embedding methods achieve neural prediction accuracy in multiple ventral visual cortical areas that equals or exceeds that of models derived using today’s best supervised methods and that the mapping of these neural network models’ hidden layers is neuroanatomically consistent across the ventral stream. Strikingly, we find that these methods produce brain-like representations even when trained solely with real human child developmental data collected from head-mounted cameras, despite the fact that these datasets are noisy and limited. We also find that semisupervised deep contrastive embeddings can leverage small numbers of labeled examples to produce representations with substantially improved error-pattern consistency to human behavior. Taken together, these results illustrate a use of unsupervised learning to provide a quantitative model of a multiarea cortical brain system and present a strong candidate for a biologically plausible computational theory of primate sensory learning.

show abstract

“…the best performance of NPID over the validation set was with k = 25). We follow the unsupervised as well as self-supervised representation learning literatures, 15 – 18 , 24 where cosine similarity has been used as a metric to describe the distance between two features on a unit sphere space.…”

Section: Methodsmentioning

confidence: 99%

Meibography Phenotyping and Classification From Unsupervised Discriminative Feature Learning

Yeh

Lin

2021

Trans. Vis. Sci. Tech.

View full text Add to dashboard Cite

Purpose The purpose of this study was to develop an unsupervised feature learning approach that automatically measures Meibomian gland (MG) atrophy severity from meibography images and discovers subtle relationships between meibography images according to visual similarity. Methods One of the latest unsupervised learning approaches is to apply feature learning based on nonparametric instance discrimination (NPID), a convolutional neural network (CNN) backbone model trained to encode meibography images into 128-dimensional feature vectors. The network aims to learn a similarity metric across all instances (e.g. meibography images) and groups visually similar instances together. A total of 706 meibography images with corresponding meiboscores were collected and annotated for the use of network learning and performance evaluation. Results Four hundred ninety-seven meibography images were used for network learning and tuning, whereas the remaining 209 images were used for network model evaluations. The proposed nonparametric instance discrimination approach achieved 80.9% meiboscore grading accuracy on average, outperforming the clinical team by 25.9%. Additionally, a 3D feature visualization and agglomerative hierarchical clustering algorithms were used to discover the relationship between meibography images. Conclusions The proposed NPID approach automatically analyses MG atrophy severity from meibography images without prior image annotations, and categorizes the gland characteristics through hierarchical clustering. This method provides quantitative information on the MG atrophy severity based on the analysis of phenotypes. Translational Relevance The study presents a Meibomian gland atrophy evaluation method for meibography images based on unsupervised learning. This method may be used to aid diagnosis and management of Meibomian gland dysfunction without prior image annotations, which require time and resources.

show abstract

Contrastive Multiview Coding

Cited by 1,288 publications

References 66 publications

Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Self-Supervised Contrastive Learning of Protein Representations By Mutual Information Maximization

Unsupervised neural network models of the ventral visual stream

Meibography Phenotyping and Classification From Unsupervised Discriminative Feature Learning

Contact Info

Product

Resources

About