AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries

Hu, Qianjiang; Wang, Xiao; Hu, Wei; Qi, Guo-Jun

doi:10.1109/cvpr46437.2021.00113

Cited by 86 publications

(69 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Synthesizing harder negatives in latent space using Mixup [53] has also been considered [25] but does not take an adversarial perspective. Other work, AdCo [20], also takes an adversarial viewpoint in latent space (see Tab. 1 for comparison to IFM).…”

Section: Visualizing Implicit Feature Modificationmentioning

confidence: 99%

“…1 benchmarks IFM on ImageNet100 [44] using MoCo-v2, observing improvements of 0.9%. We also compare results on ImageNet100 to AdCo [20], another adversarial method for contrastive learning. We adopt the official code and use the exact same training and finetuning hyperparameters as for MoCo-v2 and IFM.…”

Section: Performance On Downstream Tasksmentioning

confidence: 99%

“…For the AdCo-specific hyperparamters -negatives learning rate lr neg and negatives temperature τ neg -we use a grid search and report the best result. We search over all combinations lr neg ∈ {1, 2, 3, 4} and τ neg ∈ {0.02, 0.1}, which includes the AdCo default ImageNet1K recommendations lr neg = 3 and τ neg = 0.02 [20]. That the resulting AdCo performance of 78.9% is slightly below MoCo-v2 serves to highlight the sensitivity of AdCo to its method-specific hyperparameters, and the challenge of tuning them.…”

Section: Performance On Downstream Tasksmentioning

confidence: 99%

“…For the AdCo specific parameters we perform a simple grid search for the following two hyperparameters: negatives learning rate lr neg and negatives temperature τ neg . We search over all combinations lr neg ∈ {1, 2, 3, 4} and τ neg ∈ {0.02, 0.1}, which includes the AdCo default ImageNet1K recommendations lr neg = 3 and τ neg = 0.02 [20]. The result reported for AdCo in Tab.…”

Section: C41 Imagenet100mentioning

confidence: 99%

See 3 more Smart Citations

Can contrastive learning avoid shortcut solutions?

Robinson¹,

Li²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

The generalization of representations learned via contrastive learning depends crucially on what features of the data are extracted. However, we observe that the contrastive loss does not always sufficiently guide which features are extracted, a behavior that can negatively impact the performance on downstream tasks via "shortcuts", i.e., by inadvertently suppressing important predictive features. We find that feature extraction is influenced by the difficulty of the so-called instance discrimination task (i.e., the task of discriminating pairs of similar points from pairs of dissimilar ones). Although harder pairs improve the representation of some features, the improvement comes at the cost of suppressing previously well represented features. In response, we propose implicit feature modification (IFM), a method for altering positive and negative samples in order to guide contrastive models towards capturing a wider variety of predictive features. Empirically, we observe that IFM reduces feature suppression, and as a result improves performance on vision and medical imaging tasks. The code is available at: https://github.com/joshr17/IFM.

show abstract

Section: Visualizing Implicit Feature Modificationmentioning

confidence: 99%

Section: Performance On Downstream Tasksmentioning

confidence: 99%

Section: Performance On Downstream Tasksmentioning

confidence: 99%

Section: C41 Imagenet100mentioning

confidence: 99%

See 2 more Smart Citations

Can contrastive learning avoid shortcut solutions?

Robinson¹,

Li²,

Yu³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…An important paradigm called contrastive learning aims to train an encoder to be contrastive between the representations of positive samples and negative samples [18], [47], [48], [49], [50]. Recent contrastive learning frameworks for graph data can be divided into two categories [7]: context-instance contrast and context-context contrast.…”

Section: Graph Contrastive Learningmentioning

confidence: 99%

Self-Supervised Graph Representation Learning via Topology Transformations

Gao¹,

Hu²,

Qi³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data to enable the wide applicability of Graph Convolutional Neural Networks (GCNNs). We formalize the proposed model from an information-theoretic perspective, by maximizing the mutual information between topology transformations and node representations before and after the transformations. We derive that maximizing such mutual information can be relaxed to minimizing the cross entropy between the applied topology transformation and its estimation from node representations. In particular, we seek to sample a subset of node pairs from the original graph and flip the edge connectivity between each pair to transform the graph topology. Then, we self-train a representation encoder to learn node representations by reconstructing the topology transformations from the feature representations of the original and transformed graphs. In experiments, we apply the proposed model to the downstream node and graph classification tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.

show abstract

AdCSE: An Adversarial Method for Contrastive Learning of Sentence Embeddings

Duan

Xie

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries

Cited by 86 publications

References 16 publications

Can contrastive learning avoid shortcut solutions?

Can contrastive learning avoid shortcut solutions?

Self-Supervised Graph Representation Learning via Topology Transformations

AdCSE: An Adversarial Method for Contrastive Learning of Sentence Embeddings

Contact Info

Product

Resources

About