Hard Negative Examples are Hard, but Useful

Xuan, Hong; Stylianou, Abby; Liu, Xiaotong; Pless, Robert

doi:10.1007/978-3-030-58568-6_8

Cited by 66 publications

(32 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As with previous studies [1,14,24], CL benefits from hard negative samples, i.e. samples close to the anchor node such that cannot be distinguished easily.…”

Section: Structure-aware Hard Negative Miningmentioning

confidence: 75%

“…However, the previous scheme assumes that all negative samples make equal contribution to the CL objective. Previous research in metric learning [14] and visual representation learning [1,24] has established that the hard negative sample is of particular concern in effective CL. To be specific, the more similar a negative sample to its anchor, the more helpful it is for learning effective representatives.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning

Zhu¹,

Xu²,

Cui³

et al. 2021

Preprint

View full text Add to dashboard Cite

Recently, heterogeneous Graph Neural Networks (GNNs) have become a de facto model for analyzing HGs, while most of them rely on a relative large number of labeled data. In this work, we investigate Contrastive Learning (CL), a key component in self-supervised approaches, on HGs to alleviate the label scarcity problem. We first generate multiple semantic views according to metapaths and network schemas. Then, by pushing node embeddings corresponding to different semantic views close to each other (positives) and pulling other embeddings apart (negatives), one can obtain informative representations without human annotations. However, this CL approach ignores the relative hardness of negative samples, which may lead to suboptimal performance. Considering the complex graph structure and the smoothing nature of GNNs, we propose a structure-aware hard negative mining scheme that measures hardness by structural characteristics for HGs. By synthesizing more negative nodes, we give larger weights to harder negatives with limited computational overhead to further boost the performance. Empirical studies on three real-world datasets show the effectiveness of our proposed method. The proposed method consistently outperforms existing state-of-the-art methods and notably, even surpasses several supervised counterparts.

show abstract

“…As with previous studies [1,14,24], CL benefits from hard negative samples, i.e. samples close to the anchor node such that cannot be distinguished easily.…”

Section: Structure-aware Hard Negative Miningmentioning

confidence: 75%

Section: Introductionmentioning

confidence: 99%

Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning

Zhu¹,

Xu²,

Cui³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…To excavate effective negative samples, these methods heavily depend on the large batch sizes [7] or memory bank [18]. Utilizing hard negative samples has long been recognized as an effective approach to boost model performance [17,29,54,45]. In the contrastive learning studies, [10,43] modify the contrastive learning loss to make it assign greater weights to the hard negative samples.…”

Section: Related Workmentioning

confidence: 99%

Robust Contrastive Learning Using Negative Samples with Diminished Semantics

Ge¹,

Mishra²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Unsupervised learning has recently made exceptional progress because of the development of more effective contrastive learning methods. However, CNNs are prone to depend on low-level features that humans deem non-semantic. This dependency has been conjectured to induce a lack of robustness to image perturbations or domain shift. In this paper, we show that by generating carefully designed negative samples, contrastive learning can learn more robust representations with less dependence on such features. Contrastive learning utilizes positive pairs that preserve semantic information while perturbing superficial features in the training images. Similarly, we propose to generate negative samples in a reversed way, where only the superfluous instead of the semantic features are preserved. We develop two methods, texture-based and patch-based augmentations, to generate negative samples. These samples achieve better generalization, especially under out-of-domain settings. We also analyze our method and the generated texture-based samples, showing that texture features are indispensable in classifying particular ImageNet classes and especially finer classes. We also show that model bias favors texture and shape features differently under different test settings. Our code, trained models, and ImageNet-Texture dataset can be found at https://github.com/ SongweiGe/Contrastive-Learning-with-Non-Semantic-Negatives.

show abstract

“…Such losses distinguish between similar/positive samples and dissimilar/negative samples in a batch during training. However, even these most simple losses are often considered difficult to optimize, and practitioners often rely on various training methods which can be useful in practice but interact in an unclear way with the optimization process: for example negative mining [42,52,55,56], memory banks [24,32,45,51,53,54], and positiveonly formulations [12,21,46].…”

Section: Introductionmentioning

confidence: 99%

Simple and Effective Balance of Contrastive Losses

Sors¹,

Rezende²,

Ibrahimi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Contrastive losses have long been a key ingredient of deep metric learning and are now becoming more popular due to the success of self-supervised learning. Recent research has shown the benefit of decomposing such losses into two sub-losses which act in a complementary way when learning the representation network: a positive term and an entropy term. Although the overall loss is thus defined as a combination of two terms, the balance of these two terms is often hidden behind implementation details and is largely ignored and sub-optimal in practice. In this work, we approach the balance of contrastive losses as a hyperparameter optimization problem, and propose a coordinate descent-based search method that efficiently find the hyperparameters that optimize evaluation performance. In the process, we extend existing balance analyses to the contrastive margin loss, include batch size in the balance, and explain how to aggregate loss elements from the batch to maintain near-optimal performance over a larger range of batch sizes. Extensive experiments with benchmarks from deep metric learning and self-supervised learning show that optimal hyper-parameters are found faster with our method than with other common search methods.* Work partially done while Sarah was interning at NAVER LABS Europe.

show abstract

Hard Negative Examples are Hard, but Useful

Cited by 66 publications

References 19 publications

Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning

Structure-Aware Hard Negative Mining for Heterogeneous Graph Contrastive Learning

Robust Contrastive Learning Using Negative Samples with Diminished Semantics

Simple and Effective Balance of Contrastive Losses

Contact Info

Product

Resources

About