ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Abstract

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pretrained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new stateof-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

Virtual Augmentation Supported Contrastive Learning of Sentence Representations

“…Nevertheless, we focus on unsupervised contrastive learning and form the positive pairs via data augmentation, since such methods are more cost-effective and applicable across different domains and languages. Along this line, many approaches have been developed recently, where the augmentations are obtained via sampling from surrounding or nearby contexts (Logeswaran and Lee, 2018;Giorgi et al, 2020), word or feature-level perturbation Yan et al, 2021), back-translation (Fang and Xie, 2020), sentencelevel corruption using an auxiliary language model (Meng et al, 2021), intermediate representations of BERT (Kim et al, 2021), and dropout (Yan et al, 2021;Gao et al, 2021).…”

“…Various contrastive learning based approaches have been proposed for learning sentence representations, with the main difference lies in how the augmentations are generated (Fang and Xie, 2020;Giorgi et al, 2020;Meng et al, 2021;Yan et al, 2021;Kim et al, 2021;Gao et al, 2021). Somewhat surprisingly, a recent work (Gao et al, 2021) empirically shows that augmentations obtained by Dropout (Srivastava et al, 2014), i.e., feeding the same instance to the encoder twice, outperforms common text augmentation operations on the text directly, including cropping, word deletion, or synonym replacement.…”

Sentence-aware Contrastive Learning for Open-Domain Passage Retrieval

“…Contrastive learning (CL) recently is attracting researchers' attention in all area. After witnessing its superiority in Computer Vision tasks (Chen et al 2020;He et al 2020), researchers in NLP are also applying this techniques (Wu et al 2020;Karpukhin et al 2020;Yan et al 2021;Giorgi et al 2021;Gao, Yao, and Chen 2021). For the concern of ODPR, the research lines of CL can be divided into two types: (i) Improving the sampling strategies for positive samples and hard negative samples.…”

Coarse to Fine: Video Retrieval before Moment Localization

“…To promote the robustness, some data augmentations are used in training. Following [5], we adopt token shuffling, cutoff and dropout. Token shuffling strategy aims to randomly shuffle order of the tokens in the token embeddings.…”