DiffCSE: Difference-based Contrastive Learning for Sentence Embeddings

Chuang, Yung-Sung; Dangovski, Rumen; Luo, Hongyin; Zhang, Yang; Chang, Shiyu; Soljačić, Marin; Li, Shang-Wen; Yih, Wen-tau; Kim, Yoon; Glass, James

doi:10.18653/v1/2022.naacl-main.311

Cited by 50 publications

(47 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, we failed to reproduce the results of [16]. We build the supervised version of DiffCSE using the same objectives by ourselves because Chuang et al [4] does not consider supervise settings in their work. Although D2CSE outperforms DiffCSE, both methods integrating equivariant contrastive learning tend to achieve lower performance than original contrastive learning.…”

Section: Supervised Methods Resultsmentioning

confidence: 96%

“…Gao et al [11] propose SimCSE that uses dropout to generate randomly perturbed sentence vectors for the same textual input. Numerous approaches [4,16,29,32,35,36] have followed SimCSE successfully.…”

Section: Introductionmentioning

confidence: 99%

“…Several approaches [4,8] point out the problem of invariant contrastive learning like SimCSE that the representation is only tailored to be insensitive to any perturbations of embeddings. Especially, DiffCSE directly inspired by equivariant contrastive learning [8] rather encourages representation to be sensitive to semantically meaningful transformation by employing the discriminator of ELECTRA [5] with a novel conditional replaced token detection (CRTD) task.…”

Section: Introductionmentioning

confidence: 99%

“…For both alignment and uniformity, lower number means better. ( † from[4]; ‡ measured by us; STS averages over all 7 test sets. )…”

mentioning

confidence: 99%

See 3 more Smart Citations

Prompts for Creativity

Lee¹,

O’Mahony

Lebeck

2023

Creativity and Innovation

View full text Add to dashboard Cite

This paper describes Difference-aware Deep continuous prompt for Contrastive Sentence Embeddings (D2CSE) that learns sentence embeddings. Compared to state-of-the-art approaches, D2CSE computes sentence vectors that are exceptional to distinguish a subtle difference in similar sentences by employing a simple neural architecture for continuous prompts. Unlike existing architectures that require multiple pretrained language models (PLMs) to process a pair of the original and corrupted (subtly modified) sentences, D2CSE avoids cumbersome fine-tuning of multiple PLMs by only optimizing continuous prompts by performing multiple tasks-i.e., contrastive learning and conditional replaced token detection all done in a self-guided manner. D2CSE overloads a single PLM on continuous prompts and greatly saves memory consumption as a result. The number of training parameters in D2CSE is reduced to about 1% of existing approaches while substantially improving the quality of sentence embeddings. We evaluate D2CSE on seven Semantic Textual Similarity (STS) benchmarks, using three different metrics, namely, Spearman's rank correlation, recall@K for a retrieval task, and the anisotropy of an embedding space measured in alignment and uniformity. Our empirical results suggest that shallow (not too meticulously devised) continuous prompts can be honed effectively for multiple NLP tasks and lead to improvements upon existing state-of-the-art approaches.

show abstract

Section: Supervised Methods Resultsmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

“…For both alignment and uniformity, lower number means better. ( † from[4]; ‡ measured by us; STS averages over all 7 test sets. )…”

mentioning

confidence: 99%

See 2 more Smart Citations

Prompts for Creativity

Lee¹,

O’Mahony

Lebeck

2023

Creativity and Innovation

View full text Add to dashboard Cite

show abstract

“…In contrast, our method computes the MI on attention. Other related works include (Chuang et al, 2022;Liu et al, 2022).…”

Section: Introductionmentioning

confidence: 99%

miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings

Klein¹,

Nabi²

2022

Preprint

View full text Add to dashboard Cite

This paper presents miCSE, a mutual information-basedContrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the syntactic consistency across augmented views for every single sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to stateof-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. The proposed approach is conceptually simple, easy to implement and optimize, yet empirically powerful. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding. 1

show abstract