miCSE: Mutual Information Contrastive Learning for Low-shot Sentence Embeddings

Abstract

This paper presents miCSE, a mutual information-basedContrastive learning framework that significantly advances the state-of-the-art in few-shot sentence embedding. The proposed approach imposes alignment between the attention pattern of different views during contrastive learning. Learning sentence embeddings with miCSE entails enforcing the syntactic consistency across augmented views for every single sentence, making contrastive self-supervised learning more sample efficient. As a result, the proposed approach shows strong performance in the few-shot learning domain. While it achieves superior results compared to stateof-the-art methods on multiple benchmarks in few-shot learning, it is comparable in the full-shot scenario. The proposed approach is conceptually simple, easy to implement and optimize, yet empirically powerful. This study opens up avenues for efficient self-supervised learning methods that are more robust than current contrastive methods for sentence embedding. 1

SelfCCL: Curriculum Contrastive Learning by Transferring Self-Taught Knowledge for Fine-Tuning BERT

“…Refs. [4,[6][7][8][9][10][11][12][13][14][15] use contrastive learning. BERT-Flow and BERT-whitening [5,55] are post-processing models that apply flow-network and whitening to enhance BERT, respectively.…”

“…The best average result is in bold in the last column. †: [33], ‡: [9], ♠: [6], ♣: [10], : [56], : [4], ♥: [7], ♦: [8], : [13] , : [14], * : [15], SBERT-base-nli-v2: reproduced by ourselves, and the rest of the results are taken from Ref. [12].…”

“…To overcome this problem, a thread of research has thus been working on applying contrastive learning techniques under the pre-train-then-fine-tune paradigm of BERT [4,[6][7][8][9][10][11][12][13][14]. Contrastive learning pulls similar samples closer together, but pushes out dissimilar samples [15].…”