Text semantic matching with an enhanced sample building method based on contrastive learning

Wu, Lishan; Hu, Jie; Teng, Fei; Li, Tianrui; Du, Shengdong

doi:10.1007/s13042-023-01823-8

Cited by 4 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, researchers have achieved great advances in sentence representation based on contrastive learning with pre-trained language models [6][7][8][9][10]. On the one hand, the largescale pre-trained language models (PLMs), typified by BERT [11], are trained with unlabeled data, improving the state-of-the-art results in most downstream tasks.…”

Section: Introductionmentioning

confidence: 99%

Grouped Contrastive Learning of Self-Supervised Sentence Representation

Wang,

Zhang,

Lei

et al. 2023

Applied Sciences

View full text Add to dashboard Cite

This paper proposes a method called Grouped Contrastive Learning of self-supervised Sentence Representation (GCLSR), which can learn an effective and meaningful representation of sentences. Previous works maximize the similarity between two vectors to be the objective of contrastive learning, suffering from the high-dimensionality of the vectors. In addition, most previous works have adopted discrete data augmentation to obtain positive samples and have directly employed a contrastive framework from computer vision to perform contrastive training, which could hamper contrastive training because text data are discrete and sparse compared with image data. To solve these issues, we design a novel framework of contrastive learning, i.e., GCLSR, which divides the high-dimensional feature vector into several groups and respectively computes the groups’ contrastive losses to make use of more local information, eventually obtaining a more fine-grained sentence representation. In addition, in GCLSR, we design a new self-attention mechanism and both a continuous and a partial-word vector augmentation (PWVA). For the discrete and sparse text data, the use of self-attention could help the model focus on the informative words by measuring the importance of every word in a sentence. By using the PWVA, GCLSR can obtain high-quality positive samples used for contrastive learning. Experimental results demonstrate that our proposed GCLSR achieves an encouraging result on the challenging datasets of the semantic textual similarity (STS) task and transfer task.

show abstract