CLSEP: Contrastive learning of sentence embedding with prompt

Wang, Qian; Zhang, Weiqi; Lei, Tianyi; Cao, Yu; Peng, Dezhong; Wang, Xu

doi:10.1016/j.knosys.2023.110381

Cited by 16 publications

(4 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…DiffCSE [27] combines contrastive loss for data augmentation of insensitive examples and replacement detection loss for sensitive examples, resulting in improved sentence embedding. CLSEP [28] introduces a novel data augmentation strategy called partial word embedding augmentation (PWVA) for text data, which enhances data in the word embedding space while preserving more semantic information, leading to better sentence embedding.…”

Section: Related Work a Learning For Sentence Embeddingmentioning

confidence: 99%

Core-View Contrastive Learning Network for Building Lightweight Cross-Domain Consultation System

Zheng,

Xu,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

Cross-domain Consultation Systems have become essential in numerous critical applications, for instance, an online citizen complaint system. However, addressing complaints with distinct orality characteristics often necessitates retrieving and integrating knowledge from diverse professional domains. This scenario represents a typical cross-domain problem. Nevertheless, the prevailing approach of utilizing generative large language models to tackle this problem presents challenges including model scale and drawbacks like hallucination and limited interpretability. To address these challenges, we proposed a novel approach called the Core-View Contrastive Learning (CVCL) network. Leveraging contrastive learning techniques with an integrated core-adaptive augmentation module, the CVCL network achieves accuracy in cross-domain information matching. Our objective is to construct a lightweight, precise, and interpretable cross-domain consultation system, overcoming the limitations encountered with large language models in addressing such challenges. Empirical validation of our proposed method using real-world datasets demonstrates its effectiveness. Our experiments show that the proposed method achieves comparable performance to large language models in terms of accuracy in text-matching tasks and surpasses the best baseline model by over 24 percentage points in F1-score for classification tasks. Additionally, our lightweight model achieved a performance level of 96% compared to the full model, while utilizing only 6% of the parameters.

show abstract

Section: Related Work a Learning For Sentence Embeddingmentioning

confidence: 99%

Core-View Contrastive Learning Network for Building Lightweight Cross-Domain Consultation System

Zheng,

Xu,

Chen

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Contrastive learning obtains promising results in representation learning [14,17,22]. Generally, a Siamese network is used to construct the contrastive framework and conduct contrastive training [14].…”

Section: Representation Learning Based On Contrastive Learningmentioning

confidence: 99%

Grouped Contrastive Learning of Self-Supervised Sentence Representation

Wang,

Zhang,

Lei

et al. 2023

Applied Sciences

Self Cite

View full text Add to dashboard Cite

This paper proposes a method called Grouped Contrastive Learning of self-supervised Sentence Representation (GCLSR), which can learn an effective and meaningful representation of sentences. Previous works maximize the similarity between two vectors to be the objective of contrastive learning, suffering from the high-dimensionality of the vectors. In addition, most previous works have adopted discrete data augmentation to obtain positive samples and have directly employed a contrastive framework from computer vision to perform contrastive training, which could hamper contrastive training because text data are discrete and sparse compared with image data. To solve these issues, we design a novel framework of contrastive learning, i.e., GCLSR, which divides the high-dimensional feature vector into several groups and respectively computes the groups’ contrastive losses to make use of more local information, eventually obtaining a more fine-grained sentence representation. In addition, in GCLSR, we design a new self-attention mechanism and both a continuous and a partial-word vector augmentation (PWVA). For the discrete and sparse text data, the use of self-attention could help the model focus on the informative words by measuring the importance of every word in a sentence. By using the PWVA, GCLSR can obtain high-quality positive samples used for contrastive learning. Experimental results demonstrate that our proposed GCLSR achieves an encouraging result on the challenging datasets of the semantic textual similarity (STS) task and transfer task.

show abstract

“…The assessment of PLM-based sentence representation relies on two crucial characteristics: generalization and robustness. While considerable research efforts have been dedicated to developing universal sentence embeddings using PLMs (Reimers and Gurevych, 2019;Zhang et al, 2020;Ni et al, 2022;Neelakantan et al, 2022;Wang et al, 2023;Bölücü et al, 2023), it is worth noting that despite their promising performance across various downstream classification tasks (Sun et al, 2019;, demonstrating proficiency in generalization, these representations exhibit limitations in terms of robustness in adversarial settings and are vulnerable to diverse adversarial attacks (Nie et al, 2020;. Existing research (Garg and Ramakrishnan, 2020;Hauser et al, 2023) highlights the poor robustness of these representations, such as BERTbased representations, which can be deceived by replacing a few words in the input sentence.…”

Section: Introductionmentioning

confidence: 99%

RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training

Asl,

Blanco,

Takabi

2023

Findings of the Association for Computational Linguistics: EMNLP 2023

View full text Add to dashboard Cite

Pre-trained language models (PLMs) have demonstrated exceptional performance across a wide range of natural language processing tasks. The utilization of PLM-based sentence embeddings enables the generation of contextual representations that capture rich semantic information. However, despite their success with unseen samples, current PLM-based representations suffer from poor robustness in adversarial settings. In this paper, we propose RobustEmbed, a self-supervised sentence embedding framework that enhances both generalization and robustness in various text representation tasks and against a diverse set of adversarial attacks. By generating high-risk adversarial perturbations to promote higher invariance in the embedding space and leveraging the perturbation within a novel contrastive objective approach, RobustEmbed effectively learns high-quality sentence embeddings. Our extensive experiments validate the superiority of RobustEmbed over the state-of-the-art self-supervised representations in adversarial settings, while also showcasing relative improvements in seven semantic textual similarity (STS) tasks and six transfer tasks. Specifically, our framework achieves a significant reduction in attack success rate from 75.51% to 39.62% for the BERTAttack attack technique, along with enhancements of 1.20% and 0.40% in STS tasks and transfer tasks, respectively.

show abstract

CLSEP: Contrastive learning of sentence embedding with prompt

Cited by 16 publications

References 13 publications

Core-View Contrastive Learning Network for Building Lightweight Cross-Domain Consultation System

Core-View Contrastive Learning Network for Building Lightweight Cross-Domain Consultation System

Grouped Contrastive Learning of Self-Supervised Sentence Representation

RobustEmbed: Robust Sentence Embeddings Using Self-Supervised Contrastive Pre-Training

Contact Info

Product

Resources

About