TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation

Abstract

A sentence embedding vector can be obtained by connecting a global average pooling (GAP) to a pre-trained language model. The problem of such a sentence embedding vector using a GAP is that it is generated with the same weight for all words appearing in the sentence. We propose a novel sentence embedding-method-based model Token Attention-SentenceBERT (TA-SBERT) to address this problem. The rationale of TA-SBERT is to enhance the performance of sentence embedding by introducing three strategies. First, we convert the base form while preprocessing the input sentence to reduce misunderstanding. Second, we propose a novel Token Attention (TA) technique that distinguishes important words to produce more informative sentence vectors. Third, we increase stability of fine-tuning to avoid catastrophic forgetting by adding a reconstruction loss to the word embedding vector. Extensive ablation studies demonstrate that our TA-SBERT outperforms the original SentenceBERT (SBERT) in the sentence vector evaluation using semantic textual similarity (STS) tasks and the SentEval toolkit.

Enhancing Bug Localization Using Phase-Based Approach

“…Subsequently, the recommended package source files and past related bugs to the specific package will be input into this task. Moreover, a text similarity technique which is sentence BERT [22] will take the new bug report structured text to get the most similar source code file and if attached with old bugs. The output of the source code recommendation phase will be a list of source code files sorted in descending order according to their similarity to new bug report that needs to be fixed.…”

“…The text similarity is applied between the new bug report text and all files related to the software package recommended by the previous phase. According to [22] sentence BERT is applied to fine-tune BERT architecture for semantic similarity. As in Figure 5, the input from the source code files and related data from the software package will be entered into the pre-trained BERT.…”

CharAs-CBert: Character Assist Construction-Bert Sentence Representation Improving Sentiment Classification

“…The learning process could be completed by using an untagged corpus. To balance the role of different words, Seo et al [24] divided sentences into the form of Token, and combined the attention mechanism and sentence BERT, aiming to assign corresponding weights to different words through the attention mechanism to highlight the role of keywords. This improves the accuracy of subsequent tasks, but in the sentence decomposition and representation stage, it is easy to ignore the internal structure information of words, which weakens the representation effect of global semantics.…”

MAI-CBert: Multidirectional Attention Interaction Construction-Bert Sentiment Sentence Representation

“…To address these limitations, Shi [23] considered the feature capture ability of Transformer and used Transformer in sentence representation tasks to learn the generality of sentences. Seo et al [11] combined the attention mechanism with the language model Bert to improve the limitations of the pre-trained language model in the sentence representation task, that is, when the language model is used for sentence representation, it only generates sentence representations by obtaining word weights for Bert, which is easy Ignore the global and contextual context of the sentence. Kim et al [24] propose a contrastive learning method that utilizes self-guidance to improve the quality of Bert sentence representations by fine-tuning Bert in a self-supervised manner that improves sentence representations without relying on additional processing such as data augmentation.…”

“…Although these methods improve the accuracy of sentence representation when the sentence is complex, it is prone to semantic ambiguity and ignores the context of the word, that is when representing a sentence, the role of the same word in different sentences, and the semantics of the expression may be different. The traditional sentence embedding methods used in SBert [10] and TA-SBert [11], given that all words appearing in a sentence have the same weight, do not take these limitations into account, and at the same time, ignore the correlation between these feature information. Therefore, to address this limitation, we combine sentence constructions and propose a Multi-directional Attention Interaction Construction-Bert Sentence Representation Framework (MAI-CBert), which pays attention to salient words through multiple directions and assigns more Effective weights to produce sentence vectors with rich details.…”