Improving Sentence Representations via Component Focusing

Yin, Xiaoya; Zhang, Wu; Zhu, Wenhao; Liu, Shuang; Yao, Tengjun

doi:10.3390/app10030958

Cited by 11 publications

(7 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The input data length has a fixed value to match the matrix dimension of the CNN module, and a pad is added to enforce the same length for all data. CF-SBERT [24] uses the Siamese BERT and extracts the sentence vector using a GAP. A new sentence is generated using important component data obtained using Part-Of-Speech (POS) tagging from input data.…”

Section: Related Workmentioning

confidence: 99%

TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation

et al. 2022

View full text Add to dashboard Cite

A sentence embedding vector can be obtained by connecting a global average pooling (GAP) to a pre-trained language model. The problem of such a sentence embedding vector using a GAP is that it is generated with the same weight for all words appearing in the sentence. We propose a novel sentence embedding-method-based model Token Attention-SentenceBERT (TA-SBERT) to address this problem. The rationale of TA-SBERT is to enhance the performance of sentence embedding by introducing three strategies. First, we convert the base form while preprocessing the input sentence to reduce misunderstanding. Second, we propose a novel Token Attention (TA) technique that distinguishes important words to produce more informative sentence vectors. Third, we increase stability of fine-tuning to avoid catastrophic forgetting by adding a reconstruction loss to the word embedding vector. Extensive ablation studies demonstrate that our TA-SBERT outperforms the original SentenceBERT (SBERT) in the sentence vector evaluation using semantic textual similarity (STS) tasks and the SentEval toolkit.

show abstract

Section: Related Workmentioning

confidence: 99%

TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation

et al. 2022

View full text Add to dashboard Cite

show abstract

“…We measured the Spearman rank correlation between the cosine similarity of the sentence embedding and the gold labels. Spearman’s rank correlation has been used to measure semantic textual similarity in other studies [ 19 , 55 , 116 ]. We show the performance in Table 6 , highlighting the templates used in our article in bold.…”

Section: Resultsmentioning

confidence: 99%

The HoPE Model Architecture: a Novel Approach to Pregnancy Information Retrieval Based on Conversational Agents

Montenegro

Costa

2022

J Healthc Inform Res

View full text Add to dashboard Cite

Conversational agents are used to communicating with humans in a friendly manner. To achieve the highest level of performance, agents need to respond assertively and fastly. Transformer architectures are shown to produce excellent performances on recent tasks; however, for tasks involving conversational agents, they may have a lower speed performance. The main goal of this study is to evaluate and propose a HoPE (Healthcare Obstetric in PrEgnancy) model that is tailored to pregnancy data. We carried out a dataset extraction and construction process based on collections of health documents related to breastfeeding, childcare, pregnant care, nutrition, risks, vaccines, exams, and physical exercises. We evaluated two pre-trained models in the Portuguese language for the conversational agent architecture proposal and chose the one with the best performance to compose the HoPE architecture. The BERTimbau model, which has been trained on data augmentation strategies, proves to be able to retrieve information quickly and most accurately than others. For the fine-tuning process, we achieved a Spearman correlation of 95.55 on BERTimbau augmented with a few pairs (1.500 pairs). The HoPE model architecture achieved an F1-Score of 0.89, outperforming other combinations tested in this study. We will evaluate this approach for clinical studies in future studies.

show abstract

“…Choi et al [25] proposed a sentence representation method for CNN-SBert, which aims to extract sentence vectors by adding a CNN module. Yin et al [26] proposed a method of CF-SBert, which uses Siamese Bert, extracts sentence vectors with the help of GAP, and uses part-of-speech tagging important component data obtained from input data to generate new sentences.…”

Section: Relate Workmentioning

confidence: 99%

MAI-CBert: Multidirectional Attention Interaction Construction-Bert Sentiment Sentence Representation

2022

View full text Add to dashboard Cite

Considering that when the traditional network structure is connected with the pre-trained language model for sentence embedding representation, it relies on simple word weights to generate sentence vectors, which easily ignores the global and contextual semantic details of sentences, reducing the accuracy of sentence representation. To address these issues, we develop a Multidirectional Attention Interaction Construction-Bert Sentence Representation Framework (MAI-CBert). First, the model uses deep attention to transform the input emotional sentences to reduce misunderstanding caused by the unbalanced distribution of word weights in the sentences; at the same time, it applies horizontal and vertical attention to emotional constructions, which are generated from two different directions. Construction vectors, aim to focus on words with salient features in sentences and establish close dependencies. Second, the dynamic interaction strategy is adopted to realize the interaction between attention in different directions, so that the information flow forms a complementary relationship, resulting in more effective sentence vectors. Notably, the loss function is refactored to improve the representation robustness of the model to avoid catastrophic forgetting. Experimental results on the SemEval-14 and ACL-14 baseline datasets demonstrate that the MAI-CBert sentence representation framework is robust and competitive.

show abstract

Improving Sentence Representations via Component Focusing

Cited by 11 publications

References 29 publications

TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation

TA-SBERT: Token Attention Sentence-BERT for Improving Sentence Representation

The HoPE Model Architecture: a Novel Approach to Pregnancy Information Retrieval Based on Conversational Agents

MAI-CBert: Multidirectional Attention Interaction Construction-Bert Sentiment Sentence Representation

Contact Info

Product

Resources

About