Conditional BERT Contextual Augmentation

Wu, Xing; Lv, Shangwen; Zang, Liangjun; Han, Jizhong; Hu, Songlin

doi:10.1007/978-3-030-22747-0_7

Cited by 245 publications

(177 citation statements)

References 31 publications

Supporting

Mentioning

143

Contrasting

Order By: Relevance

“…Google first proposed the BERT model, and it completely subverted the logic of training word vectors before training specific tasks in natural language processing [ 24 ]. Methods of fine-tuning the BERT model, such as extended text preprocessing and layer adjustment, have been proved to improve the results substantially [ 66 ]. Wu et al proposed a conditional BERT method, which can enhance the text classification ability of original BERT method by predicting the conditions of masked words [ 67 ].…”

Section: Problem Statementmentioning

confidence: 99%

Application of BERT to Enable Gene Classification Based on Clinical Evidence

Xiang

et al. 2020

BioMed Research International

View full text Add to dashboard Cite

The identification of profiled cancer-related genes plays an essential role in cancer diagnosis and treatment. Based on literature research, the classification of genetic mutations continues to be done manually nowadays. Manual classification of genetic mutations is pathologist-dependent, subjective, and time-consuming. To improve the accuracy of clinical interpretation, scientists have proposed computational-based approaches for automatic analysis of mutations with the advent of next-generation sequencing technologies. Nevertheless, some challenges, such as multiple classifications, the complexity of texts, redundant descriptions, and inconsistent interpretation, have limited the development of algorithms. To overcome these difficulties, we have adapted a deep learning method named Bidirectional Encoder Representations from Transformers (BERT) to classify genetic mutations based on text evidence from an annotated database. During the training, three challenging features such as the extreme length of texts, biased data presentation, and high repeatability were addressed. Finally, the BERT+abstract demonstrates satisfactory results with 0.80 logarithmic loss, 0.6837 recall, and 0.705 F -measure. It is feasible for BERT to classify the genomic mutation text within literature-based datasets. Consequently, BERT is a practical tool for facilitating and significantly speeding up cancer research towards tumor progression, diagnosis, and the design of more precise and effective treatments.

show abstract

Section: Problem Statementmentioning

confidence: 99%

Application of BERT to Enable Gene Classification Based on Clinical Evidence

Xiang

et al. 2020

BioMed Research International

View full text Add to dashboard Cite

show abstract

“…Augmentation is done by sampling from the returned probability distribution. Kobayashi [42] trained a Bi-Directional LSTM language model with this approach, and Wu et al [43] enhanced the approach by using BERT as an underlying model. All the mentioned methods were evaluated in this study, and the implementation was done using the NLPAug library [44] except for contextual augmentation for which the released code by the authors of [42] was used.…”

Section: Similar Word Substitution Augmentationmentioning

confidence: 99%

Transformer-based deep neural network language models for Alzheimer’s disease detection from targeted speech

Roshanzamir

Aghajan

Baghshah

2020

Preprint

View full text Add to dashboard Cite

Background: We developed transformer-based deep learning models based on natural language processing for early diagnosis of Alzheimer’s disease from the picture description test.Methods: The lack of large datasets poses the most important limitation for using complex models that do not require feature engineering. Transformer-based pre-trained deep language models have recently made a large leap in NLP research and application. These models are pre-trained on available large datasets to understand natural language texts appropriately, and are shown to subsequently perform well on classification tasks with small training sets. The overall classification model is a simple classifier on top of the pre-trained deep language model.Results: The models are evaluated on picture description test transcripts of the Pitt corpus, which contains data of 170 AD patients with 257 interviews and 99 healthy controls with 243 interviews. The large bidirectional encoder representations from transformers (BERTLarge) embedding with logistic regression classifier achieves classification accuracy of 88.08%, which improves thestate-of-the-art by 2.48%.Conclusions: Using pre-trained language models can improve AD prediction. This not only solves the problem of lack of sufficiently large datasets, but also reduces the need for expert-defined features.

show abstract

“…The method is replacing randomly chosen words in a sentence with a mixture of multiple related words based on a distributional representation. The contextual data augmentation studies [23,29], which are the most similar to ours, used a bidirectional LSTM-LM or MLM to change the words in a sentence using a fill-in-the-blank task. These contextual data augmentation methods achieved the state-of-the-art results on the text classification benchmark dataset.…”

Section: Data Augmentationmentioning

confidence: 99%

“…In order to alleviate the aforementioned problem, many studies take advantage of augmentation of the existing data [9,[19][20][21][22][23][24]. Data augmentation is used for small amounts of data, but also unbalanced data.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Self-Supervised Contextual Data Augmentation for Natural Language Processing

Park

Ahn

2019

Symmetry

View full text Add to dashboard Cite

In this paper, we propose a novel data augmentation method with respect to the target context of the data via self-supervised learning. Instead of looking for the exact synonyms of masked words, the proposed method finds words that can replace the original words considering the context. For self-supervised learning, we can employ the masked language model (MLM), which masks a specific word within a sentence and obtains the original word. The MLM learns the context of a sentence through asymmetrical inputs and outputs. However, without using the existing MLM, we propose a label-masked language model (LMLM) that can include label information for the mask tokens used in the MLM to effectively use the MLM in data with label information. The augmentation method performs self-supervised learning using LMLM and then implements data augmentation through the trained model. We demonstrate that our proposed method improves the classification accuracy of recurrent neural networks and convolutional neural network-based classifiers through several experiments for text classification benchmark datasets, including the Stanford Sentiment Treebank-5 (SST5), the Stanford Sentiment Treebank-2 (SST2), the subjectivity (Subj), the Multi-Perspective Question Answering (MPQA), the Movie Reviews (MR), and the Text Retrieval Conference (TREC) datasets. In addition, since the proposed method does not use external data, it can eliminate the time spent collecting external data, or pre-training using external data.

show abstract

Conditional BERT Contextual Augmentation

Cited by 245 publications

References 31 publications

Application of BERT to Enable Gene Classification Based on Clinical Evidence

Application of BERT to Enable Gene Classification Based on Clinical Evidence

Transformer-based deep neural network language models for Alzheimer’s disease detection from targeted speech

Self-Supervised Contextual Data Augmentation for Natural Language Processing

Contact Info

Product

Resources

About