BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Lee, Jinhyuk; Yoon, Wonjin; Kim, Sungdong; Kim, Donghyeon; Kim, Sunkyu; So, Chan Ho; Kang, Jaewoo

doi:10.1093/bioinformatics/btz682

Cited by 3,933 publications

(3,149 citation statements)

References 33 publications

Supporting

Mentioning

3,115

Contrasting

Unclassified

Order By: Relevance

“…While our experiments demonstrate this improvement only in three applications, we believe that it can generalize to other prediction models that rely on combinations of formalized and natural language knowledge. Our normalization method is currently limited by its reliance on lexical matching to identify mentions of ontology classes, while novel natural language methods often use machine learning models for this purpose as well (Lee et al, 2020). In future work, more experiments with different named entity recognition and normalization approaches are needed to improve our method.…”

Section: Discussionmentioning

confidence: 99%

Self-normalizing learning on biomedical ontologies using a deep Siamese neural network

Smaili

Gao

Hoehndorf

2020

Preprint

View full text Add to dashboard Cite

Motivation:Ontologies are widely used in biomedicine for the annotation and standardization of data. One of the main roles of ontologies is to provide structured background knowledge within a domain as well as a set of labels, synonyms, and definitions for the classes within a domain. The two types of information provided by ontologies have been extensively exploited in natural language processing and machine learning applications. However, they are commonly used separately, and thus it is unknown if joining the two sources of information can further benefit data analysis tasks. Results: We developed a novel method that applies named entity recognition and normalization methods on texts to connect the structured information in biomedical ontologies with the information contained in natural language. We apply this normalization both to literature and to the natural language information contained within ontologies themselves. The normalized ontologies and text are then used to generate embeddings, and relations between entities are predicted using a deep Siamese neural network model that takes these embeddings as input. We demonstrate that our novel embedding and prediction method using self-normalized biomedical ontologies significantly outperforms the state-of-the-art methods in embedding ontologies on two benchmark tasks: prediction of interactions between proteins and prediction of gene-disease associations. Our method also allows us to apply ontology-based annotations and axioms to the prediction of toxicological effects of chemicals where our method shows superior performance. Our method is generic and can be applied in scenarios where ontologies consisting of both structured information and natural language labels or synonyms are used.

show abstract

Section: Discussionmentioning

confidence: 99%

Self-normalizing learning on biomedical ontologies using a deep Siamese neural network

Smaili

Gao

Hoehndorf

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…We optimize BERT by first fine-tuning it with a domain-specific text corpus then fine-tune the resulting model for the different subtasks. This has been shown to be a strong baseline for various NLP tasks [2,20,55] including ABSA. Fine-tuning LMs for specific subtasks.…”

Section: Fine-tuning Pre-trained Language Modelsmentioning

confidence: 99%

Snippext: Semi-supervised Opinion Mining with Augmented Data

Miao

Wang

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

Online services are interested in solutions to opinion mining, which is the problem of extracting aspects, opinions, and sentiments from text. One method to mine opinions is to leverage the recent success of pre-trained language models which can be fine-tuned to obtain highquality extractions from reviews. However, fine-tuning language models still requires a non-trivial amount of training data.In this paper, we study the problem of how to significantly reduce the amount of labeled training data required in fine-tuning language models for opinion mining. We describe Snippext, an opinion mining system developed over a language model that is fine-tuned through semi-supervised learning with augmented data. A novelty of Snippext is its clever use of a two-prong approach to achieve state-of-the-art (SOTA) performance with little labeled training data through: (1) data augmentation to automatically generate more labeled training data from existing ones, and (2) a semi-supervised learning technique to leverage the massive amount of unlabeled data in addition to the (limited amount of) labeled data. We show with extensive experiments that Snippext performs comparably and can even exceed previous SOTA results on several opinion mining tasks with only half the training data required. Furthermore, it achieves new SOTA results when all training data are leveraged. By comparison to a baseline pipeline, we found that Snippext extracts significantly more fine-grained opinions which enable new opportunities of downstream applications. ACM Reference Format:

show abstract

“…However, to address several limitations, we choose to train our own clinical BERT model in this work. First, existing models are initialized from BioBERT [39] or BERT BASE [16], though SciBERT [6] outperforms BioBERT on a number of downstream tasks. Secondly, existing models do not satisfactorily encode the personal health identifiers (PHI) within the notes (e.g., [**2126-9-19**]), either leaving them as is, or removing them altogether.…”

Section: Pretrained Clinical Embeddingsmentioning

confidence: 99%

Hurtful words

Zhang

Abdalla

et al. 2020

Proceedings of the ACM Conference on Health, Inference, and Learning

View full text Add to dashboard Cite

In this work, we examine the extent to which embeddings may encode marginalized populations differently, and how this may lead to a perpetuation of biases and worsened performance on clinical tasks. We pretrain deep embedding models (BERT) on medical notes from the MIMIC-III hospital dataset, and quantify potential disparities using two approaches. First, we identify dangerous latent relationships that are captured by the contextual word embeddings using a fill-in-the-blank method with text from real clinical notes and a log probability bias score quantification. Second, we evaluate performance gaps across different definitions of fairness on over 50 downstream clinical prediction tasks that include detection of acute and chronic conditions. We find that classifiers trained from BERT representations exhibit statistically significant differences in performance, often favoring the majority group with regards to gender, language, ethnicity, and insurance status. Finally, we explore shortcomings of using adversarial debiasing to obfuscate subgroup information in contextual word embeddings, and recommend best practices for such deep embedding models in clinical settings.

show abstract

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Cited by 3,933 publications

References 33 publications

Self-normalizing learning on biomedical ontologies using a deep Siamese neural network

Self-normalizing learning on biomedical ontologies using a deep Siamese neural network

Snippext: Semi-supervised Opinion Mining with Augmented Data

Hurtful words

Contact Info

Product

Resources

About