Fine-Tuning Bidirectional Encoder Representations From Transformers (BERT)–Based Models on Large-Scale Electronic Health Record Notes: An Empirical Study

Abstract

Background The bidirectional encoder representations from transformers (BERT) model has achieved great success in many natural language processing (NLP) tasks, such as named entity recognition and question answering. However, little prior work has explored this model to be used for an important task in the biomedical and clinical domains, namely entity normalization. Objective We aim to investigate the effectiveness of BERT-based models for biomedical or clinical entity normalization. In addition, our second objective is to investigate whether the domains of training data influence the performances of BERT-based models as well as the degree of influence. Methods Our data was comprised of 1.5 million unlabeled electronic health record (EHR) notes. We first fine-tuned BioBERT on this large collection of unlabeled EHR notes. This generated our BERT-based model trained using 1.5 million electronic health record notes (EhrBERT). We then further fine-tuned EhrBERT, BioBERT, and BERT on three annotated corpora for biomedical and clinical entity normalization: the Medication, Indication, and Adverse Drug Events (MADE) 1.0 corpus, the National Center for Biotechnology Information (NCBI) disease corpus, and the Chemical-Disease Relations (CDR) corpus. We compared our models with two state-of-the-art normalization systems, namely MetaMap and disease name normalization (DNorm). Results EhrBERT achieved 40.95% F1 in the MADE 1.0 corpus for mapping named entities to the Medical Dictionary for Regulatory Activities and the Systematized Nomenclature of Medicine—Clinical Terms (SNOMED-CT), which have about 380,000 terms. In this corpus, EhrBERT outperformed MetaMap by 2.36% in F1. For the NCBI disease corpus and CDR corpus, EhrBERT also outperformed DNorm by improving the F1 scores from 88.37% and 89.92% to 90.35% and 93.82%, respectively. Compared with BioBERT and BERT, EhrBERT outperformed them on the MADE 1.0 corpus and the CDR corpus. Conclusions Our work shows that BERT-based models have achieved state-of-the-art performance for biomedical and clinical entity normalization. BERT-based models can be readily fine-tuned to normalize any kind of named entities.

Validation of deep learning natural language processing algorithm for keyword extraction from pathology reports in electronic health records

“…Several NLP studies on electronic health records have attempted to create models that accomplish multiple tasks based on an advanced deep learning approach. Li et al developed a BERT-based model for electronic health records (EhrBERT) to normalize biomedical entities on the standard vocabulary 11 . In comparison, our model could extract pathological keywords stated in the original text and intuitively summarize narrative reports while preserving the intention of the vocabulary.…”

“…The bidirectional encoder representations from transformers (BERT) model is one of the latest deep learning language models based on attention mechanisms 10 . It has been applied in many kinds of biomedical natural language processing (NLP) research, including clinical entity normalization, text mining (i.e., BioBERT), breast cancer concept extraction, and discharge summary diagnosis extraction [11][12][13][14] .…”

Innovations in research and clinical care using patient‐generated health data

“…BERT set new performance records across nearly all natural language processing benchmarks and, for the first time, achieved human‐level performance on several of these. BERT and related technologies are being applied to unstructured biomedical 215,216 and clinical texts 217,218 . Translation of these new technologies into the clinical domain is still in the early stages, but progress is accelerating.…”

Modern Clinical Text Mining: A Guide and Review

“…They produced two models, one for generic clinical text and another for discharge summaries, which they released publicly (35). They and others have demonstrated that BERT models fine-tuned on clinical corpora improve the state of the art on clinical concept recognition, de-identification, inference, and concept normalization tasks (88,89), though in at least one case, UMLS features still contributed valuable additional information (90).…”