Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing

Abstract

Motivation: A perennial challenge for biomedical researchers and clinical practitioners is to stay abreast with the rapid growth of publications and medical notes. Natural language processing (NLP) has emerged as a promising direction for taming information overload. In particular, large neural language models facilitate transfer learning by pretraining on unlabeled text, as exemplified by the successes of BERT models in various NLP applications. However, fine-tuning such models for an end task remains challenging, especially with small labeled datasets, which are common in biomedical NLP. Results:We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that finetuning performance may be sensitive to pretraining settings, especially in low-resource domains. Large models have potential to attain better performance, but increasing model size also exacerbates finetuning instability. We thus conduct a comprehensive exploration of techniques for addressing fine-tuning instability. We show that these techniques can substantially improve fine-tuning performance for lowresource biomedical NLP applications. Specifically, freezing lower layers is helpful for standard BERT-BASE models, while layerwise decay is more effective for BERT-LARGE and ELECTRA models. For low-resource text similarity tasks such as BIOSSES, reinitializing the top layer is the optimal strategy. Overall, domainspecific vocabulary and pretraining facilitate more robust models for fine-tuning. Based on these findings, we establish new state of the art on a wide range of biomedical NLP applications. Availability and implementation: To facilitate progress in biomedical NLP, we release our state-of-the-art pretrained and fine-tuned models: https://aka.ms/BLURB.

Medical BigBERTa: An Optimized Transformer for Long Medical Document

“…We, therefore, introduce a tokenizer training stage that generates a new vocabulary file, by combining the biomedical and clinical corpora used to train and fine-tune the final model. The main difference between our approach and most BERT variants is that other LMs are based technically on a continuous training approach, whereby the source model is fine-tuned on a specific domain corpus [13], whereas our model adopts a similar approach to that of Tinn et al [8] by including a dedicated tokenizer in the process.…”

“…• training epoch i = [5 : 100] • per device training batch size β = [4,8,12,16,32,64] By running 200 trials for each search, the resultant customized hyperparameters helped boost our performance to an average 3.11% F1 score for all downstream tasks.…”

“…EHRs contain vast amounts of structured and unstructured data, which can be used not only to fine-tune predictive algorithms and drug compatibility tests but also to help understand the course of diseases and patient histories. Researchers have proposed a variety of NLP-based models to better analyze biomedical documents [4], [5], [7], [8]. With medical texts representing 80% of the EHR data [1], it is imperative to develop more robust and efficient language models (LMs), which can then be used to better understand and extract the relevant information contained in those texts.…”

Medical BigBERTa: An Optimized Transformer for Long Medical Document

“…EHRs contain a tremendous amount of structured and unstructured data, which can be used to fine-tune predictive algorithms and drug compatibilities and help to understand the course of diseases and patients. Various researchers have proposed adapted NLP models to address better biomedical documents [4], [5], [7], [8]. With medical texts representing 80% of the EHR data [1], it's imperative to develop more robust and efficient language models which can be used to understand and extract relevant information contained in those texts.…”

“…We thus conducted a tokenizer training that generated a new vocabulary file, combining biomedical and clinical corpora used to train and fine-tune the final model. The main difference between our approach and most BERT variants is that those LM are technically based on a continuous training approach where the source model is fine-tuned on a specific domain corpus [13], while ours adopts a similar approach as [8] by including in the process a dedicated tokenizer.…”