Named entity recognition in electronic health records using transfer learning bootstrapped Neural Networks

Abstract

Neural networks (NNs) have become the state of the art in many machine learning applications, such as image, sound [1] and natural language processing [2,3]. However, the success of NNs remains dependent on the availability of large labelled datasets, such as in the case of electronic health records (EHRs). With scarce data, NNs are unlikely to be able to extract this hidden information with practical accuracy. In this study, we develop an approach that solves these problems for named entity recognition, obtaining 94.6 F1 score in I2B2 2009 Medical Extraction Challenge [6], 4.3 above the architecture that won the competition. To achieve this, we bootstrap our NN models through transfer learning by pretraining word embeddings on a secondary task performed on a large pool of unannotated EHRs and using the output embeddings as a foundation of a range of NN architectures. Beyond the official I2B2 challenge, we further achieve 82.4 F1 on extracting relationships between medical terms using attention-based seq2seq models bootstrapped in the same manner.

Medical Information Extraction in the Age of Deep Learning

“…Wei et al [62] Gligic et al [63] Zeng et al [64] Unanue et al [65] Li et al [66] Wunnava et al [67] Dandala et al [70] Tao et al [71] Chapman et al [72] Unanue et al [65] NE type Word2vec embeddings self-trained on MIMIC-III [58] Word2vec embeddings self-trained on i2b2 2009 [57] word & character embeddings pre-trained on Wikipedia GloVe [53] pre-trained on CommonCrawl + GloVe self-trained on MIMIC-III [58] pre-trained [16] pre-trained on Wikipedia, EHR notes, and PubMed [68,69] pretrained Notes and consists of 1,092 de-identified EHR notes from 21 cancer patients. Each note was annotated with medication information (drug name, dosage, route, frequency, duration), ADEs, indication (symptom as reason for drug administration), other signs and symptoms, severity (of disease/symptom), and relations among those entities, resulting in 79,000 mention annotations.…”

“…Wei et al [62] Tao et al [71] Gligic et al [63] Li et al [66] Dandala et al [70] Chapman et al [72] Wunnava et al [67] Gligic et al [63] Wei et al [62] Tao et al [71] Wunnava et al [67] Chapman et al [72] Li et al [66] Dandala et al [70] Wei et al [62] Gligic et al [63] Tao et al [71] Wunnava et al [67] Li et al [66] Chapman et al [72] Dandala et al [70] Wei et al [62] Li et al [66] Wunnava et al [67] Dandala et al [70] Chapman et al [72] Gligic et al [63] Tao et al [71] Wunnava et al [67] Yang et al [73] Dandala et al [70] Li et al [66] Wei et al [62] Chapman et al [72] Word2vec embeddings self-trained on MIMIC-III [58] (as feature for CRF: GloVe [53] embeddings self-trained on MIMIC-III [58])…”

“…There seems to be no generalizable winner for either choice of embeddings given the current state of evaluations, but self-training on medical raw data, such as MIMIC-III, challenge data sets, or, more advisable, using the now available BioSentVec [18] and BlueBERT [22] embeddings pre-trained on MIMIC-III, might be advantageous. 12 Interestingly, the transfer learning approach advocated by Gligic et al [63] performs well for some medication NER tasks, but fails to deliver competitive results for the medication relation task (cf. Table 5).…”

“… 12 Interestingly, the transfer learning approach advocated by Gligic et al 63 performs well for some medication NER tasks, but fails to deliver competitive results for the medication relation task (cf. Table 5 ).…”

Identifying Predictors of Suicide in Severe Mental Illness: A Feasibility Study of a Clinical Prediction Rule (Oxford Mental Illness and Suicide Tool or OxMIS)

“…This makes extraction of data difficult, relying either on resource-intensive manual review of clinical records, or, increasingly, automated natural language processing (NLP) algorithms (15,16). NLP processes have been applied to extract a variety of information from free-text clinical records including medication use (17,18), self-harm (19,20), and socio-demographic history (21,22); such approaches have addressed model development using limited numbers of annotated text examples (23). One approach to NLP for EHRs is to build information retrieval systems based on named entity recognition (NER), where a model is trained to recognize concepts which are related to variables of interest.…”