A large language model for electronic health records

Abstract

There is an increasing interest in developing artificial intelligence (AI) systems to process and interpret electronic health records (EHRs). Natural language processing (NLP) powered by pretrained language models is the key technology for medical AI systems utilizing clinical narratives. However, there are few clinical language models, the largest of which trained in the clinical domain is comparatively small at 110 million parameters (compared with billions of parameters in the general domain). It is not clear how large clinical language models with billions of parameters can help medical AI systems utilize unstructured EHRs. In this study, we develop from scratch a large clinical language model—GatorTron—using >90 billion words of text (including >82 billion words of de-identified clinical text) and systematically evaluate it on five clinical NLP tasks including clinical concept extraction, medical relation extraction, semantic textual similarity, natural language inference (NLI), and medical question answering (MQA). We examine how (1) scaling up the number of parameters and (2) scaling up the size of the training data could benefit these NLP tasks. GatorTron models scale up the clinical language model from 110 million to 8.9 billion parameters and improve five clinical NLP tasks (e.g., 9.6% and 9.5% improvement in accuracy for NLI and MQA), which can be applied to medical AI systems to improve healthcare delivery. The GatorTron models are publicly available at: https://catalog.ngc.nvidia.com/orgs/nvidia/teams/clara/models/gatortron_og.

Assessing the Value of ChatGPT for Clinical Decision Support Optimization

“…For example, using a publicly available clinical language model, GatorTron, based on 90 billion words of de-identified clinical notes from the University of Florida (UF) Health, PubMed articles, and Wikipedia, might be a good start. [34] Another example is BioGPT, a GPT-like model trained on PubMed articles. [35] In addition, UpToDate is an important source of CDS content for alert development and should also be considered for adding to the language model.…”

An overview and a roadmap for artificial intelligence in hematology and oncology

“…NLP is a branch of AI that deals with the interpretation and manipulation of human language (Yim et al 2016;Sorin et al 2020;Kung, et al 2022;Yang et al 2022;Singhal, et al 2022). For example, NLP is used in chatbots and digital assistants such as Siri or Alexa, which are capable of understanding natural language commands and providing relevant information in response.…”

“…In medicine, language is often used as an unstructured way to store and transmit information. In this context, NLP can be used to extract information from clinical reports or electronic health records (Yang et al 2022;Thomas et al 2014). It should be noted that current systems do not simply search and extract text but integrate the context (e.g., it is essential whether a diagnosis is present or excluded or how the temporal dependencies between reported events are).…”

Analysis of large-language model versus human performance for genetics questions

“…Model training involves unsupervised learning, where the models learn to predict the next word in a sentence without explicit labels or annotations. 6 We aimed to explore how well ChatGPT would perform compared to human respondents in answering questions about genetics.…”

“…Model training involves unsupervised learning, where the models learn to predict the next word in a sentence without explicit labels or annotations. 6…”