RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models

Veen, Dave Van; Cara, Van Uden,; Maayane, Attias,; Pareek, Anuj; Bluethgen, Christian; Polacin, Malgorzata; Chiu, Wah; Jean-Benoit, Delbrouck,; Chaves, J.; Langlotz, Curtis P.; Chaudhari, Akshay; Pauly, John M.

doi:10.18653/v1/2023.bionlp-1.42

Cited by 12 publications

(7 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Crucially, machine-generated summaries must be non-inferior to that of seasoned clinicians-especially when used to support sensitive clinical decision-making. Recent work in clinical natural language processing (NLP) has demonstrated potential on medical text [66,75], adapting to the medical domain by either training a new model [59,70], fine-tuning an existing model [67,72], or supplying task-specific examples in the model prompt [46,72]. However, adapting LLMs to summarize a diverse set of clinical tasks has not been thoroughly explored, nor has non-inferiority to humans been achieved.…”

Section: Methodsmentioning

confidence: 99%

“…ICL is a lightweight adaptation method that requires no altering of model weights; instead, one includes a handful of in-context examples directly within the model prompt [39]. This simple approach provides the model with context, enhancing LLM performance for a particular task or domain [46,72]. We implemented this by choosing, for each sample in our test set, the m nearest neighbors training samples in the embedding space of the PubMedBERT model [16].…”

Section: Adaptation Methodsmentioning

confidence: 99%

“…Quantized low-rank adaptation (QLoRA). Low-rank adaptation (LoRA) [32] has emerged as an effective, lightweight approach for fine-tuning LLMs by altering a small subset of model weights-often < 0.1% [72].…”

Section: Adaptation Methodsmentioning

confidence: 99%

“…A derivative model, FLAN-T5 [14,43], improved performance via instruction prompt tuning. This T5 model family has proven effective for various clinical NLP tasks [40,72]. The FLAN-UL2 model [15,65] was introduced recently, which featured an increased context length (four-fold that of FLAN-T5) and a modified pre-training procedure called unified language learning (UL2).…”

Section: Large Language Modelsmentioning

confidence: 99%

See 3 more Smart Citations

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

Veen,

Uden,

Blankemeier

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

Sifting through vast textual data and summarizing key information from electronic health records (EHR) imposes a substantial burden on how clinicians allocate their time. Although large language models (LLMs) have shown immense promise in natural language processing (NLP) tasks, their efficacy on a diverse range of clinical summarization tasks has not yet been rigorously demonstrated. In this work, we apply domain adaptation methods to eight LLMs, spanning six datasets and four distinct clinical summarization tasks: radiology reports, patient questions, progress notes, and doctor-patient dialogue. Our thorough quantitative assessment reveals trade-offs between models and adaptation methods in addition to instances where recent advances in LLMs may not improve results. Further, in a clinical reader study with ten physicians, we show that summaries from our best-adapted LLMs are preferable to human summaries in terms of completeness and correctness. Our ensuing qualitative analysis highlights challenges faced by both LLMs and human experts. Lastly, we correlate traditional quantitative NLP metrics with reader study scores to enhance our understanding of how these metrics align with physician preferences. Our research marks the first evidence of LLMs outperforming human experts in clinical text summarization across multiple tasks. This implies that integrating LLMs into clinical workflows could alleviate documentation burden, empowering clinicians to focus more on personalized patient care and the inherently human aspects of medicine.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Adaptation Methodsmentioning

confidence: 99%

Section: Adaptation Methodsmentioning

confidence: 99%

Section: Large Language Modelsmentioning

confidence: 99%

See 2 more Smart Citations

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

Veen,

Uden,

Blankemeier

et al. 2023

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Large language models (LLMs) have the potential to revolutionize our medical system[54]. They can already streamline report generation and summarization[61, 59, 8, 37], answer biomedical questions with[60, 5, 59] and without[50, 49, 38] images, and could soon effectively interpret multimodal data for precision medicine in the clinic[6]. Importantly, as humans primarily interact with the world through language, LLMs are poised to be the point of access to the multimodal medical AI solutions of the future[36].…”

Section: Mainmentioning

confidence: 99%

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

Hager,

Jungmann,

Bhagat

et al. 2024

Preprint

View full text Add to dashboard Cite

Clinical decision making is one of the most impactful parts of a physician's responsibilities and stands to benefit greatly from AI solutions and large language models (LLMs) in particular. However, while LLMs have achieved excellent performance on medical licensing exams, these tests fail to assess many skills that are necessary for deployment in a realistic clinical decision making environment, including gathering information, adhering to established guidelines, and integrating into clinical workflows. To understand how useful LLMs are in real-world settings, we must evaluate them in the wild, i.e. on real-world data under realistic conditions. Here we have created a curated dataset based on the MIMIC-IV database spanning 2400 real patient cases and four common abdominal pathologies as well as a framework to simulate a realistic clinical setting. We show that current state-of-the-art LLMs do not accurately diagnose patients across all pathologies (performing significantly worse than physicians on average), follow neither diagnostic nor treatment guidelines, and cannot interpret laboratory results, thus posing a serious risk to the health of patients. Furthermore, we move beyond diagnostic accuracy and demonstrate that they cannot be easily integrated into existing workflows because they often fail to follow instructions and are sensitive to both the quantity and order of information. Overall, our analysis reveals that LLMs are currently not ready for clinical deployment while providing a dataset and framework to guide future studies.

show abstract

A Novel Approach for Medical E-Consent: Leveraging Language Models for Informed Consent Management

Naji,

Masmoudi,

Baazaoui Zghal

2024

Communications in Computer and Information Science

View full text Add to dashboard Cite

RadAdapt: Radiology Report Summarization via Lightweight Domain Adaptation of Large Language Models

Cited by 12 publications

References 0 publications

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

Clinical Text Summarization: Adapting Large Language Models Can Outperform Human Experts

Evaluating and Mitigating Limitations of Large Language Models in Clinical Decision Making

A Novel Approach for Medical E-Consent: Leveraging Language Models for Informed Consent Management

Contact Info

Product

Resources

About