Large language models encode clinical knowledge

Abstract

Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of models typically rely on automated evaluations based on limited benchmarks. Here, to address these limitations, we present MultiMedQA, a benchmark combining six existing medical question answering datasets spanning professional medicine, research and consumer queries and a new dataset of medical questions searched online, HealthSearchQA. We propose a human evaluation framework for model answers along multiple axes including factuality, comprehension, reasoning, possible harm and bias. In addition, we evaluate Pathways Language Model1 (PaLM, a 540-billion parameter LLM) and its instruction-tuned variant, Flan-PaLM2 on MultiMedQA. Using a combination of prompting strategies, Flan-PaLM achieves state-of-the-art accuracy on every MultiMedQA multiple-choice dataset (MedQA3, MedMCQA4, PubMedQA5 and Measuring Massive Multitask Language Understanding (MMLU) clinical topics6), including 67.6% accuracy on MedQA (US Medical Licensing Exam-style questions), surpassing the prior state of the art by more than 17%. However, human evaluation reveals key gaps. To resolve this, we introduce instruction prompt tuning, a parameter-efficient approach for aligning LLMs to new domains using a few exemplars. The resulting model, Med-PaLM, performs encouragingly, but remains inferior to clinicians. We show that comprehension, knowledge recall and reasoning improve with model scale and instruction prompt tuning, suggesting the potential utility of LLMs in medicine. Our human evaluations reveal limitations of today’s models, reinforcing the importance of both evaluation frameworks and method development in creating safe, helpful LLMs for clinical applications.

RETRACTED: New Artificial Intelligence ChatGPT Performs Poorly on the 2022 Self-assessment Study Program for Urology

“…More recently, however, advances in large language models (LLMs) have created an opportunity to adapt AI technology into a tool for mediating human interaction. 6,7…”

“…More recently, however, advances in large language models (LLMs) have created an opportunity to adapt AI technology into a tool for mediating human interaction. 6,7 LLMs are "foundation models," or large, pretrained, multilayer deep neural networks that leverage contextual relationships between words, phrases, and concepts to predict the likelihood of the next sequence of words. 8 With adequate training and application, these AI systems can process complex information, analyze relationships between ideas, and generate coherent responses to an inquiry.…”

Harnessing Generative Artificial Intelligence to Improve Efficiency Among Urologists: Welcome ChatGPT

“…5,6 Only weeks after the release of ChatGPT, Google's DeepMind released MedPaLM, an LLM designed to answer medical questions. 7 What makes ChatGPT unique is that it is free and publicly available, boasts a user-friendly interface, and was trained with a data set that is larger than previous LLMs. ChatGPT has made generative AI technology more tangible than ever before, as evidenced by its rapid uptake by the public.…”

Machine learning in computational histopathology: Challenges and opportunities

“…In parallel, large scale diffusion models 274 have democratized the generation of high‐resolution image data using text‐prompts single sentence alone. The ramifications of this technology are only just being explored in the context of medicine 275 but the next half decade will inevitably find their utility in CPATH.…”