A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)

Abstract

Importance: Large Language Models (LLMs) can assist in a wide range of healthcare-related activities. Current approaches to evaluating LLMs make it difficult to identify the most impactful LLM application areas. Objective: To summarize the current evaluation of LLMs in healthcare in terms of 5 components: evaluation data type, healthcare task, Natural Language Processing (NLP)/Natural Language Understanding (NLU) task, dimension of evaluation, and medical specialty. Data Sources: A systematic search of PubMed and Web of Science was performed for studies published between 01-01-2022 and 02-19-2024. Study Selection: Studies evaluating one or more LLMs in healthcare. Data Extraction and Synthesis: Three independent reviewers categorized 519 studies in terms of data used in the evaluation, the healthcare tasks (the what) and the NLP/NLU tasks (the how) examined, the dimension(s) of evaluation, and the medical specialty studied. Results: Only 5% of reviewed studies utilized real patient care data for LLM evaluation. The most popular healthcare tasks were assessing medical knowledge (e.g. answering medical licensing exam questions, 44.5%), followed by making diagnoses (19.5%), and educating patients (17.7%). Administrative tasks such as assigning provider billing codes (0.2%), writing prescriptions (0.2%), generating clinical referrals (0.6%) and clinical notetaking (0.8%) were less studied. For NLP/NLU tasks, the vast majority of studies examined question answering (84.2%). Other tasks such as summarization (8.9%), conversational dialogue (3.3%), and translation (3.1%) were infrequent. Almost all studies (95.4%) used accuracy as the primary dimension of evaluation; fairness, bias and toxicity (15.8%), robustness (14.8%), deployment considerations (4.6%), and calibration and uncertainty (1.2%) were infrequently measured. Finally, in terms of medical specialty area, most studies were in internal medicine (42%), surgery (11.4%) and ophthalmology (6.9%), with nuclear medicine (0.6%), physical medicine (0.4%) and medical genetics (0.2%) being the least represented. Conclusions and Relevance: Existing evaluations of LLMs mostly focused on accuracy of question answering for medical exams, without consideration of real patient care data. Dimensions like fairness, bias and toxicity, robustness, and deployment considerations received limited attention. To draw meaningful conclusions and improve LLM adoption, future studies need to establish a standardized set of LLM applications and evaluation dimensions, perform evaluations using data from routine care, and broaden testing to include administrative tasks as well as multiple medical specialties. Keywords: Large Language Models, Generative Artificial Intelligence, Healthcare, Dimensions of Evaluation, Evaluation Metrics.

QUEST-AI: A System for Question Generation, Verification, and Refinement using AI for USMLE-Style Exams

“…10 A systematic review by Bedi et al highlights the various healthcare applications of LLMs. 11 Studies have demonstrated their utility in tasks such as diagnosis 12 , medical report generation 13 , treatment recommendations 14 , and clinical referrals. 14,15 16 Additionally, Fraser et al studied the diagnostic accuracy of LLMs, providing further evidence of their potential in clinical settings.…”

Evaluating the Effectiveness and Safety of Large Language Model in Generating Type 2 Diabetes Mellitus Management Plans: A Comparative Study with Medical Experts Based on Real Patient Records

“…Studies have noted that while LLMs perform well in tasks like answering medical exam questions, their application in direct patient care and other complex medical scenarios remains underexplored and often lacks integration with real patient data. [3]…”

“…Studies have noted that while LLMs perform well in tasks like answering medical exam questions, their application in direct patient care and other complex medical scenarios remains underexplored and often lacks integration with real patient data. [3] Moreover, the evaluation of these models in healthcare has often focused narrowly on specific tasks, such as NLP tasks related to summarization and conversation, without a broad application across various medical specialties. This has limited the understanding of their broader potential and areas where they may not perform as expected.…”