Guidelines For Rigorous Evaluation of Clinical LLMs For Conversational Reasoning

Abstract

Large Language Models (LLMs) show promise for medical diagnosis, but traditional evaluations using static exam questions overlook the complexity of real-world clinical dialogues. We introduce a multi-agent conversational framework where doctor-AI and patient-AI agents interact to diagnose medical conditions, evaluated by a grader-AI agent and medical experts. We assessed the diagnostic accuracy of GPT-4 and GPT-3.5, in conversational versus static settings using 140 cases focusing on skin diseases. Our study revealed a decline in diagnostic accuracy, unmasking key limitations in LLMs’ ability to integrate details from conversational interactions to improve diagnostic performance. We introduced Conversational Summarization, a technique that enhanced performance, and expert review identified deficiencies compared to human dermatologists in comprehensive history gathering, appropriate use of terminology, and reliability. Our findings advocate for nuanced, rigorous evaluation of LLMs before clinical integration, and our framework represents a significant advancement toward responsible testing methodologies in medicine.

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3...…”

“…In addition, data-related limitations were identified, including limited access to data on the internet (n=22/89, 24.7%) 38,39,41,43,54–57,59,60,64,76,79,82–84,88,91,94,96,104,109 , the undisclosed origin of training data (n=36/89, 40.5%) 25,26,29,30,32,34,36,37,40,46,47,50,51,53–60,64,65,70,71,76,82,83,91,94–96,101,105,109 , limitations in providing, evaluating, and validating references (n=20/89, 22.5%) 45,49,54–57,65,71,73,76,80,83,85,91,94,96,98,101,103,105 , and storage/processing of sensitive health information (n=8/89, 9%) 13,34,46,55,62,76,83,109 . Further second-order concepts included black-box algorithms, i.e., non-explainable AI (n=12/89, 13.5%) 27,36,55,57,65,73,76,83,91,94,103,105 , limited engagement and dialogue capabilities (n=10/89) 13,27,28,37,…”

“…A minority of reports identified biases in the output, which were related to language (n=2/89, 2.3%) 32,36 , insurance status 103 , underserved racial groups 26 , or underrepresented procedures 34 (n=1/89, 1.1%, each). Finally, many authors suggested that performance was related to the prompting/input provided or the environment, i.e., depending on the evidence (n=7/89, 7.9%) 52,68,69,71,81,82,95 , complexity (n=11/89, 12.4%) 28,34,44,46,70,74,76,79,94,102 , specificity (n=13/89, 14.6%) 27,38,41,56,70,72,74,76,78,81,95,100,101 , quantity (n=3/89, 3.4%) 26,52,74 of the input, type of conversation (n=3/89, 3.4%) 27,51,90 , or the appropriateness of the output related to the target group (n=9/89, 10.1%) 46,49,51,54,72,90,93,97,109 , provider/organization (n=4/89, 4.5%) 13,50,60,88 , and local/national medical resources (n=5/89, 5.6%) 34,50,58,60,73 . Figure 3 illustrates the hierarchical tree structure and quantity of the codes derived from the thematic synthesis of limitations.…”

“…One study (n=1/89, 1.1%) was published in 2022 24 , 84 (n=84/89, 94.4%) in 2023 13, , and 4 (n=4/89, 4.5%) in 2024 [108][109][110][111] (all of which were peer-reviewed publications of preprints published in 2023). Most studies were quantitative non-randomized (n=84/89, 94.4%) 13,[25][26][27]103,104,106,107,[109][110][111] , 4 (n=4/89, 4.5%) 28,102,105,108 had a qualitative study design, and one (n=1/89, 1.1%) 24 was quantitative randomized according to the MMAT 2018 criteria. However, the LLM outputs were often first analyzed quantitatively but followed by a qualitative analysis of certain responses.…”

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,[26][27][28][29][31][32][33][34][36][37][38][39][40][42][43][44][45][46][47][48][49][52][53][54][56][57][58][59][60][61]63,[65][66][67]71,72,74,75,77,78,[81][82][83][84][85][86]…”