Performance of ChatGPT as an AI-assisted decision support tool in medicine: a proof-of-concept study for interpreting symptoms and management of common cardiac conditions (AMSTELHEART-2)

Abstract

Background: It is thought that ChatGPT, an advanced language model developed by OpenAI, may in the future serve as an AI-assisted decision support tool in medicine. Objective: To evaluate the accuracy of ChatGPTs recommendations on medical questions related to common cardiac symptoms or conditions. Methods: We tested ChatGPTs ability to address medical questions in two ways. First, we assessed its accuracy in correctly answering cardiovascular trivia questions (n=50), based on quizzes for medical professionals. Second, we entered 20 clinical case vignettes on the ChatGPT platform and evaluated its accuracy compared to expert opinion and clinical course. Results: We found that ChatGPT correctly answered 74% of the trivia questions, with slight variation in accuracy in the domains coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%). In the case vignettes, ChatGPTs response matched in 90% of the cases with the actual advice given. In more complex cases, where physicians (general practitioners) asked other physicians (cardiologists) for assistance or decision support, ChatGPT was correct in 50% of cases, and often provided incomplete or inappropriate recommendations when compared with expert consultation. Conclusions: Our study suggests that ChatGPT has potential as an AI-assisted decision support tool in medicine, particularly for straightforward, low-complex medical questions, but further research is needed to fully evaluate its potential.

ChatGPT’s Performance in Cardiac Arrest and Bradycardia Simulations Using the American Heart Association's Advanced Cardiovascular Life Support Guidelines: Exploratory Study

“…Although the overall accuracy of ChatGPT was lower than the standard passing threshold in academics of 70%, the demonstrated accuracy of ChatGPT was comparable to one other study of complex cardiac clinical vignettes which had 50% accuracy (50/100). Compared with experts, ChatGPT gave inaccurate or incomplete responses [ 29 ]. Even with these caveats, this proof-of-concept study illustrates the potential and perils of using ChatGPT for real-time decision support in clinical settings.…”

Exploring the role of large language models in radiation emergency response

“…Of the 20 suggestions that scored the highest, 9 were generated by ChatGPT [38] examined the accuracy and reproducibility of ChatGPT in answering questions regarding knowledge, management, and emotional support for cirrhosis and hepatocellular carcinoma (HCC). In [39] it was found that ChatGPT correctly answered 74% of the trivia questions related to heart diseases. Specifically, the accuracy of ChatGPT scored impressively in the domains of coronary artery disease (80%), pulmonary and venous thrombotic embolism (80%), atrial fibrillation (70%), heart failure (80%) and cardiovascular risk management (60%) [40] evaluated ChatGPT as a support tool for breast tumor board decision making [41] assessed ChatGPT's capacity for clinical decision support in paediatrics [7] evaluated the capacity of ChatGPT as a clinical decision support in triaging patients for appropriate imaging services [42] did a comparative analysis of humans and LLMs in decision making abilities.…”

Künstliche Intelligenz in der kardiovaskulären Medizin – Status und Perspektiven

“…Aktuelle Entwicklungen unter Verwendung von ChatGPT als KIgestütztes Entscheidungshilfe-Tool in der Medizin eröffnet Perspektiven [11], aktuell vor allem noch bei wenig komplexen medizinischen Fragen. Es sind jedoch weitere wissenschaftliche Untersuchungen erforderlich, um das Potenzial vollständig zu bewerten.…”