AI chatbots not yet ready for clinical use

Abstract

As large language models (LLMs) expand and become more advanced, so does the natural language processing capabilities of conversational AI, or ‘chatbots’. OpenAI’s recent release, ChatGPT, uses transformer-based model and deep learning algorithms to enable human-like text generation and question-answering on general domain knowledge, while a healthcare-specific Large Language Model (LLM), Gatortron has focused on the real-world healthcare domain knowledge. As LLMs advance to achieve near human-level performances on medical question and answering benchmarks, it is probable that Conversational AI will soon be developed for use in healthcare. In this article we briefly compare the performance of two different approaches to generative LLMs - ChatGPT and Foresight NLP model, in forecasting relevant diagnoses based on clinical vignettes. We also discuss important considerations and limitations of transformer-based chatbots for clinical use.

Clinical decision‐making in benzodiazepine deprescribing by healthcare providers vs. AI‐assisted approach

“…Given their rapid rate of advancement, it is probable that language model-based conversational AI will soon be developed for healthcare use. 34 In the near future, the identified shortcomings from our study could be overcome, and AI-based clinical decision support tools could prove valuable in decreasing HCP burnout and enhancing the quality of care. More research involving the use of chatbots in different aspects of patients' drug therapy is needed and collecting such data can decipher chatbot potentials and limitations and consequently help in further development of AI models for therapy optimization.…”

“…Overall, our analysis revealed chatbots to be a promising tool for HCP. Given their rapid rate of advancement, it is probable that language model‐based conversational AI will soon be developed for healthcare use 34 . In the near future, the identified shortcomings from our study could be overcome, and AI‐based clinical decision support tools could prove valuable in decreasing HCP burnout and enhancing the quality of care.…”

“…Therefore, we strongly agree with Yeung et al who emphasized that without an adequately skilled 'prompter' or an 'astute user', chatbot may hallucinate misleading outputs with high certainty that can perpetuate harmful health beliefs, reinforce biases or pose significant clinical risk. 34 It will be essential in the future to educate HCP for optimal use of chatbots.…”

Assessing the research landscape and clinical utility of large language models: A scoping review

“…Another signi cant area for investigation is the incorporation of social determinants of health (SDOH) and real, anonymized, consented patient cases to add complexity to LLM assessments [20,30,40]. While there is an abundance of research utilizing USMLE standardized, textbook-style questions, there is a scarcity of studies that leverage real patient data, which has been recognized as a limitation in several papers [20,30,40]. By including social factors such as socioeconomic status (SES), race, and sexuality, biases in the answers generated by LLMs can be directly assessed.…”

Guidelines For Rigorous Evaluation of Clinical LLMs For Conversational Reasoning

“…These AI models have the ability to engage in nuanced and complex conversations 10–12 , making them ideal candidates for extracting comprehensive patient histories and assisting physicians in generating differential diagnoses. However, a considerable gap remains in assessing these models’ readiness for application in real-world clinical scenarios 13–15 .…”