Large Language Model (LLM)-Powered Chatbots Fail to Generate Guideline-Consistent Content on Resuscitation and May Provide Potentially Harmful Advice

Abstract

Introduction:Innovative large language model (LLM)-powered chatbots, which are extremely popular nowadays, represent potential sources of information on resuscitation for the general public. For instance, the chatbot-generated advice could be used for purposes of community resuscitation education or for just-in-time informational support of untrained lay rescuers in a real-life emergency.Study Objective:This study focused on assessing performance of two prominent LLM-based chatbots, particularly in terms of quality of the chatbot-generated advice on how to give help to a non-breathing victim.Methods:In May 2023, the new Bing (Microsoft Corporation, USA) and Bard (Google LLC, USA) chatbots were inquired (n = 20 each): “What to do if someone is not breathing?” Content of the chatbots’ responses was evaluated for compliance with the 2021 Resuscitation Council United Kingdom guidelines using a pre-developed checklist.Results:Both chatbots provided context-dependent textual responses to the query. However, coverage of the guideline-consistent instructions on help to a non-breathing victim within the responses was poor: mean percentage of the responses completely satisfying the checklist criteria was 9.5% for Bing and 11.4% for Bard (P >.05). Essential elements of the bystander action, including early start and uninterrupted performance of chest compressions with adequate depth, rate, and chest recoil, as well as request for and use of an automated external defibrillator (AED), were missing as a rule. Moreover, 55.0% of Bard’s responses contained plausible sounding, but nonsensical guidance, called artificial hallucinations, that create risk for inadequate care and harm to a victim.Conclusion:The LLM-powered chatbots’ advice on help to a non-breathing victim omits essential details of resuscitation technique and occasionally contains deceptive, potentially harmful directives. Further research and regulatory measures are required to mitigate risks related to the chatbot-generated misinformation of public on resuscitation.

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3...…”

“…Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3.2%) 13,83,84 , Large Language Model Meta-AI (LLaMA; n=3/124, 2.4%) 55 , or Claude by Anthropic (n=1/124, 0.8%) 55 . The majority of applications were p...…”

“…In addition, data-related limitations were identified, including limited access to data on the internet (n=22/89, 24.7%) 38,39,41,43,54–57,59,60,64,76,79,82–84,88,91,94,96,104,109 , the undisclosed origin of training data (n=36/89, 40.5%) 25,26,29,30,32,34,36,37,40,46,47,50,51,53–60,64,65,70,71,76,82,83,91,94–96,101,105,109 , limitations in providing, evaluating, and validating references (n=20/89, 22.5%) 45,49,54–57,65,71,73,76,80,83,85,91,94,96,98,101,103,105 , and storage/processing of sensitive health information (n=8/89, 9%) 13,34,46,55,62,76,83,109 . Further second-order concepts included black-box algorithms, i.e., non-explainable AI (n=12/89, 13.5%) 27,36,55,57,65,73,76,83,91,94,103,105 , limited engagement and dialogue capabilities (n=10/89) 13,27,28,37,…”

“…The evaluation of limitations in output data yielded 7 second-order codes concerning the non-reproducibility (n=38/89, 42.7%) 28,29,34,38,39,41,43,45,46,49,54–61,64,65,71–73,76,80,82,83,85,90,91,94,96,98,99,101,103–105 , non-comprehensiveness (n=78/89, 87.6%) 13,25,26,28–30,32–44,46,48–62,64,65,67–79,81–98,100,102–107,109–111 , incorrectness (n=78/89, 87.6%) 13,25–44,46,49–52,54–62,64–66,69–79,81–85,87–107,109–111 , (un-)safeness (n=39/89, 43.8%) 28,30,35,37,39,40,42–44,46,50,51,57–60,62,64,65,69,70,73,74,76,78–80,82,84,85,91…”

“…For non-reproducibility, key concepts included the non-deterministic nature of the output, e.g., due to inconsistent results across multiple iterations (n=34/89, 38.2%) 28,29,34,38,39,41,43,46,58–61,72,76,82,90,94,98,99,101,103,104 and the inability to provide reliable references (n=20/89, 22.5%) 45,49,54–57,65,71,73,76,80,83,85,91,94,96,98,101,103,105 . Non-comprehensiveness included nine concepts related to generic/non-personalized output (n=34/89, 38.2%) 13,28,30,34,37,38,41,43,49,51,56,57,59,61,65,70,77,79,81,84–86,90,94,95,100,102–107,110 , incompleteness of output (n=68/89, 76.4%) 13,25,26,28–30,32,34–39,41–44,46,49–52,55–62,64,65,67–69,72–77,79,81–86,89–98,…”