Popular large language model chatbots’ accuracy, comprehensiveness, and self-awareness in answering ocular symptom queries

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3...…”

“…A total of 18 (n=18/89, 20.2%) studies reported the presence of conflicts of interest and funding. 13,24,38,40,54,58,59,67,69–71,74,80,84,96,103,105,111 Most studies did not report information about the institutional review board (IRB) approval (n=55/89, 61.8%) or deemed IRB approval unnecessary (n=28/89, 31.5%). Six studies obtained IRB approval (n=6/89, 6.7%).…”

“…Most reports evaluated LLMs in English (n=88/89, 98.9%) 13,24–103,105–111 , followed by Arabic (n=2/84, 2.3%) 32,104 , Mandarin (n=2/84, 2.3%) 36,75 , and Korean or Spanish (n=1/89, 1.1%, respectively) 75 . The top-five specialties studied were ophthalmology (n=10/89, 11.2%) 37,40,48,51,65,74,97,98,100,101 , gastro-enterology (n=9/89, 10.1%) 25,32,34,36,39,61,62,72,96 , head and neck surgery/otolaryngology (n=8/89, 9%) 35,42,56,64,66,76,78,79 , and radiology 59,70,88–90,110 or plastic surgery 45,47,49,102,107,108 (n=6/89, 6.7%, respectively). A schematic illustration of the identified concepts of LLM applications in patient care is shown in Figure 2.…”

“…The evaluation of limitations in output data yielded 7 second-order codes concerning the non-reproducibility (n=38/89, 42.7%) 28,29,34,38,39,41,43,45,46,49,54–61,64,65,71–73,76,80,82,83,85,90,91,94,96,98,99,101,103–105 , non-comprehensiveness (n=78/89, 87.6%) 13,25,26,28–30,32–44,46,48–62,64,65,67–79,81–98,100,102–107,109–111 , incorrectness (n=78/89, 87.6%) 13,25–44,46,49–52,54–62,64–66,69–79,81–85,87–107,109–111 , (un-)safeness (n=39/89, 43.8%) 28,30,35,37,39,40,42–44,46,50,51,57–60,62,64,65,69,70,73,74,76,78–80,82,84,85,91…”

“…Some of the incorrect information could be attributed to what is commonly known as hallucination (n=38/89, 42.7%) 25,28,32,33,35–38,40–44,49–51,57–60,65,73,74,76,77,81,83,85,91,94,96–98,100,103,106,107,109 , i.e., the creation of entirely fictitious or false information that has no basis in the input provided or in reality (e.g., “You may be asked to avoid eating or drinking for a few hours before the scan” for a bone scan). However, numerous instances of misinformation were more appropriately classified under alternative concepts of the original psychiatric analogy, as described in detail by Currie et al 43,112,113 These include illusion (n=12/89, 13.5%) 28,36,38,43,57,59,77,78,85,88,94,105 , which is characterized by the generation of deceptive perceptions or the distortion of information by conflating similar but separate concepts (e.g., suggesting that MRI-type sounds might be experienced during standard nuclear medicine imaging), delirium (n=34/89, 38.2%) 13,26,28,30,37,43,50,58,59,61,65,70,72–75,77,79,81–85,90–92,94,95,98,102,103,107,109,110 , which indicates significant gaps in vital information, resulting in a fragmented or confused understanding of a subject (e.g., omission of crucial information about caffeine cessation for stress myocardial perfusion scans), extrapolation (n=11/89, 12.4%) 43,59,…”