Appropriateness and Readability of ChatGPT-4-Generated Responses for Surgical Treatment of Retinal Diseases

Momenaei, Bita; Wakabayashi, Taku; Shahlaee, Abtin; Durrani, Asad F.; Pandit, Saagar; Wang, K; Mansour, Hana A.; Abishek, R; Xu, David S.; Sridhar, Jayanth; Yonekawa, Yoshihiro; Kuriyan, Ajay E.

doi:10.1016/j.oret.2023.05.022

Cited by 66 publications

(40 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The already published literature is mostly focused on the use of ChatGpt (Microsoft Bing) in generative patient education material in different specialties of medicine, such as gastroenterology (10), cardiology (11), bariatric surgery (12, 13), otolaryngology (14), ophthalmology (15), and prostate cancer (16). In our study, we have not only evaluated the Microsoft Bing and Google Bard models individually but also compared them in their ability to produce quality patient education content.…”

Section: Discussionmentioning

confidence: 99%

Microsoft Bing vs. Google Bard in Neurology: A Comparative Study of AI-Generated Patient Education Material

Nazir,

Ahmad,

Mal

et al. 2023

Preprint

View full text Add to dashboard Cite

BackgroundPatient education is an essential component of healthcare, and artificial intelligence (AI) language models such as Google Bard and Microsoft Bing have the potential to improve information transmission and enhance patient care. However, it is crucial to evaluate the quality, accuracy, and understandability of the materials generated by these models before applying them in medical practice. This study aimed to assess and compare the quality of patient education materials produced by Google Bard and Microsoft Bing in response to questions related to neurological conditions.MethodsA cross-sectional study design was used to evaluate and compare the ability of Google Bard and Microsoft Bing to generate patient education materials. The study included the top ten prevalent neurological diseases based on WHO prevalence data. Ten board-certified neurologists and four neurology residents evaluated the responses generated by the models on six quality metrics. The scores for each model were compiled and averaged across all measures, and the significance of any observed variations was assessed using an independent t-test.ResultsGoogle Bard performed better than Microsoft Bing in all six-quality metrics, with an overall mean score of 79% and 69%, respectively. Google Bard outperformed Microsoft Bing in all measures for eight questions, while Microsoft Bing performed marginally better in terms of objectivity and clarity for the epilepsy query.ConclusionThis study showed that Google Bard performs better than Microsoft Bing in generating patient education materials for neurological diseases. However, healthcare professionals should take into account both AI models’ advantages and disadvantages when providing support for health information requirements. Future studies can help determine the underlying causes of these variations and guide cooperative initiatives to create more user-focused AI-generated patient education materials. Finally, researchers should consider the perception of patients regarding AI-generated patient education material and its impact on implementing these solutions in healthcare settings.

show abstract

Section: Discussionmentioning

confidence: 99%

Microsoft Bing vs. Google Bard in Neurology: A Comparative Study of AI-Generated Patient Education Material

Nazir,

Ahmad,

Mal

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Large language models (LLMs) are increasingly being integrated into medical decision-making and patient education and have the potential to be a medical artificial intelligence catalyst in ophthalmology . LLM chatbots have demonstrated encouraging and consistent performances on stimulated Ophthalmic Knowledge Assessment Program examination questions .…”

Section: Introductionmentioning

confidence: 99%

“…arge language models (LLMs) are increasingly being integrated into medical decision-making and patient education and have the potential to be a medical artificial intelligence catalyst in ophthalmology. [1][2][3][4] LLM chatbots have demonstrated encouraging and consistent performances on stimulated Ophthalmic Knowledge Assessment Program examination questions. 3 Furthermore, the diagnostic capabilities of LLM chatbots compared with 3 ophthalmology trainees for glaucoma and 2 specialists for retina indicate their potential role in enhancing objective and efficient clinical diagnoses.…”

mentioning

confidence: 95%

“…3 Furthermore, the diagnostic capabilities of LLM chatbots compared with 3 ophthalmology trainees for glaucoma and 2 specialists for retina indicate their potential role in enhancing objective and efficient clinical diagnoses. 1,2,5 While these studies showcase the potential of LLM chatbots in specific domains, a broader evaluation of their accuracy, including in comparison with attending-level ophthalmologists, is warranted, particularly for addressing real-life clinical case scenarios. 2,[5][6][7] In this study, we compared an LLM chatbot's responses with those of fellowship-trained glaucoma and retina specialists to explore the potential of LLMs in clinical ophthalmology.…”

mentioning

confidence: 99%

“…1,2,5 While these studies showcase the potential of LLM chatbots in specific domains, a broader evaluation of their accuracy, including in comparison with attending-level ophthalmologists, is warranted, particularly for addressing real-life clinical case scenarios. 2,[5][6][7] In this study, we compared an LLM chatbot's responses with those of fellowship-trained glaucoma and retina specialists to explore the potential of LLMs in clinical ophthalmology.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Assessment of a Large Language Model’s Responses to Questions and Cases About Glaucoma and Retina Management

Huang,

Hirabayashi,

Barna

et al. 2024

JAMA Ophthalmol

View full text Add to dashboard Cite

ImportanceLarge language models (LLMs) are revolutionizing medical diagnosis and treatment, offering unprecedented accuracy and ease surpassing conventional search engines. Their integration into medical assistance programs will become pivotal for ophthalmologists as an adjunct for practicing evidence-based medicine. Therefore, the diagnostic and treatment accuracy of LLM-generated responses compared with fellowship-trained ophthalmologists can help assess their accuracy and validate their potential utility in ophthalmic subspecialties.ObjectiveTo compare the diagnostic accuracy and comprehensiveness of responses from an LLM chatbot with those of fellowship-trained glaucoma and retina specialists on ophthalmological questions and real patient case management.Design, Setting, and ParticipantsThis comparative cross-sectional study recruited 15 participants aged 31 to 67 years, including 12 attending physicians and 3 senior trainees, from eye clinics affiliated with the Department of Ophthalmology at Icahn School of Medicine at Mount Sinai, New York, New York. Glaucoma and retina questions (10 of each type) were randomly selected from the American Academy of Ophthalmology’s Commonly Asked Questions. Deidentified glaucoma and retinal cases (10 of each type) were randomly selected from ophthalmology patients seen at Icahn School of Medicine at Mount Sinai–affiliated clinics. The LLM used was GPT-4 (version dated May 12, 2023). Data were collected from June to August 2023.Main Outcomes and MeasuresResponses were assessed via a Likert scale for medical accuracy and completeness. Statistical analysis involved the Mann-Whitney U test and the Kruskal-Wallis test, followed by pairwise comparison.ResultsThe combined question-case mean rank for accuracy was 506.2 for the LLM chatbot and 403.4 for glaucoma specialists (n = 831; Mann-Whitney U = 27976.5; P &lt; .001), and the mean rank for completeness was 528.3 and 398.7, respectively (n = 828; Mann-Whitney U = 25218.5; P &lt; .001). The mean rank for accuracy was 235.3 for the LLM chatbot and 216.1 for retina specialists (n = 440; Mann-Whitney U = 15518.0; P = .17), and the mean rank for completeness was 258.3 and 208.7, respectively (n = 439; Mann-Whitney U = 13123.5; P = .005). The Dunn test revealed a significant difference between all pairwise comparisons, except specialist vs trainee in rating chatbot completeness. The overall pairwise comparisons showed that both trainees and specialists rated the chatbot’s accuracy and completeness more favorably than those of their specialist counterparts, with specialists noting a significant difference in the chatbot’s accuracy (z = 3.23; P = .007) and completeness (z = 5.86; P &lt; .001).Conclusions and RelevanceThis study accentuates the comparative proficiency of LLM chatbots in diagnostic accuracy and completeness compared with fellowship-trained ophthalmologists in various clinical scenarios. The LLM chatbot outperformed glaucoma specialists and matched retina specialists in diagnostic and treatment accuracy, substantiating its role as a promising diagnostic adjunct in ophthalmology.

show abstract

Large Language Models and the Shoreline of Ophthalmology

Young,

Zhao

2024

JAMA Ophthalmol

View full text Add to dashboard Cite

show abstract

Appropriateness and Readability of ChatGPT-4-Generated Responses for Surgical Treatment of Retinal Diseases

Cited by 66 publications

References 14 publications

Microsoft Bing vs. Google Bard in Neurology: A Comparative Study of AI-Generated Patient Education Material

Microsoft Bing vs. Google Bard in Neurology: A Comparative Study of AI-Generated Patient Education Material

Assessment of a Large Language Model’s Responses to Questions and Cases About Glaucoma and Retina Management

Large Language Models and the Shoreline of Ophthalmology

Contact Info

Product

Resources

About