ChatGPT and Beyond: An overview of the growing field of large language models and their use in ophthalmology

Purpose: The purpose of this study was to assess the accuracy and completeness of 3 large language models (LLMs) to generate information about antibody–drug conjugate (ADC)-associated ocular toxicities. Methods: There were 22 questions about ADCs, tisotumab vedotin, and mirvetuximab soravtansine that were developed and input into ChatGPT 4.0, Bard, and LLaMa. Answers were rated by 4 ocular toxicity experts using standardized 6-point Likert scales on accuracy and completeness. ANOVA tests were conducted for comparison between the 3 subgroups, followed by pairwise t-tests. Interrater variability was assessed with Fleiss kappa tests. Results: The mean accuracy score was 4.62 (SD 0.89) for ChatGPT, 4.77 (SD 0.90) for Bard, and 4.41 (SD 1.09) for LLaMA. Both ChatGPT (P = 0.03) and Bard (P = 0.003) scored significantly better for accuracy when compared with LLaMA. The mean completeness score was 4.43 (SD 0.91) for ChatGPT, 4.57 (SD 0.93) for Bard, and 4.42 (SD 0.99) for LLaMA. There were no significant differences in completeness scores between groups. Fleiss kappa assessment for interrater variability was good (0.74) for accuracy and fair (0.31) for completeness. Conclusions: All 3 LLMs had relatively high accuracy and completeness ratings, showing LLMs are able to provide sufficient answers for niche topics of ophthalmology. Our results indicate that ChatGPT and Bard may be slightly better at providing more accurate answers than LLaMA. As further research and treatment plans are developed for ADC-associated ocular toxicities, these LLMs should be reassessed to see if they provide complete and accurate answers that remain in line with current medical knowledge.

show abstract

The Use of Artificial Intelligence Based Chat Bots in Ophthalmology Triage

David,

ZLOTO,

Katz

et al. 2024

Preprint

View full text Add to dashboard Cite

Purpose - To evaluate AI-based chat bots ability to accurately answer common patient's questions in the field of ophthalmology. Methods - An experienced ophthalmologist curated a set of 20 representative questions and responses were sought from two AI generative models: OpenAI's ChatGPT and Google's Bard (Gemini Pro). Eight expert ophthalmologists from different sub-specialties assessed each response, blinded to the source, and ranked them by three metrics – accuracy, comprehensiveness, and clarity, on a 1-5 scale. Results - For accuracy, ChatGPT scored a median of 4.0, whereas Bard scored a median of 3.0. In terms of comprehensiveness, ChatGPT achieved a median score of 4.5, compared to Bard which scored a median of 3.0. Regarding clarity, ChatGPT maintained a higher score with a median of 5.0, compared to Bard's median score of 4.0. All comparisons were statistically significant (p<0.001). Conclusion - AI-based chat bots can provide relatively accurate and clear responses for addressing common ophthalmological inquiries. ChatGPT surpassed Bard in all measured metrics. While these AI models exhibit promise, further research is indicated to improve their performance and allow them to be used as a reliable medical tool.

show abstract

ChatGPT and Beyond: An overview of the growing field of large language models and their use in ophthalmology

Cited by 4 publications

References 60 publications

Generative AI and Otolaryngology—Head & Neck Surgery

Generative AI and Otolaryngology—Head & Neck Surgery

Accuracy and Completeness of Large Language Models About Antibody–Drug Conjugates and Associated Ocular Adverse Effects

The Use of Artificial Intelligence Based Chat Bots in Ophthalmology Triage

Contact Info

Product

Resources

About