Comparison of Ophthalmologist and Large Language Model Chatbot Responses to Online Patient Eye Care Questions

Abstract

ImportanceLarge language models (LLMs) like ChatGPT appear capable of performing a variety of tasks, including answering patient eye care questions, but have not yet been evaluated in direct comparison with ophthalmologists. It remains unclear whether LLM-generated advice is accurate, appropriate, and safe for eye patients.ObjectiveTo evaluate the quality of ophthalmology advice generated by an LLM chatbot in comparison with ophthalmologist-written advice.Design, Setting, and ParticipantsThis cross-sectional study used deidentified data from an online medical forum, in which patient questions received responses written by American Academy of Ophthalmology (AAO)–affiliated ophthalmologists. A masked panel of 8 board-certified ophthalmologists were asked to distinguish between answers generated by the ChatGPT chatbot and human answers. Posts were dated between 2007 and 2016; data were accessed January 2023 and analysis was performed between March and May 2023.Main Outcomes and MeasuresIdentification of chatbot and human answers on a 4-point scale (likely or definitely artificial intelligence [AI] vs likely or definitely human) and evaluation of responses for presence of incorrect information, alignment with perceived consensus in the medical community, likelihood to cause harm, and extent of harm.ResultsA total of 200 pairs of user questions and answers by AAO-affiliated ophthalmologists were evaluated. The mean (SD) accuracy for distinguishing between AI and human responses was 61.3% (9.7%). Of 800 evaluations of chatbot-written answers, 168 answers (21.0%) were marked as human-written, while 517 of 800 human-written answers (64.6%) were marked as AI-written. Compared with human answers, chatbot answers were more frequently rated as probably or definitely written by AI (prevalence ratio [PR], 1.72; 95% CI, 1.52-1.93). The likelihood of chatbot answers containing incorrect or inappropriate material was comparable with human answers (PR, 0.92; 95% CI, 0.77-1.10), and did not differ from human answers in terms of likelihood of harm (PR, 0.84; 95% CI, 0.67-1.07) nor extent of harm (PR, 0.99; 95% CI, 0.80-1.22).Conclusions and RelevanceIn this cross-sectional study of human-written and AI-generated responses to 200 eye care questions from an online advice forum, a chatbot appeared capable of responding to long user-written eye health posts and largely generated appropriate responses that did not differ significantly from ophthalmologist-written responses in terms of incorrect information, likelihood of harm, extent of harm, or deviation from ophthalmologist community standards. Additional research is needed to assess patient attitudes toward LLM-augmented ophthalmologists vs fully autonomous AI content generation, to evaluate clarity and acceptability of LLM-generated answers from the patient perspective, to test the performance of LLMs in a greater variety of clinical contexts, and to determine an optimal manner of utilizing LLMs that is ethical and minimizes harm.

Assessment of a Large Language Model’s Responses to Questions and Cases About Glaucoma and Retina Management

“…Another corneal disease study emphasized the updated chatbot’s superiority over its predecessor and its promising accuracy, although not consistently surpassing human experts . Another study found that LLM-generated ophthalmic advice to online forum questions is nearly as safe and accurate as ophthalmologists . These studies emphasize the emerging role of LLM chatbots in ophthalmology, highlighting their strengths and areas needing refinement.…”

“…Large language models (LLMs) are increasingly being integrated into medical decision-making and patient education and have the potential to be a medical artificial intelligence catalyst in ophthalmology . LLM chatbots have demonstrated encouraging and consistent performances on stimulated Ophthalmic Knowledge Assessment Program examination questions .…”

Utility of ChatGPT for Automated Creation of Patient Education Handouts: An Application in Neuro-Ophthalmology

“…8 Although not an investigation into patient educational handouts, the most extensive study to-date evaluated ophthalmologist- vs. ChatGPT-derived responses to online forum patient eye care queries. 9 Across 200 responses, human- and non–human-derived texts were distinguishable in 61% of cases, although both did not significantly differ in the writing of incorrect information, likelihood and extent of harm, and deviation from practice standards. 9 These encouraging findings are in relative keeping with the results of our study, which evaluated ChatGPT in the separate context of writing patient educational handouts.…”

“…9 Across 200 responses, human- and non–human-derived texts were distinguishable in 61% of cases, although both did not significantly differ in the writing of incorrect information, likelihood and extent of harm, and deviation from practice standards. 9 These encouraging findings are in relative keeping with the results of our study, which evaluated ChatGPT in the separate context of writing patient educational handouts. By leveraging ChatGPT, there remains potential to efficiently educate patients, empowering them to better engage in their health care by reducing knowledge asymmetry and related power-imbalance in the therapeutic relationship.…”

Leveraging ChatGPT for ophthalmic education: A critical appraisal

“…The study suggests that with enhancements, ChatGPT holds potential for application in clinical settings for efficient and objective glaucoma diagnoses. ChatGPT has also been utilized in assessing patient questionnaire on Optic Disc Drusens, 22 comparative analysis of information on myopia with GPT 3 and Google Bard, 23 compare online patient eye care questions with Ophthalmologists, triage of Ophthalmic conditions, 24 successfully navigating through the French language version of the European Board of Ophthalmology (EBO) examination, 25 assessing the performance in Italian Residency Admission National Exam Compared to 15,869 Medical Graduates, 26 literature review of dry eye diseases, 27 writing operative notes, 28 response for patient based questionnaire on vernal keratoconjunctivitis, 29 and Ophthalmic knowledge assessment and information on common retinal diseases. 30 In terms of advancements, integrating AI like ChatGPT with imaging technologies in ophthalmology, such as Optical Coherence Tomography (OCT) and fundus photography, could significantly enhance diagnostic accuracy and speed.…”