Evaluating ChatGPT on Orbital and Oculofacial Disorders: Accuracy and Readability Insights

Abstract

Purpose: To assess the accuracy and readability of responses generated by the artificial intelligence model, ChatGPT (version 4.0), to questions related to 10 essential domains of orbital and oculofacial disease. Methods: A set of 100 questions related to the diagnosis, treatment, and interpretation of orbital and oculofacial diseases was posed to ChatGPT 4.0. Responses were evaluated by a panel of 7 experts based on appropriateness and accuracy, with performance scores measured on a 7-item Likert scale. Inter-rater reliability was determined via the intraclass correlation coefficient. Results: The artificial intelligence model demonstrated accurate and consistent performance across all 10 domains of orbital and oculofacial disease, with an average appropriateness score of 5.3/6.0 (“mostly appropriate” to “completely appropriate”). Domains of cavernous sinus fistula, retrobulbar hemorrhage, and blepharospasm had the highest domain scores (average scores of 5.5 to 5.6), while the proptosis domain had the lowest (average score of 5.0/6.0). The intraclass correlation coefficient was 0.64 (95% CI: 0.52 to 0.74), reflecting moderate inter-rater reliability. The responses exhibited a high reading-level complexity, representing the comprehension levels of a college or graduate education. Conclusions: This study demonstrates the potential of ChatGPT 4.0 to provide accurate information in the field of ophthalmology, specifically orbital and oculofacial disease. However, challenges remain in ensuring accurate and comprehensive responses across all disease domains. Future improvements should focus on refining the model’s correctness and eventually expanding the scope to visual data interpretation. Our results highlight the vast potential for artificial intelligence in educational and clinical ophthalmology contexts.

Evaluating accuracy and reproducibility of ChatGPT responses to patient-based questions in Ophthalmology: An observational study

“…[9] It was also shown that ChatGPT performs accurately when responding to questions about orbital and oculofacial disorders, with an average appropriateness score of 5.3/6.0 ("mostly appropriate" to "completely appropriate"). [10] Our study found that ChatGPT scored best in the infectious disorders section (73.3%), and poorest in the retinal disorders section (50%). Antaki et al showed that the legacy model performed best in general medicine (75%), fundamentals (60%), and cornea (60%), but as well in glaucoma, (37.5%), and pediatrics and strabismus (42.5%), and neuro-ophthalmology (25%), [7] which is in contradiction to Madadi et al findings that showed the potential to diagnose cases related to neuroophthalmology with comparable accuracy to certified neuroophthalmologist, with estimated accuracy of 59% and 82% for ChatGPT 3.5 and ChatGPT 4.0, respectively.…”

Re: “Orbital and Oculofacial Diseases and Artificial Intelligence: Evaluating the Accuracy and Readability of ChatGPT”

“…Sarcoidosis is an idiopathic multisystem disorder wherein ocular involvement is seen in approximately 25% of the cases with anterior uveitis being the most common presentation. 1 It can affect any part of the eye including the orbit or lacrimal system with dacryoadenitis being the most common extraocular involvement. The reported incidence of cutaneous involvement in sarcoidosis is 12% to 27% and the incidence of eyelid involvement is even lower.…”