Performance of an Upgraded Artificial Intelligence Chatbot for Ophthalmic Knowledge Assessment

Abstract

This cross-sectional study assesses the accuracy of answers generated by an updated version of a popular chatbot to board certification examination preparation questions.

Advances in Artificial Intelligence Chatbot Technology in Ophthalmology—Reply

“…Lin and colleagues make the argument that since this time, the accuracy of ChatGPT has increased. We completely agree and note the findings of our recent investigation published in JAMA Ophthalmology . Here, we found that ChatGPT-4 correctly answered 105 of 125 questions of the same question bank in March 2023 (84%), remarkably outperforming the previous model of this chatbot …”

Exploring the Test-Taking Capabilities of Chatbots—From Surgeon to Sommelier

“…In March 2023, OpenAI unveiled the latest iteration of the chatbot, which is built on the generative pretrained transformer (GPT)–4 architecture. In JAMA Ophthalmology , Mihalache et al examine the performance of the updated version of this chatbot on a question bank designed to help candidates prepare for ophthalmology board examinations, building on previous work evaluating the older GPT-3.5 architecture …”

“…Mihalache et al expand on this idea by evaluating performance of the updated version of this chatbot on the OphthoQuestions bank, which contains questions relevant to the Ophthalmic Knowledge Assessment Program and Written Qualifying Examination. Their earlier study using GPT-3 reported that 73 of 125 text-based multiple-choice questions (58%) were answered correctly, which is fairly moderate performance considering that 25% would be answered correctly on random chance alone.…”

“…It is equally important to take a cautious approach to the perceived benefits of these capabilities. Mihalache et al comment generally on the possible utility of chatbots as education tools, but like most novel AI technologies, it is essential to consider the specifics of its implementation. While a chatbot scoring 84% is impressive, it is still likely to be overlooked by students next to the official answer guide provided by OphthoQuestions.…”

“…It appears these safety concerns are also a focus for ChatGPT’s developers, with GPT-4 reportedly achieving a 29% improvement in responding to sensitive requests such as medical advice in accordance with policies . The study by Mihalache et al contributes to the ongoing debate surrounding the potential for large language models within medical education and beyond, but it is necessary to remain vigilant to ensure these tools are implemented in a way that is robust, reliable, safe, and fair.…”