Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology

Tan, Ting Fang; Thirunavukarasu, Arun James; Campbell, J. Peter; Keane, Pearse A.; Pasquale, Louis R.; Abramoff, Michael D.; Kalpathy-Cramer, Jayashree; Lum, Flora; Kim, Judy E.; Baxter, Sally L.; Ting, Daniel Shu Wei

doi:10.1016/j.xops.2023.100394

Cited by 32 publications

(9 citation statements)

References 57 publications

(78 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The development of domain-specific models holds considerable promise in specialized areas like medicine. [2] LLM landscape has recently expanded with several models, not limited to ChatGPT, capable of processing images. Some of these models are noted for their superior performance.…”

Section: Discussionmentioning

confidence: 99%

“…Specifically, in ophthalmology, the efficacy of these models has been explored in various exams, including the Basic and Clinical Science Course (BCSC) Self-Assessment Program, OphthoQuestions question banks, and FRCOphth examinations. [2][3][4] Furthermore, it is important to note that LLMs are evolving and the performance of the model may improve dramatically in a short period of time.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Performance of GPT-4V(ision) in Ophthalmology: Use of Images in Clinical Questions

Tomita,

Nishida,

Kitaguchi

et al. 2024

Preprint

View full text Add to dashboard Cite

Background/aims: To compare the diagnostic accuracy of Generative Pre-trained Transformer with Vision (GPT)-4 and GPT-4 with Vision (GPT-4V) for clinical questions in ophthalmology. Methods: The questions were collected from the "Diagnosis This" section on the American Academy of Ophthalmology website. We tested 580 questions and presented GPT-4V with the same questions under two conditions: 1) multimodal model, incorporating both the question text and associated images, and 2) text-only model. We then compared the difference in accuracy between the two conditions using the chi-square test. The percentage of general correct answers was also collected from the website. Results: The GPT-4V model demonstrated higher accuracy with images (71.7%) than without images (66.7%, p<0.001). Both GPT-4 models showed higher accuracy than the general correct answers on the website [64.6 (95%CI, 62.9 to 66.3)]. Conclusions: The addition of information from images enhances the performance of GPT-4V in diagnosing clinical questions in ophthalmology. This suggests that integrating multimodal data could be crucial in developing more effective and reliable diagnostic tools in medical fields.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Performance of GPT-4V(ision) in Ophthalmology: Use of Images in Clinical Questions

Tomita,

Nishida,

Kitaguchi

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the remarkable performance of GPT-4 in ophthalmology examination questions suggests that LLMs may be able to provide useful input in clinical contexts, either to assist clinicians in their day-to-day work or with their education or preparation for examinations [ 3 , 13 , 14 , 27 ]. Further improvement in performance may be obtained by specific fine-tuning of models with high quality ophthalmological text data, requiring curation and deidentification [ 29 ]. GPT-4 may prove especially useful where access to ophthalmologists is limited: provision of advice, diagnosis, and management suggestions by a model with FRCOphth Part 2-level knowledge and reasoning ability is likely to be superior to non-specialist doctors and allied healthcare professionals working without support, as their exposure to and knowledge of eye care is limited [ 27 , 30 , 31 ].…”

Section: Discussionmentioning

confidence: 99%

Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study

Thirunavukarasu,

Mahmood,

Malem

et al. 2024

PLOS Digit Health

Self Cite

View full text Add to dashboard Cite

Large language models (LLMs) underlie remarkable recent advanced in natural language processing, and they are beginning to be applied in clinical contexts. We aimed to evaluate the clinical potential of state-of-the-art LLMs in ophthalmology using a more robust benchmark than raw examination scores. We trialled GPT-3.5 and GPT-4 on 347 ophthalmology questions before GPT-3.5, GPT-4, PaLM 2, LLaMA, expert ophthalmologists, and doctors in training were trialled on a mock examination of 87 questions. Performance was analysed with respect to question subject and type (first order recall and higher order reasoning). Masked ophthalmologists graded the accuracy, relevance, and overall preference of GPT-3.5 and GPT-4 responses to the same questions. The performance of GPT-4 (69%) was superior to GPT-3.5 (48%), LLaMA (32%), and PaLM 2 (56%). GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64–90%), ophthalmology trainees (median 59%, range 57–63%), and unspecialised junior doctors (median 43%, range 41–44%). Low agreement between LLMs and doctors reflected idiosyncratic differences in knowledge and reasoning with overall consistency across subjects and types (p>0.05). All ophthalmologists preferred GPT-4 responses over GPT-3.5 and rated the accuracy and relevance of GPT-4 as higher (p<0.05). LLMs are approaching expert-level knowledge and reasoning skills in ophthalmology. In view of the comparable or superior performance to trainee-grade ophthalmologists and unspecialised junior doctors, state-of-the-art LLMs such as GPT-4 may provide useful medical advice and assistance where access to expert ophthalmologists is limited. Clinical benchmarks provide useful assays of LLM capabilities in healthcare before clinical trials can be designed and conducted.

show abstract

“…The CoT prompting has achieved the state-of-the-art performances in arithmetic and symbolic reasoning 17 , 18 . The model is instructed in the CoT prompting to provide step-by-step reasoning in generating a final answer, which could be few-shot or zero-shot 19 . Utilizing structured prompting, which includes important components such as context, the expected behavior, and the format of the output, is another strategy for achieving optimal outcomes.…”

Section: Methodsmentioning

confidence: 99%

Evaluating the strengths and weaknesses of large language models in answering neurophysiology questions

Shojaee-Mend,

Mohebbati,

Amiri

et al. 2024

Sci Rep

View full text Add to dashboard Cite

Large language models (LLMs), like ChatGPT, Google’s Bard, and Anthropic’s Claude, showcase remarkable natural language processing capabilities. Evaluating their proficiency in specialized domains such as neurophysiology is crucial in understanding their utility in research, education, and clinical applications. This study aims to assess and compare the effectiveness of Large Language Models (LLMs) in answering neurophysiology questions in both English and Persian (Farsi) covering a range of topics and cognitive levels. Twenty questions covering four topics (general, sensory system, motor system, and integrative) and two cognitive levels (lower-order and higher-order) were posed to the LLMs. Physiologists scored the essay-style answers on a scale of 0–5 points. Statistical analysis compared the scores across different levels such as model, language, topic, and cognitive levels. Performing qualitative analysis identified reasoning gaps. In general, the models demonstrated good performance (mean score = 3.87/5), with no significant difference between language or cognitive levels. The performance was the strongest in the motor system (mean = 4.41) while the weakest was observed in integrative topics (mean = 3.35). Detailed qualitative analysis uncovered deficiencies in reasoning, discerning priorities, and knowledge integrating. This study offers valuable insights into LLMs’ capabilities and limitations in the field of neurophysiology. The models demonstrate proficiency in general questions but face challenges in advanced reasoning and knowledge integration. Targeted training could address gaps in knowledge and causal reasoning. As LLMs evolve, rigorous domain-specific assessments will be crucial for evaluating advancements in their performance.

show abstract

Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology

Cited by 32 publications

References 57 publications

Performance of GPT-4V(ision) in Ophthalmology: Use of Images in Clinical Questions

Performance of GPT-4V(ision) in Ophthalmology: Use of Images in Clinical Questions

Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study

Evaluating the strengths and weaknesses of large language models in answering neurophysiology questions

Contact Info

Product

Resources

About