Performance of Large Language Models on a Neurology Board–Style Examination

Abstract

ImportanceRecent advancements in large language models (LLMs) have shown potential in a wide array of applications, including health care. While LLMs showed heterogeneous results across specialized medical board examinations, the performance of these models in neurology board examinations remains unexplored.ObjectiveTo assess the performance of LLMs on neurology board–style examinations.Design, Setting, and ParticipantsThis cross-sectional study was conducted between May 17 and May 31, 2023. The evaluation utilized a question bank approved by the American Board of Psychiatry and Neurology and was validated with a small question cohort by the European Board for Neurology. All questions were categorized into lower-order (recall, understanding) and higher-order (apply, analyze, synthesize) questions based on the Bloom taxonomy for learning and assessment. Performance by LLM ChatGPT versions 3.5 (LLM 1) and 4 (LLM 2) was assessed in relation to overall scores, question type, and topics, along with the confidence level and reproducibility of answers.Main Outcomes and MeasuresOverall percentage scores of 2 LLMs.ResultsLLM 2 significantly outperformed LLM 1 by correctly answering 1662 of 1956 questions (85.0%) vs 1306 questions (66.8%) for LLM 1. Notably, LLM 2’s performance was greater than the mean human score of 73.8%, effectively achieving near-passing and passing grades in the neurology board examination. LLM 2 outperformed human users in behavioral, cognitive, and psychological–related questions and demonstrated superior performance to LLM 1 in 6 categories. Both LLMs performed better on lower-order than higher-order questions, with LLM 2 excelling in both lower-order and higher-order questions. Both models consistently used confident language, even when providing incorrect answers. Reproducible answers of both LLMs were associated with a higher percentage of correct answers than inconsistent answers.Conclusions and RelevanceDespite the absence of neurology-specific training, LLM 2 demonstrated commendable performance, whereas LLM 1 performed slightly below the human average. While higher-order cognitive tasks were more challenging for both models, LLM 2’s results were equivalent to passing grades in specialized neurology examinations. These findings suggest that LLMs could have significant applications in clinical neurology and health care with further refinements.

Replies to Queries in Gynecologic Oncology by Bard, Bing and the Google Assistant

“…There are multiple ways that can provide users with greater confidence about the accuracy of AI search results. The use of a "custom attribution engine" as announced by Adobe would allow users to verify AI findings through source citation [14]. This type of approach should allow users to interpret narrative results in terms of source information and determine if there is any distortion in the AI narrative.…”

Evaluating the Cognitive Levels of Generative AI via Bloom’s Taxonomy: A Cross-sectional Study (Preprint)

“…The study found that ChatGPT models performed notably poorer on questions involving concepts of redefinition or invention [27]. Emerging evidence suggests that AI may exhibit its own cognitive process, as indirectly indicated by a trend of improved performance on questions at the lower levels of Bloom's taxonomy, particularly in disciplines such as neurology, radiology, physiology, microbiology, and biochemistry [28][29][30][31][32]. In these investigations, the majority of questions assessed originated from internal materials or were inaccessible to ChatGPT models during the study periods.…”

Evaluating the strengths and weaknesses of large language models in answering neurophysiology questions

“…Previous publications evaluating LLMs across various disciplines have covered fields such as, gastroenterology 7 , pathology 8 , neurology 9 , physiology 6 , 10 , and solving case vignettes in physiology 11 . In a cross-sectional study, the performance of LLMs on neurology board–style examinations were assessed using a question bank approved by the American Board of Psychiatry and Neurology.…”

“…In a cross-sectional study, the performance of LLMs on neurology board–style examinations were assessed using a question bank approved by the American Board of Psychiatry and Neurology. The questions were categorized into lower-order and higher-order based on the Bloom taxonomy for learning and assessment 9 . To the best of our knowledge there was no study specifically on evaluating LLMs in the field of neurophysiology.…”