Performance of ChatGPT, GPT-4, and Google Bard on a Neurosurgery Oral Boards Preparation Question Bank

Abstract

BACKGROUND AND OBJECTIVES: General large language models (LLMs), such as ChatGPT (GPT-3.5), have demonstrated the capability to pass multiple-choice medical board examinations. However, comparative accuracy of different LLMs and LLM performance on assessments of predominantly higher-order management questions is poorly understood. We aimed to assess the performance of 3 LLMs (GPT-3.5, GPT-4, and Google Bard) on a question bank designed specifically for neurosurgery oral boards examination preparation. METHODS: The 149-question Self-Assessment Neurosurgery Examination Indications Examination was used to query LLM accuracy. Questions were inputted in a single best answer, multiple-choice format. χ2, Fisher exact, and univariable logistic regression tests assessed differences in performance by question characteristics. RESULTS: On a question bank with predominantly higher-order questions (85.2%), ChatGPT (GPT-3.5) and GPT-4 answered 62.4% (95% CI: 54.1%-70.1%) and 82.6% (95% CI: 75.2%-88.1%) of questions correctly, respectively. By contrast, Bard scored 44.2% (66/149, 95% CI: 36.2%-52.6%). GPT-3.5 and GPT-4 demonstrated significantly higher scores than Bard (both P < .01), and GPT-4 outperformed GPT-3.5 (P = .023). Among 6 subspecialties, GPT-4 had significantly higher accuracy in the Spine category relative to GPT-3.5 and in 4 categories relative to Bard (all P < .01). Incorporation of higher-order problem solving was associated with lower question accuracy for GPT-3.5 (odds ratio [OR] = 0.80, P = .042) and Bard (OR = 0.76, P = .014), but not GPT-4 (OR = 0.86, P = .085). GPT-4's performance on imaging-related questions surpassed GPT-3.5's (68.6% vs 47.1%, P = .044) and was comparable with Bard's (68.6% vs 66.7%, P = 1.000). However, GPT-4 demonstrated significantly lower rates of “hallucination” on imaging-related questions than both GPT-3.5 (2.3% vs 57.1%, P < .001) and Bard (2.3% vs 27.3%, P = .002). Lack of question text description for questions predicted significantly higher odds of hallucination for GPT-3.5 (OR = 1.45, P = .012) and Bard (OR = 2.09, P < .001). CONCLUSION: On a question bank of predominantly higher-order management case scenarios for neurosurgery oral boards preparation, GPT-4 achieved a score of 82.6%, outperforming ChatGPT and Google Bard.

Demographic Representation in 3 Leading Artificial Intelligence Text-to-Image Generators

“…Researchers and developers must consider the impact of their models on society and work toward creating AI systems with appropriate training data sets, feedback mechanisms, and guardrails that accurately reflect our diverse world. Moreover, the active involvement of surgeons and other clinicians in the refinement of these models, such as serving in advisory roles, may help combat inaccuracies in the perception of the discipline of surgery by the general public …”

“…Moreover, the active involvement of surgeons and other clinicians in the refinement of these models, such as serving in advisory roles, may help combat inaccuracies in the perception of the discipline of surgery by the general public. 7,22…”

Will code one day run a code? Performance of language models on ACEM primary examinations and implications

“…Chat GPT has failed to achieve a passing grade in a gastroenterology board-like examination, 1 achieves close to or near pass levels in cardiology and radiology practice question, 2,3 while outstanding performance has been noted in Neurosurgery board finals. 4 It is unclear to what extent this applies to emergency medicine, a field that requires knowledge from a broad range of interdisciplinary sciences and disciplines.…”

Letter: The Urgency of Neurosurgical Leadership in the Era of Artificial Intelligence

“…As medical professionals, we have an ethical duty to ensure that AI technologies adhere to the highest safety and efficacy standards, regardless of the technical capabilities of such systems. Neurosurgical cases often have substantive clinical equipoise, and existing AI models can struggle with higher-order management questions or even produce “hallucinations.” 2,3 Accordingly, our recent work has elucidated that certain AI systems not infrequently confabulate fabricated or incorrect answer rationales, especially when lacking contextual data. 3 However, even if AI models did not hallucinate or struggle with higher-order reasoning, the onus would still rest on neurosurgeons to assess suitability for clinical application.…”

“…Neurosurgical cases often have substantive clinical equipoise, and existing AI models can struggle with higher-order management questions or even produce “hallucinations.” 2,3 Accordingly, our recent work has elucidated that certain AI systems not infrequently confabulate fabricated or incorrect answer rationales, especially when lacking contextual data. 3 However, even if AI models did not hallucinate or struggle with higher-order reasoning, the onus would still rest on neurosurgeons to assess suitability for clinical application. Superimposing human judgement remains critical in achieving optimal patient outcomes.…”