neuroGPT-X: Towards an Accountable Expert Opinion Tool for Vestibular Schwannoma

Abstract

Background The global launch of ChatGPT on November 30, 2022 has sparked widespread public interest in large language models, and interest from the medical community is growing. Indeed, recent preprints on medRxiv have examined ChatGPT and GPT-3 in the context of standardized exams, including the United States Medical Licensing Examination and the Ophthalmic Knowledge Assessment Program. These studies demonstrate modest performance relative to national averages. In this work, we enhanced OpenAI's GPT-3 model through zero-shot learning, anticipating it outperforms experienced neurosurgeons. Furthermore, we aimed to address accountability by including in-text citations and references to the responses provided by GPT-3. Methods The analysis comprised the (i) development of a structured dataset via web scraping, (ii) development of a chat-based platform called neuroGPT-X, (iii) recruitment of experienced neurosurgeons across international centers to create questions and evaluate GPT and surgeon responses regarding the comprehensive management of vestibular schwannoma, and (iv) analysis of evaluation results. The survey was split into blinded and unblinded phases. In the blinded phase, a neurosurgeon with 30+ years of experience managing patients with vestibular schwannoma curated 15 questions regarding common clinical and surgical contexts of vestibular schwannoma. Then, four neurosurgeons, ChatGPT (January 30, 2023 model, aka naive GPT), and a context-enriched GPT model independently provided their responses. Three experienced neurosurgeons blindly evaluated the responses across accuracy, coherence, relevance, thoroughness, speed, and overall rating. All seven neurosurgeons were unblinded to all responses and provided their thoughts on the potential of expert large language models in the clinical setting. Findings Both the naive and content-enriched GPT models provide faster responses to the standardized question set (p<0.01) than expert neurosurgeon respondents. Moreover, the generated responses for both models were consistently non-inferior in accuracy, coherence, relevance, thoroughness, and overall performance, and often demonstrated superiority when compared to expert responses. Importantly, we enriched the performance of GPT with relevant scientific literature without significantly affecting speed (p>0.999) or performance across all domains (p>0.999). Further, we develop neuroGPT-X, a chat-based platform designed to improve clinical decision-making and mitigate limitations of human memory. neuroGPT-X incorporates features such as in-text citations and references to enable accurate, relevant, and reliable information in real-time. Interpretation A context-enriched GPT model was non-inferior and often outperformed experienced neurosurgeons in generating specialist-level responses to a complex neurosurgical problem. We show that the context-enrichment of large language models is well-suited to transform clinical practice by providing subspecialty-level answers to clinical questions in an accountable manner.

The Expanding Role of ChatGPT (Chat-Generative Pre-Trained Transformer) in Neurosurgery: A Systematic Review of Literature and Conceptual Framework

“…In the realm of scientific publications, researchers addressed AI-generated research paper fabrication and plagiarism, emphasizing the importance of detecting and preventing AI-generated plagiarism to uphold the integrity of scholarly works [2]. Moreover, ChatGPT was showcased as a valuable tool for enhancing medical case reporting and study data analysis for manuscript production [8,9,11,12]. The impact of ChatGPT on medical education was explored, with discussions centering on strategies for responsible implementation to enhance learning experiences, promote critical thinking, and address individual learning needs in healthcare education [10].…”

“…By providing information on treatment options, potential risks, and benefits, as well as postoperative care, it enhances patient satisfaction and engagement [25]. Notwithstanding these applications, furthermore, ChatGPT has been shown to aid in scientific production, facilitating text generation, with a number of scientific articles being produced by such means, as well as being an adjuvant in education [8][9][10][11][12].…”

“…By generating outlines, identifying relevant literature, and offering suggestions for sentence structure and grammar, ChatGPT can streamline the writing process for researchers. This could significantly reduce the time and effort required to produce high-quality scientific manuscripts, although questions may arise regarding the potential loss of individual creativity [6][7][8][9][10]. This article aims to discuss, therefore, the potential applications of ChatGPT in different neurosurgery topics and its impact on the field.…”

Innovations in Medicine: Exploring ChatGPT’s Impact on Rare Disorder Management

“…For these reasons, Guo and coworkers created a trained version of ChatGPT, called neuroGPT-X, and evaluated it, comparing with naïve ChatGPT and leading neurosurgical experts worldwide. Despite its ability to support the neurosurgeons, the human expert's evaluation remains necessary, mainly to ensure the safety and reliability of chatbot responses [96].…”

Generative Artificial Intelligence Through ChatGPT and Other Large Language Models in Ophthalmology

“… 77 Therefore, LLMs exhibit a risk of perpetuating socioeconomic stereotypes and negative generalizations against minorities in ethnicity, religion, and gender. 78 , 79 Other risks to patient safety may arise when LLM applications are misused to spread misinformation or extract confidential patient information. For instance, they can craft unique variants of the same phishing lure and effectively bypass safety filters that detect possible scams based on identical text sequences.…”