ChatGPT Knowledge Evaluation in Basic and Clinical Medical Sciences: Multiple Choice Question Examination-Based Performance

Abstract

The Chatbot Generative Pre-Trained Transformer (ChatGPT) has garnered great attention from the public, academicians and science communities. It responds with appropriate and articulate answers and explanations across various disciplines. For the use of ChatGPT in education, research and healthcare, different perspectives exist with some level of ambiguity around its acceptability and ideal uses. However, the literature is acutely lacking in establishing a link to assess the intellectual levels of ChatGPT in the medical sciences. Therefore, the present study aimed to investigate the knowledge level of ChatGPT in medical education both in basic and clinical medical sciences, multiple-choice question (MCQs) examination-based performance and its impact on the medical examination system. In this study, initially, a subject-wise question bank was established with a pool of multiple-choice questions (MCQs) from various medical textbooks and university examination pools. The research team members carefully reviewed the MCQ contents and ensured that the MCQs were relevant to the subject’s contents. Each question was scenario-based with four sub-stems and had a single correct answer. In this study, 100 MCQs in various disciplines, including basic medical sciences (50 MCQs) and clinical medical sciences (50 MCQs), were randomly selected from the MCQ bank. The MCQs were manually entered one by one, and a fresh ChatGPT session was started for each entry to avoid memory retention bias. The task was given to ChatGPT to assess the response and knowledge level of ChatGPT. The first response obtained was taken as the final response. Based on a pre-determined answer key, scoring was made on a scale of 0 to 1, with zero representing incorrect and one representing the correct answer. The results revealed that out of 100 MCQs in various disciplines of basic and clinical medical sciences, ChatGPT attempted all the MCQs and obtained 37/50 (74%) marks in basic medical sciences and 35/50 (70%) marks in clinical medical sciences, with an overall score of 72/100 (72%) in both basic and clinical medical sciences. It is concluded that ChatGPT obtained a satisfactory score in both basic and clinical medical sciences subjects and demonstrated a degree of understanding and explanation. This study’s findings suggest that ChatGPT may be able to assist medical students and faculty in medical education settings since it has potential as an innovation in the framework of medical sciences and education.

Medical knowledge of ChatGPT in public health, infectious diseases, COVID-19 pandemic, and vaccines: multiple choice questions examination based performance

“…The overall performance of ChatGPT was lesser than the medical students. More recently, Meo and Al-Masri ( 32 ) reported that ChatGPT obtained 72% marks, a reasonable score in basic and clinical medical sciences MCQs-based examination. However, in another study, Meo and Al-Khlaiwi et al ( 33 ) compared the Bard and ChatGPT knowledge in three different topics including endocrinology, diabetes, and diabetes technology through MCQ examination.…”

“…ChatGPT provides accurate information, however, it might generate incorrect answer responses. The most probable reason is that the information is an outdated misinterpretation of complex questions, lengthy questions, and formulas ( 32 , 33 ).…”

“…ChatGPT offers opportunities to enhance learning, research, assessment, and various spheres of educational settings. However, it’s essential to address ethical, quality assurance, and training considerations to maximize its benefits while mitigating potential risks ( 32 ).…”

The Potential Applications and Challenges of ChatGPT in the Medical Field

“…Meo’s team created a topic-based question bank containing multiple-choice questions from various medical textbooks and university exam banks to study the ChatGPT’s level of knowledge, multiple-choice questions in basic and clinical medicine and clinical medical education, education in terms of knowledge level, performance on multiple-choice exams and its impact on the medical examination system. 28 The results showed that ChatGPT obtained 37/50 (74%) scores in basic medicine and 35/50 (70%) scores in clinical medicine, with an overall mean score of 72/100 (72%). The authors concluded that the ChatGPT achieved satisfactory scores in both basic and clinical medicine subjects and demonstrated some degree of comprehension and interpretation, suggesting that the ChatGPT may have a tutoring role for teachers as well as students in a medical education setting.…”

Comparing the Performance of ChatGPT-4 and Medical Students on MCQs at Varied Levels of Bloom’s Taxonomy

“… 1 ChatGPT (Chat Generative Pre-trained Transformer), a state-of-the-art language model, has emerged as a potential tool in medical education. 2 This tool operates primarily through prompt interpretation and is capable of producing reasoned responses that are difficult to distinguish from human-produced language. 3 Its intrinsic transformer architecture also enables ChatGPT to be proficient in understanding natural language.…”