Evaluating ChatGPT's Ability to Solve Higher-Order Questions on the Competency-Based Medical Education Curriculum in Medical Biochemistry

Abstract

BackgroundHealthcare-related artificial intelligence (AI) is developing. The capacity of the system to carry out sophisticated cognitive processes, such as problem-solving, decision-making, reasoning, and perceiving, is referred to as higher cognitive thinking in AI. This kind of thinking requires more than just processing facts; it also entails comprehending and working with abstract ideas, evaluating and applying data relevant to the context, and producing new insights based on prior learning and experience. ChatGPT is an artificial intelligence-based conversational software that can engage with people to answer questions and uses natural language processing models. The platform has created a worldwide buzz and keeps setting an ongoing trend in solving many complex problems in various dimensions. Nevertheless, ChatGPT's capacity to correctly respond to queries requiring higher-level thinking in medical biochemistry has not yet been investigated. So, this research aimed to evaluate ChatGPT's aptitude for responding to higher-order questions on medical biochemistry. ObjectiveIn this study, our objective was to determine whether ChatGPT can address higher-order problems related to medical biochemistry. MethodsThis cross-sectional study was done online by conversing with the current version of ChatGPT (14 March 2023, which is presently free for registered users). It was presented with 200 medical biochemistry reasoning questions that require higher-order thinking. These questions were randomly picked from the institution's question bank and classified according to the Competency-Based Medical Education (CBME) curriculum's competency modules. The responses were collected and archived for subsequent research. Two expert biochemistry academicians examined the replies on a zero to five scale. The score's accuracy was determined by a one-sample Wilcoxon signed rank test using hypothetical values. ResultThe AI software answered 200 questions requiring higher-order thinking with a median score of 4.0 (Q1=3.50, Q3=4.50). Using a single sample Wilcoxon signed rank test, the result was less than the hypothetical maximum of five (p=0.001) and comparable to four (p=0.16). There was no difference in the replies to questions from different CBME modules in medical biochemistry (Kruskal-Wallis p=0.39). The inter-rater reliability of the scores scored by two biochemistry faculty members was outstanding (ICC=0.926 (95% CI: 0.814-0.971); F=19; p=0.001) ConclusionThe results of this research indicate that ChatGPT has the potential to be a successful tool for answering questions requiring higher-order thinking in medical biochemistry, with a median score of four out of five. However, continuous training and development with data of recent advances are essential to improve performance and make it functional for the ever-growing field of academic medical usage.

Efficacy of ChatGPT in solving attitude, ethics, and communication case scenario used for competency-based medical education in India: A case study

“…[ 8 ] Previous studies have proven its capacity to solve cases and complex questions in various domains. [ 9 10 ] Chatbots can help in both generating the cases and also provide solution to the cases. For example, AI-based chatbots could be developed that can provide students with immediate feedback on their responses to case scenarios based on AETCOM competencies.…”

Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology

“…Recent studies conducted by medical professionals in India have shown that the ChatGPT is a reliable tool for solving problems that require higher-level thinking, interpretation, analysis, evaluation, or formulation of opinions and predictions based on evidence in pathology, biochemistry, and microbiology [8][9][10]. In addition, ChatGPT was proficient in answering straightforward queries seeking factual information in microbiology [8].…”

Embracing the future: Integrating ChatGPT into China’s nursing education system

“…, ChatGPT software has been successfully used to pass the ( ) Basic Life Support ( ) test [ 27 ] and the Plastic Surgery In-Training Exam (PSITE) [ 28 ]. ChatGPT has produced good outcomes in surgical education and training while resolving more complex medical biochemistry-related issues [ 30 ]. These findings demonstrate that ChatGPT’s knowledge is suitable for teaching foundational courses.…”