Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology

Abstract

BackgroundArtificial intelligence (AI) is evolving for healthcare services. Higher cognitive thinking in AI refers to the ability of the system to perform advanced cognitive processes, such as problem-solving, decision-making, reasoning, and perception. This type of thinking goes beyond simple data processing and involves the ability to understand and manipulate abstract concepts, interpret, and use information in a contextually relevant way, and generate new insights based on past experiences and accumulated knowledge. Natural language processing models like ChatGPT is a conversational program that can interact with humans to provide answers to queries. ObjectiveWe aimed to ascertain the capability of ChatGPT in solving higher-order reasoning in the subject of pathology. MethodsThis cross-sectional study was conducted on the internet using an AI-based chat program that provides free service for research purposes. The current version of ChatGPT (January 30 version) was used to converse with a total of 100 higher-order reasoning queries. These questions were randomly selected from the question bank of the institution and categorized according to different systems. The responses to each question were collected and stored for further analysis. The responses were evaluated by three expert pathologists on a zero to five scale and categorized into the structure of the observed learning outcome (SOLO) taxonomy categories. The score was compared by a one-sample median test with hypothetical values to find its accuracy. ResultA total of 100 higher-order reasoning questions were solved by the program in an average of 45.31±7.14 seconds for an answer. The overall median score was 4.08 (Q1-Q3: 4-4.33) which was below the hypothetical maximum value of five (one-test median test p <0.0001) and similar to four (one-test median test p = 0.14). The majority (86%) of the responses were in the "relational" category in the SOLO taxonomy. There was no difference in the scores of the responses for questions asked from various organ systems in the subject of Pathology (Kruskal Wallis p = 0.55). The scores rated by three pathologists had an excellent level of interrater reliability (ICC = 0.975 [95% CI: 0.965-0.983]; F = 40.26; p < 0.0001). ConclusionThe capability of ChatGPT to solve higher-order reasoning questions in pathology had a relational level of accuracy. Hence, the text output had connections among its parts to provide a meaningful response. The answers from the program can score approximately 80%. Hence, academicians or students can get help from the program for solving reasoning-type questions also. As the program is evolving, further studies are needed to find its accuracy level in any further versions.

Assessing the Capability of ChatGPT in Answering First- and Second-Order Knowledge Questions on Microbiology as per Competency-Based Medical Education Curriculum

“…Based on our findings, ChatGPT is capable of providing answers to both first- and second-order knowledge questions with approximately 80% accuracy. This corroborates the findings of Sinha et al [ 4 ]. There was no difference in the application's capability to solve both types of questions.…”

“…In recent years, natural language processing (NLP) and machine learning (ML) techniques have revolutionized the field of question-answering systems [ 1 ]. ChatGPT is one such artificial intelligence (AI) language model that has been trained to understand the nuances of natural language and provide accurate answers to a wide range of questions [ 2 , 3 , 4 ]. While ChatGPT has shown promising results in solving problems in various fields, its ability to accurately answer first- and second-order knowledge questions in microbiology has not been extensively evaluated.…”

Evaluating ChatGPT's Ability to Solve Higher-Order Questions on the Competency-Based Medical Education Curriculum in Medical Biochemistry

“…Numerous past investigations have evaluated the potential of ChatGPT for medical education purposes. The observation is similar to a recent study conducted by Sinha et al demonstrating the utility of ChatGPT for solving higher-order problems in pathology [12]. The study conducted by Das et al had a similar outcome where ChatGPT was shown to answer first and second-order questions of microbiology with 80% accuracy, thereby corroborating with our study [13].…”

ChatGPT: A Brief Narrative Review

“…ChatGPT was developed using data from The Common Crawl database, which includes copyrighted content from publishers, as well as works by individual authors and scholars [42]. There is a chance that the advice offered if ChatGPT is utilized to provide legal or financial advice could not be correct or current.…”