Assessment of Pathology Domain-Specific Knowledge of ChatGPT and Comparison to Human Performance

Abstract

Context.— Artificial intelligence algorithms hold the potential to fundamentally change many aspects of society. Application of these tools, including the publicly available ChatGPT, has demonstrated impressive domain-specific knowledge in many areas, including medicine. Objectives.— To understand the level of pathology domain-specific knowledge for ChatGPT using different underlying large language models, GPT-3.5 and the updated GPT-4. Design.— An international group of pathologists (n = 15) was recruited to generate pathology-specific questions at a similar level to those that could be seen on licensing (board) examinations. The questions (n = 15) were answered by GPT-3.5, GPT-4, and a staff pathologist that recently passed their Canadian pathology licensing exams. Participants were instructed to score answers on a 5-point scale and to predict which answer was written by ChatGPT. Results.— GPT-3.5 performed at a similar level to the staff pathologist, while GPT-4 outperformed both. The overall score for both GPT-3.5 and GPT-4 was within the range of meeting expectations for a trainee writing licensing examinations. In all but one question, the reviewers were able to correctly identify the answers generated by GPT-3.5. Conclusions.— By demonstrating the ability of ChatGPT to answer pathology-specific questions at a level similar to (GPT-3.5) or exceeding (GPT-4) a trained pathologist, this study highlights the potential of large language models to be transformative in this space. In the future, more advanced iterations of these algorithms with increased domain-specific knowledge may have the potential to assist pathologists and enhance pathology resident training.

ChatGPT Versus National Eligibility cum Entrance Test for Postgraduate (NEET PG)

“…It was also noted that in our study, excessive verbosity was shown to be frequently linked to evasive and generic answers, leading to incorrect responses. In contrast to a recent study that showed a significant improvement in GPT-4's performance when compared to both its predecessor and human volunteers, our investigation revealed that GPT-4 performed worse than humans [17].…”

Enhancing clinical decision‐making: Optimizing ChatGPT's performance in hypertension care

“…It is important to note the shift from GPT-3.5 to GPT-4 marks a substantial improvement in the functionality and efficiency of large language models in various medical subspecialties. [18][19][20][21][22] Our findings show that GPT-4′s accuracy in responding to questions related to hyperten-sion has increased by 20%, achieving a 77% success rate. 10 Despite the need for a subscription to access GPT-4, its enhanced performance strongly supports its recommended use.…”

Applications of Large Language Models in Pathology

“…ChatGPT can provide an interactive educational experience, summarize important concepts [16], and give instant access to information [17]. It can tailor the educational experience according to an individual's needs by providing personalized responses to questions with immediate feedback [18].…”

“…In a paper using ChatGPT to answer pathology-related questions, such as "explain why transfusion-related diseases are avoidable", it was able to provide credible responses in most cases, achieving an overall median rating of 4.08 out 5 [21]. In a recently published study, the domain-specific knowledge of ChatGPT 3.5 in pathology was assessed to be the same level as a staff pathologist, while ChatGPT 4 exceeded that of a trained pathologist [17]. There may be instances where ChatGPT may not have information about certain topics in its knowledge base.…”