ChatGPT Performs on the Chinese National Medical Licensing Examination

Abstract

INTRODUCTION: ChatGPT, a language model developed by OpenAI, uses a 175 billion parameter Transformer architecture for natural language processing tasks. This study aimed to compare the knowledge and interpretation ability of ChatGPT with those of medical students in China by administering the Chinese National Medical Licensing Examination (NMLE) to both ChatGPT and medical students. METHODS We evaluated the performance of ChatGPT in two years' worth of the NMLE, which consists of four units. At the same time, the exam results were compared to those of medical students who had studied for five years at medical colleges. RESULTS ChatGPT’s performance was lower than that of the medical students, and ChatGPT’s correct answer rate was related to the year in which the exam questions were released. CONCLUSION ChatGPT’s knowledge and interpretation ability for the NMLE were not yet comparable to those of medical students in China. It is probable that these abilities will improve through deep learning.

ChatGPT in Healthcare: A Taxonomy and Systematic Review

“…et al [74] raised concerns that ChatGPT, a language model, passing the exam indicates the flawness of the exam system 1 . Besides USMLE, ChatGPT was also tested on the Chinese National Medical Licensing Examination [112] and the AHA BLS / CLS Exams 2016 [32], on both of which ChatGPT failed to achieve passing scores.…”

A Survey of GPT-3 Family Large Language Models Including ChatGPT and GPT-4

“…The recent works explored GLLMs for a variety of clinical NLP tasks like question answering [117], [195], [317], [320], [322], [323], [326], [333], [335], [337], [338], [341], [342], text de-identification [318], dialogue summarization [319], [328], [330], named entity recognition [149], [321], relation extraction [321], text classification [138], [321], [326], [335], semantic similarity [321], [326], text simplification [324], [327], [343], relation classification [149], [326], text summarization [325], [331], natural language inference [137], [138], [326], [335], word sense disambiguation [329], biomedical evidence extraction [329], coreference resolution [329], medical status extraction [329], medical attribute extraction [329], synonym generation [334], clinical decision support [336...…”

“…However, the accuracy rate of ChatGPT is still less than the accuracy rate of physicians. Wang et al [333] evaluated the performance of the ChatGPT model in answering medical questions in the Chinese language. Here, ChatGPT is prompted with questions in both English and Chinese to avoid language barriers.…”

Evaluation of ChatGPT's performance in providing treatment recommendations for pediatric diseases

“…1 In the medical field, ChatGPT has passed the United States Medical Licensing Examination but failed the Chinese equivalent. 2,3 Growing interest surrounds Chat-GPT's potential to assist physicians in clinical medicine. 4,5 An expanding body of research continues to evaluate ChatGPT's performance in various medical applications.…”

“…ChatGPT, a general‐purpose artificial intelligence language model with over 100 million users, has been widely applied to various fields 1 . In the medical field, ChatGPT has passed the United States Medical Licensing Examination but failed the Chinese equivalent 2,3 . Growing interest surrounds ChatGPT's potential to assist physicians in clinical medicine 4,5 …”