Accuracy of ChatGPT on Medical Questions in the National Medical Licensing Examination in Japan: Evaluation Study

Abstract

Background ChatGPT (OpenAI) has gained considerable attention because of its natural and intuitive responses. ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers, as stated by OpenAI as a limitation. However, considering that ChatGPT is an interactive AI that has been trained to reduce the output of unethical sentences, the reliability of the training data is high and the usefulness of the output content is promising. Fortunately, in March 2023, a new version of ChatGPT, GPT-4, was released, which, according to internal evaluations, was expected to increase the likelihood of producing factual responses by 40% compared with its predecessor, GPT-3.5. The usefulness of this version of ChatGPT in English is widely appreciated. It is also increasingly being evaluated as a system for obtaining medical information in languages other than English. Although it does not reach a passing score on the national medical examination in Chinese, its accuracy is expected to gradually improve. Evaluation of ChatGPT with Japanese input is limited, although there have been reports on the accuracy of ChatGPT’s answers to clinical questions regarding the Japanese Society of Hypertension guidelines and on the performance of the National Nursing Examination. Objective The objective of this study is to evaluate whether ChatGPT can provide accurate diagnoses and medical knowledge for Japanese input. Methods Questions from the National Medical Licensing Examination (NMLE) in Japan, administered by the Japanese Ministry of Health, Labour and Welfare in 2022, were used. All 400 questions were included. Exclusion criteria were figures and tables that ChatGPT could not recognize; only text questions were extracted. We instructed GPT-3.5 and GPT-4 to input the Japanese questions as they were and to output the correct answers for each question. The output of ChatGPT was verified by 2 general practice physicians. In case of discrepancies, they were checked by another physician to make a final decision. The overall performance was evaluated by calculating the percentage of correct answers output by GPT-3.5 and GPT-4. Results Of the 400 questions, 292 were analyzed. Questions containing charts, which are not supported by ChatGPT, were excluded. The correct response rate for GPT-4 was 81.5% (237/292), which was significantly higher than the rate for GPT-3.5, 42.8% (125/292). Moreover, GPT-4 surpassed the passing standard (>72%) for the NMLE, indicating its potential as a diagnostic and therapeutic decision aid for physicians. Conclusions GPT-4 reached the passing standard for the NMLE in Japan, entered in Japanese, although it is limited to written questions. As the accelerated progress in the past few months has shown, the performance of the AI will improve as the large language model continues to learn more, and it may well become a decision support system for medical professionals by providing more accurate information.

Conformity of ChatGPT recommendations with the AUA/SUFU guideline on postprostatectomy urinary incontinence

“…6 Other studies have used more complex questions and clinical scenarios and reported accuracy rates ranging from 26.7% to 81.5% for the chatbot, depending on the specific methods employed or the version that was tested. 7,15,16 In common, the studies that compared the two versions of the chatbot have consistently shown superior performance for version 4. 7,15 In our study, open questions, with different levels of complexity, were posed to ChatGPT.…”

“…7,15,16 In common, the studies that compared the two versions of the chatbot have consistently shown superior performance for version 4. 7,15 In our study, open questions, with different levels of complexity, were posed to ChatGPT. We have not prompted the chatbot to base its answers on specific guidelines since we wanted to simulate "real world" scenarios, where a user, not necessarily an expert in F I G U R E 1 Performance of ChatGPT 3.5 and 4 in conceptual and case-based questions.…”

“…Several studies have assessed the reliability of ChatGPT, each using different methodologies and subsequently reporting varying levels of performance 5–11,15 . Studies using common questions of interest to patients and laypeople usually show high accuracy rates.…”

“…In a recent study evaluating the chatbot recommendations regarding breast cancer prevention and screening, an accuracy rate of 88% for ChatGPT 3.5 was reported 6 . Other studies have used more complex questions and clinical scenarios and reported accuracy rates ranging from 26.7% to 81.5% for the chatbot, depending on the specific methods employed or the version that was tested 7,15,16 . In common, the studies that compared the two versions of the chatbot have consistently shown superior performance for version 4 7,15 …”

“…Several studies have assessed the reliability of ChatGPT, each using different methodologies and subsequently reporting varying levels of performance. [5][6][7][8][9][10][11]15 Studies using common questions of interest to patients and laypeople usually show high accuracy rates. In a recent study evaluating the chatbot recommendations regarding breast cancer prevention and screening, an accuracy rate of 88% for ChatGPT 3.5 was reported.…”