Performance of ChatGPT on Chinese Master’s Degree Entrance Examination in Clinical Medicine

Li, Ke-Cheng; Bu, Zhi-Jun; Shahjalal, Md.; He, Bai-Xiang; Zhuang, Zi-Fan; Li, Chen; Liu, Jian-Ping; Wang, Bin; Liu, Zhao-Lan

doi:10.1371/journal.pone.0301702

Cited by 2 publications

(1 citation statement)

References 21 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our research con rms the advantages of Ernie Bot 4.0 and GPT-4o in handling Chinese medical questions, providing a basis for their application in Chinese medical education. Li et al (2024) discovered that ChatGPT performed exceptionally in the entrance examination for a Master's degree in clinical medicine, especially in the medical humanities subjects [17]. This study further assesses the performance of Ernie Bot 4.0, GPT-4.0, and GPT-4o across different subjects and question types, revealing their unique strengths and areas for improvement in Chinese medical exams.…”

Section: Discussionmentioning

confidence: 76%

Comparative Study of GPT-4.0, ERNIE Bot 4.0, and GPT-4o in the 2023 Chinese Medical Licensing Examination

Lian

2024

Preprint

View full text Add to dashboard Cite

This study systematically evaluates the performance of three advanced large language models (LLMs)—GPT-4.0, ERNIE Bot 4.0, and GPT-4o—in the 2023 Chinese Medical Licensing Examination. Employing a dataset of 600 standardized questions, we analyzed the accuracy of each model in answering questions from three comprehensive sections: Basic Medical Comprehensive, Clinical Medical Comprehensive, and Humanities and Preventive Medicine Comprehensive. Our results demonstrate that both ERNIE Bot 4.0 and GPT-4o significantly outperformed GPT-4.0, achieving accuracies above the national pass mark. The study further examines the strengths and limitations of each model, providing insights into their applicability in medical education and potential areas for future improvement. These findings underscore the promise and challenges of deploying LLMs in multilingual and multimodal educational contexts, suggesting a pathway towards integrating AI into medical training and assessment practices.

show abstract

Section: Discussionmentioning

confidence: 76%

Comparative Study of GPT-4.0, ERNIE Bot 4.0, and GPT-4o in the 2023 Chinese Medical Licensing Examination

Lian

2024

Preprint

View full text Add to dashboard Cite

show abstract

Assessing ChatGPT-4's Proficiency in English College Entrance Examinations Using Web Raschonline: A Comparative Study (Preprint)

Chow,

Yeh,

Chou

2024

Preprint

View full text Add to dashboard Cite

BACKGROUND ChatGPT, developed by OpenAI, is a state-of-the-art large language model that has demonstrated exceptional performance in various specialized applications. Despite the growing popularity and efficacy of artificial intelligence (AI), there is a lack of comprehensive studies assessing ChatGPT's competence, particularly ChatGPT-4V for images and text, on English tests for college entrance examinations using Rasch analysis. OBJECTIVE This study aims to (1) showcase the utility of the Rasch analysis website, RaschOnline, and (2) determine the grade achieved by ChatGPT-4V by evaluating its ability to handle questions involving both images and text, in comparison to a typical sample METHODS ChatGPT's capability was evaluated using 46 multiple-choice questions (MCQs) from the English tests conducted for Taiwan's college entrance examinations in 2024. Using a Rasch model, 300 simulated students with normal distributions were generated to compare with ChatGPT's responses. RaschOnline was used to create six visual presentations: item-difficulty plot, differential item functioning (DIF), item characteristic curve (ICC), performance plot, Wright map, and KIDMAP, to address the research objectives. RESULTS The findings revealed the following: (1) ChatGPT-4V's capacity was shown on the Wright Map, with 4.05 logits and a correct response rate of 0.98, resulting from the item 35 incorrectly answered during the visual identification of a comprehensive reading; (2) Item 35 was the most difficult (correct response rate=22%=67/301, with 1.59 logits on the item-difficulty plot); (3)An aberrant response with Zscore=-3.42 (<2.0, P<.05) on item 35 for ChatGPT-4V was shown on KIDMAP; (4) No DIF or capacity differences were observed between the two groups (P>.05 and P=.454, respectively), but significant in strata; (5) ChatGPT-4V uniquely classified as Grade A, standing out at the top on the performance plot when compared to other counterparts; (6) Item 35 displayed an good fit to the Rasch model with the infit mean square error (MNSQ) of 0.99. CONCLUSIONS Using RaschOnline, this study provides evidence that ChatGPT-4V possesses the ability to achieve a grade A when compared to a typical sample. After discussing the correct answer to item 35 with ChatGPT-4V, a 100% correct response rate was then achieved by either ChatGPT-4V or ChatGPT 4.0. ChatGPT-4V's excellent proficiency in answering MCQs with images and text from the English tests conducted in 2024 for Taiwan's college entrance examinations was evident.

show abstract

Performance of ChatGPT on Chinese Master’s Degree Entrance Examination in Clinical Medicine

Cited by 2 publications

References 21 publications

Comparative Study of GPT-4.0, ERNIE Bot 4.0, and GPT-4o in the 2023 Chinese Medical Licensing Examination

Comparative Study of GPT-4.0, ERNIE Bot 4.0, and GPT-4o in the 2023 Chinese Medical Licensing Examination

Assessing ChatGPT-4's Proficiency in English College Entrance Examinations Using Web Raschonline: A Comparative Study (Preprint)

Contact Info

Product

Resources

About