ChatGPT in medical school: how successful is AI in progress testing?

Abstract

Background As generative artificial intelligence (AI), ChatGPT provides easy access to a wide range of information, including factual knowledge in the field of medicine. Given that knowledge acquisition is a basic determinant of physicians’ performance, teaching and testing different levels of medical knowledge is a central task of medical schools. To measure the factual knowledge level of the ChatGPT responses, we compared the performance of ChatGPT with that of medical students in a progress test. Methods A total of 400 multiple-choice questions (MCQs) from the progress test in German-speaking countries were entered into ChatGPT’s user interface to obtain the percentage of correctly answered questions. We calculated the correlations of the correctness of ChatGPT responses with behavior in terms of response time, word count, and difficulty of a progress test question. Results Of the 395 responses evaluated, 65.5% of the progress test questions answered by ChatGPT were correct. On average, ChatGPT required 22.8 s (SD 17.5) for a complete response, containing 36.2 (SD 28.1) words. There was no correlation between the time used and word count with the accuracy of the ChatGPT response (correlation coefficient for time rho = −0.08, 95% CI [−0.18, 0.02], t(393) = −1.55, p = 0.121; for word count rho = −0.03, 95% CI [−0.13, 0.07], t(393) = −0.54, p = 0.592). There was a significant correlation between the difficulty index of the MCQs and the accuracy of the ChatGPT response (correlation coefficient for difficulty: rho = 0.16, 95% CI [0.06, 0.25], t(393) = 3.19, p = 0.002). Conclusion ChatGPT was able to correctly answer two-thirds of all MCQs at the German state licensing exam level in Progress Test Medicine and outperformed almost all medical students in years 1–3. The ChatGPT answers can be compared with the performance of medical students in the second half of their studies.

Below average ChatGPT performance in medical microbiology exam compared to university students

“…Previous studies have evaluated the efficacy of AI-based models in addressing MCQs across diverse healthcare disciplines, yielding varying outcomes (Newton and Xiromeriti, 2023). These assessments encompassed the utilization of ChatGPT in scenarios such as United States Medical Licensing Examination (USMLE) exam, as well as the performance within fields such as parasitology and ophthalmology, among others in various languages (e.g., Japanese, Chinese, German, Spanish) (Antaki et al, 2023;Carrasco et al, 2023;Friederichs et al, 2023;Huh, 2023;Kung et al, 2023;Takagi et al, 2023;Xiao et al, 2023). A recent scoping review examined the influence of the ChatGPT model on examination outcomes, including instances where ChatGPT demonstrated superior performance compared to students, albeit in a minority of cases (Newton and Xiromeriti, 2023).…”

Is ChatGPT detrimental to innovation? A field experiment among university students

“…Just as artificial intelligence text generators (AITGs) started to make headlines, many feared that students would use the chatbot to do assignments, write theses, and hence change -or harm -their learning trajectory (for a discussion, see (32)(33)(34)(35). This worry was deepened after several studies pointed to a reasonable performance by ChatGPT in handling academic tasks, such as taking exams and generating academic abstracts (36). A university campus, therefore, seemed to be an ideal venue for a field experiment on the probable effects of ChatGPT on innovation.…”

An overview of the capabilities of ChatGPT for medical writing and its implications for academic integrity

“…Sedaghat (2023) reported that the performance of ChatGPT to pass a medical exam is at an acceptable level and it has capability to change medical practices. A study by Friederichs et al (2023) reported that ChatGPT outperformed almost all medical students in first, second, and third years in the German State exam in Progress Test Medicine. Only students from fourth years and beyond were able to achieve comparable, but not higher scores.…”

“…Medical professionals have a responsibility to provide patients with evidence-based care, but it becomes challenging to keep up with the pace of information generation. Ultimately, this can affect the quality of patient care (Azam & Naeem, 2022;Friederichs et al, 2023). AI tools could potentially help with this.…”