Performance of ChatGPT on the Situational Judgement Test—A Professional Dilemmas–Based Examination for Doctors in the United Kingdom

Abstract

Background ChatGPT is a large language model that has performed well on professional examinations in the fields of medicine, law, and business. However, it is unclear how ChatGPT would perform on an examination assessing professionalism and situational judgement for doctors. Objective We evaluated the performance of ChatGPT on the Situational Judgement Test (SJT): a national examination taken by all final-year medical students in the United Kingdom. This examination is designed to assess attributes such as communication, teamwork, patient safety, prioritization skills, professionalism, and ethics. Methods All questions from the UK Foundation Programme Office’s (UKFPO’s) 2023 SJT practice examination were inputted into ChatGPT. For each question, ChatGPT’s answers and rationales were recorded and assessed on the basis of the official UK Foundation Programme Office scoring template. Questions were categorized into domains of Good Medical Practice on the basis of the domains referenced in the rationales provided in the scoring sheet. Questions without clear domain links were screened by reviewers and assigned one or multiple domains. ChatGPT's overall performance, as well as its performance across the domains of Good Medical Practice, was evaluated. Results Overall, ChatGPT performed well, scoring 76% on the SJT but scoring full marks on only a few questions (9%), which may reflect possible flaws in ChatGPT’s situational judgement or inconsistencies in the reasoning across questions (or both) in the examination itself. ChatGPT demonstrated consistent performance across the 4 outlined domains in Good Medical Practice for doctors. Conclusions Further research is needed to understand the potential applications of large language models, such as ChatGPT, in medical education for standardizing questions and providing consistent rationales for examinations assessing professionalism and ethics.

Below average ChatGPT performance in medical microbiology exam compared to university students

“…A plausible explanation for this discrepancy can be related to different question styles and different exam settings. Taken together, this highlights the need to assess the performance of AI-based models in various disciplines, using different questions' format, and compared to human performance (Borchert et al, 2023;Chen et al, 2023;Deiana et al, 2023;Flores-Cohaila et al, 2023;Puladi et al, 2023). Finally, it is important to acknowledge the limitations inherent in this study.…”

The Performance of Large Language Models on Quantitative and Verbal Ability Tests: Initial Evidence and Implications for Unproctored High-stakes Testing

“…In many cases, the LLM responses scored at or above the 90 th percentile, compared to human test takers. Independent research teams have subsequently examined the performance of LLMs in relation to, for example, medical situational judgment tests (Borchert et al, 2023), medical knowledge and 'soft skills' assessments (Brin et al, 2023), and both single-stimulus and forced-choice personality assessments (Phillips & Robie, 2024). These studies' findings align with OpenAI's report: advanced LLMs out-score most human test takers on several types of tests.…”

“…Published research suggests that LLMs often achieve high scores on tests that comprise extensive verbal information. As noted above, advanced LLMs appear to achieve higher scores than the majority of humans on many knowledge-based assessments (OpenAI, 2023), certain situational judgement tests (Arctic Shores, 2023b;Borchert et al, 2023), and personality questionnaires (if prompted to; Arctic Shores, 2023a;Phillips & Robie, 2024). Further, Elyoseph et al (2023) found that ChatGPT (GPT-3.5) outperformed most humans on an emotional awareness assessment comprising items with text descriptions of situations that required participants to identify an emotional state.…”

Medical students’ patterns of using ChatGPT as a feedback tool and perceptions of ChatGPT in a Leadership and Communication course in Korea: a cross-sectional study

“…The second category involves the assessment of ChatGPT’s knowledge accuracy through testing, including examinations such as United States Medical Licensing Examination, the Situational Judgement Test, and subject tests in medical school [ 8 - 12 ]. In this study, the majority of the students who responded regarding the feedback provided by ChatGPT stated that it demonstrated a high degree of accuracy.…”