Performance of ChatGPT on the Situational Judgement Test—A Professional Dilemmas–Based Examination for Doctors in the United Kingdom (Preprint)

Abstract

UNSTRUCTURED ChatGPT is a language model which has performed well on professional exams in the fields of medicine, law and business. We evaluated the performance of ChatGPT on the Situational Judgement Test (SJT); a national exam taken by all final-year medical students in the United Kingdom (UK). The exam is designed to assess attributes such as communication, team-working, patient safety, prioritisation skills, professionalism and ethics. It differs from other medical exams, such as the United States Medical Licensing Exam (USMLE), as it relies less on memorisation and factual recall. Overall, ChatGPT’s performance was impressive scoring 76% on the SJT but scoring full marks on only a minority of the questions (9%) which may reflect possible flaws in ChatGPT’s situational judgement and/or inconsistencies in the reasoning across questions in the exam itself. ChatGPT performed consistently across the four outlined domains in Good Medical Practice for Doctors. Further research is needed to understand the potential applications of large language models, such as ChatGPT, in medical education for the purpose of standardising questions and providing consistent rationales for examinations involving professionalism and ethics.