Assessment of Emergency Medicine Residents’ Clinical Reasoning: Validation of a Script Concordance Test

Abstract

Introduction A primary aim of residency training is to develop competence in clinical reasoning. However, there are few instruments that can accurately, reliably, and efficiently assess residents’ clinical decision-making ability. This study aimed to externally validate the script concordance test in emergency medicine (SCT-EM), an assessment tool designed for this purpose. Methods Using established methodology for the SCT-EM, we compared EM residents’ performance on the SCT-EM to an expert panel of emergency physicians at three urban academic centers. We performed adjusted pairwise t-tests to compare differences between all residents and attending physicians, as well as among resident postgraduate year (PGY) levels. We tested correlation between SCT-EM and Accreditation Council for Graduate Medical Education Milestone scores using Pearson’s correlation coefficients. Inter-item covariances for SCT items were calculated using Cronbach’s alpha statistic. Results The SCT-EM was administered to 68 residents and 13 attendings. There was a significant difference in mean scores among all groups (mean + standard deviation: PGY-1 59 + 7; PGY-2 62 + 6; PGY-3 60 + 8; PGY-4 61 + 8; 73 + 8 for attendings, p < 0.01). Post hoc pairwise comparisons demonstrated that significant difference in mean scores only occurred between each PGY level and the attendings (p < 0.01 for PGY-1 to PGY-4 vs attending group). Performance on the SCT-EM and EM Milestones was not significantly correlated (r = 0.12, p = 0.35). Internal reliability of the exam was determined using Cronbach’s alpha, which was 0.67 for all examinees, and 0.89 in the expert-only group. Conclusion The SCT-EM has limited utility in reliably assessing clinical reasoning among EM residents. Although the SCT-EM was able to differentiate clinical reasoning ability between residents and expert faculty, it did not between PGY levels, or correlate with Milestones scores. Furthermore, several limitations threaten the validity of the SCT-EM, suggesting further study is needed in more diverse settings.

Initial validation of a script concordance test to measure the development of clinical reasoning among physical therapy residents

“…13 This number exceeded the criteria for panel membership (at least 15 members), which has been found to provide adequate reliability. 9,16,19 The modal score for each item was determined (there were no ties), which became the answer that received full credit. 13,16 The modal scoring method was chosen to grade the residents' tests because this method allows for a single best answer, making scoring simple.…”

“…SCTs have differentiated students in geriatric medicine from clinicians, 14 students from general practitioners, 3 and residents in an emergency department from expert practicing clinicians. 19 Traditionally, 'experts' are practicing clinicians in the subject area covered by the SCT with entry-level credentials expected for practice within the profession being investigated. [9][10][11][12][13][14][15][16][17]20 To avoid confusion, we will refer to the practitioners used to validate the SCT for this study as 'practicing clinicians'.…”

Script concordance testing in genetic counseling training: A pilot study

“…Graduate training programs across multiple healthcare fields have successfully utilized SCT as a standardized way to measure trainees' clinical reasoning skills, establishing the reliability and validity of the tool in those settings (Dawson et al, 2014; Kaur et al, 2020; Lambert et al, 2009; Lubarsky et al, 2009; Steinberg et al, 2020). The tool is perceived positively by trainees and can discriminate between trainees with sound clinical judgment and those that may require remedial action (Lambert et al, 2009).…”