Competency-based simulation assessment of resuscitation skills in emergency medicine postgraduate trainees – a Canadian multi-centred study

Abstract

Background: The use of high-fidelity simulation is emerging as a desirable method for competency-based assessment in postgraduate medical education. We aimed to demonstrate the feasibility and validity of a multi-centre simulation-based Objective Structured Clinical Examination (OSCE) of resuscitation competence with Canadian Emergency Medicine (EM) trainees.Method: EM postgraduate trainees (n=98) from five Canadian academic centres participated in a high fidelity, 3-station simulation-based OSCE. Expert panels of three emergency physicians evaluated trainee performances at each centre using the Queen’s Simulation Assessment Tool (QSAT). Intraclass correlation coefficients were used to measure the inter-rater reliability, and analysis of variance was used to measure the discriminatory validity of each scenario. A fully crossed generalizability study was also conducted for each examination centre. Results: Inter-rater reliability in four of the five centres was strong with a median absolute intraclass correlation coefficient (ICC) across centres and scenarios of 0.89 [0.65-0.97]. Discriminatory validity was also strong (p < 0.001 for scenarios 1 and 3; p < 0.05 for scenario 2). Generalizability studies found significant variations at two of the study centres.Conclusions: This study demonstrates the successful pilot administration of a multi-centre, 3-station simulation-based OSCE for the assessment of resuscitation competence in post-graduate Emergency Medicine trainees.

Comparison of Simulation‐based Resuscitation Performance Assessments With In‐training Evaluation Reports in Emergency Medicine Residents: A Canadian Multicenter Study

“…A complete description of the simulation‐based OSCE administration has previously been published . EM residents at each site participated in this high‐fidelity simulation‐based resuscitation OSCE.…”

“…First, while our study involved multiple centers, we collected a convenience sample of residents of varying levels of training from each site, resulting in a potentially nonrepresentative sample. While this may have served to bias scoring of either of the assessment methods, it is unlikely that this would have significantly affected the correlations between ITER and OSCE scores. Second, portfolio contents were likely variable between sites, with the exact number and types of assessments contributing to the ITER score not available.…”

“…The QSAT is a modifiable anchored global assessment scoring tool, which was developed to allow for competency‐based assessment of any EM resident in a simulated resuscitation under a number of different clinical and team‐based circumstances . Having developed an argument for the validity of the QSAT in simulation‐based OSCEs locally, the performance of the QSAT as an assessment tool in simulation‐based resuscitation OSCEs at multiple Canadian centers was subsequently evaluated . In this multicenter study, the QSAT was found to have reasonable inter‐rater reliability and strong discriminatory validity.…”

The Nightmares Course: A Longitudinal, Multidisciplinary, Simulation-Based Curriculum to Train and Assess Resident Competence in Resuscitation

“…A validity argument for the use of the QSAT in simulated resuscitations has been previously presented. 7,8 Modifications for the Nightmares Course OSCE included adding an EPA to anchor examiners and using an entrustment decision 4 instead of the previously utilized global assessment scale. All examiners underwent a 30-minute training session led by 1 study investigator (A.K.H.…”

Through the eyes of the physician

“…Another limitation is that only one rater was used to score participants' performance using the RAT. This being said, the QSAT (on which the RAT is based) has been shown to have an acceptable inter-rater reliability in the same simulation-based OSCE context (Dagnone et al, 2016;, which suggests that using multiple raters would likely have produced similar results.…”