E-ASSESS: Creating an EPA Assessment Tool for Structured Simulated Emergency Scenarios

Abstract

Background The entrustable professional activity (EPA) assessment framework allows supervisors to assign entrustment levels to physician trainees for specific activities. Limited opportunity for direct observation of trainees hampers entrustment decisions, in particular for infrequently performed activities. Simulation allows for direct observation, so tools to assess performance of EPAs in simulation could potentially provide additional data to complement clinical assessments. Objective We developed and collected validity evidence for a simulation-based tool grounded in the EPA framework. Methods We developed E-ASSESS (EPA Assessment for Structured Simulated Emergency ScenarioS) to assess performance in 2 EPAs among pediatric residents participating in simulation-based team training in 2017–2018. We collected validity data, applying Messick's unitary view. Three raters used E-ASSESS to assign entrustment levels based on performance in simulation. We compared those ratings to entrustment levels assigned by clinical supervisors (different from the study raters) for the same residents on a separate tool designed for clinical practice. We calculated intraclass correlation (ICC) for each tool and Pearson correlation coefficients to compare ratings between tools. Results Twenty-eight residents participated in the study. The ICC between the 3 raters for entrustment ratings on E-ASSESS ranged from 0.65 to 0.77, while ICC among raters of the clinical tool were 0.59 and 0.57. We found no significant correlations between E-ASSESS ratings and clinical practice ratings for either EPA (r = -0.35 and 0.38, P > .05). Conclusions Assessment following an EPA framework in the simulation context may be useful to provide data points to inform entrustment decisions as part of resident assessment.

Variable or Fixed? Exploring Entrustment Decision Making in Workplace- and Simulation-Based Assessments

“…Our participants' multiple meanings of entrustment and variable benchmarks when assessing EPAs align with the rater cognition literature, which describes significant idiosyncrasy and variable frames of reference among raters in medical education. [15][16][17][18] Prior studies have also demonstrated that entrustment scores vary more between raters than when the same raters score clinical or communication skills on an objective structured clinical exam, 11 that entrustment scores from SBAs and WBAs do not necessarily correlate, 12 and that there is cognitive variability in how raters interpret trainee performance when using an entrustment scale 36 -all findings that may be explained by rater idiosyncrasy or a lack of construct clarity. Although our participants' variable interpretations of entrustment go against the view that entrustment is intuitive, 3,4 their lack of sustained hands-on rater training could have contributed to the observed variability.…”

“…Key questions include what influence these assessment settings may have on raters’ cognitive processes when assessing EPAs and how this influence affects how data from WBAs and SBAs can be used to make decisions in CBME programs. 14 Despite these unique challenges, many competence committees currently consider EPA assessments from WBAs and SBAs in their decision making, sometimes interchangeably, 12,27 a practice we argue needs appraisal.…”

“…11 Furthermore, one study suggests that EPA assessment scores from simulation did not correlate with scores trainees had received in the workplace. 12 Given the prevailing postpositivist philosophy of assessment 13 in CBME, these findings raise questions about how best to standardize the use of EPA data generated in SBAs when making summative entrustment decisions.…”

Twelve tips for essential aspects in the elaboration of Global Medical Curricula in a context of Medical Competencies, EPAs, and Milestones

“…Cultural change is needed, starting with integrated learning, a challenge still under construction (Kayani, Gilani and Mahboob, 2018). The organization of the assessment system is as fundamental as the development of methods applicable to simulation stations (virtual or real), and to the assessment in the workplace with feedback (Prins, Brondt and Malling, 2019;Andler et al, 2020).…”

The Reliability of Rater Variability

“…[1][2][3] In this issue of the Journal of Graduate Medical Education, Andler and colleagues present validity evidence for leveraging the simulation context to provide assessment data for entrustable professional activities (EPAs). 4 Unfortunately, they found their validity argument hampered by an unexpected finding: despite good interrater reliability for entrustmentbased simulation assessment ratings and fair interrater reliability for similar entrustment-based clinical practice ratings, there were no correlations between them. The authors ponder possible explanations for this troublesome finding and suggest that since there was only ''fair agreement at best'' for some of the behaviors, rater variability might be an explanation for the lack of correlations.…”