Relationship between ratings of performance in the simulated and workplace environments among emergency medicine residents

Abstract

Objectives The Emergency Medicine (EM) Specialty Committee of the Royal College of Physicians and Surgeons of Canada (RCPSC) specifies that resuscitation entrustable professional activities (EPAs) can be assessed in the workplace and simulated environments. However, limited validity evidence for these assessments in either setting exists. We sought to determine if EPA ratings improve over time and whether an association exists between ratings in the workplace v. simulation environment. Methods All Foundations EPA1 (F1) assessments were collected for first-year residents (n = 9) in our program during the 2018–2019 academic year. This EPA focuses on initiating and assisting in the resuscitation of critically ill patients. EPA ratings obtained in the workplace and simulation environments were compared using Lin's concordance correlation coefficient (CCC). To determine whether ratings in the two environments differed as residents progressed through training, a within-subjects analysis of variance was conducted with training environment and month as independent variables. Results We collected 104 workplace and 36 simulation assessments. No correlation was observed between mean EPA ratings in the two environments (CCC(8) = -0.01; p = 0.93). Ratings in both settings improved significantly over time (F(2,16) = 18.8; p < 0.001; η2 = 0.70), from 2.9 ± 1.2 in months 1–4 to 3.5 ± 0.2 in months 9–12. Workplace ratings (3.4 ± 0.1) were consistently higher than simulation ratings (2.9 ± 0.2) (F(2,16) = 7.2; p = 0.028; η2 = 0.47). Conclusions No correlation was observed between EPA F1 ratings in the workplace v. simulation environments. Further studies are needed to clarify the conflicting results of our study with others and build an evidence base for the validity of EPA assessments in simulated and workplace environments.

Exploring the use of rating scales with entrustment anchors in workplace‐based assessment

“…However, if all EPAs were staged appropriately, one would expect the core of discipline EPAs, which involve PGY2‐4 trainees, to show lower scores in PGY2s when compared to PGY4s, which was not the case (Table 1). An alternate explanation is that the high scores are a result of trainees selectively initiating assessments for cases where they performed well 42‐44 . Faculty may also preferentially select to evaluate ‘good’ performances/encounters because it is easier to discuss a topic or area that needs little improvement compared with an area of weakness that may require more difficult conversations and constructive criticism.…”

Simulation vs workplace-based assessment in resuscitation: a cross-specialty descriptive analysis and comparison

“…The results from three studies correlating resuscitation performance in the workplace and simulation laboratory range from no correlation to a moderate positive correlation. [13][14][15] The most recent study found that on average workplace-based entrustment scores were higher but there was no correlation at the individual resident level. 13 Our finding that entrustment scores did not differ between the workplace and simulation settings across three training programs and two stages adds to the increasing evidence to support the use of simulation-based assessment of resuscitation-focused EPAs.…”

Simulation for assessment of Entrustable Professional Activities in an emergency medicine residency program

“…Simulation provides the opportunity for assessment under more direct and standardized conditions without extraneous cognitive load for either resident or assessor. It may also mitigate the previously reported tendency for residents to only select well-done EPAs for observation requests ensuring a consistent training experience and the potential identification of deficits very early in training [9,10].…”

“…Simulation for assessment also has limitations including cost of and access to equipment and personnel, fidelity concerns, and varied correlation between performance in simulation versus workplace environments that should be weighed against the benefits [6,7,9].…”