Simulation versus real-world performance: a direct comparison of emergency medicine resident resuscitation entrustment scoring

Abstract

Background Simulation is increasingly being used in postgraduate medical education as an opportunity for competency assessment. However, there is limited direct evidence that supports performance in the simulation lab as a surrogate of workplace-based clinical performance for non-procedural tasks such as resuscitation in the emergency department (ED). We sought to directly compare entrustment scoring of resident performance in the simulation environment to clinical performance in the ED. Methods The resuscitation assessment tool (RAT) was derived from the previously implemented and studied Queen’s simulation assessment tool (QSAT) via a modified expert review process. The RAT uses an anchored global assessment scale to generate an entrustment score and narrative comments. Emergency medicine (EM) residents were assessed using the RAT on cases in simulation-based examinations and in the ED during resuscitation cases from July 2016 to June 2017. Resident mean entrustment scores were compared using Pearson’s correlation coefficient to determine the relationship between entrustment in simulation cases and in the ED. Inductive thematic analysis of written commentary was conducted to compare workplace-based with simulation-based feedback. Results There was a moderate, positive correlation found between mean entrustment scores in the simulated and workplace-based settings, which was statistically significant ( r = 0.630, n = 17, p < 0.01). Further, qualitative analysis demonstrated overall management and leadership themes were more common narratives in the workplace, while more specific task-based feedback predominated in the simulation-based assessment. Both workplace-based and simulation-based narratives frequently commented on communication skills. Conclusions In this single-center study with a limited sample size, assessment of residents using entrustment scoring in simulation settings was demonstrated to have a moderate positive correlation with assessment of resuscitation competence in the workplace. This study suggests that resuscitation performance in simulation settings may be an indicator of competence in the clinical setting. However, multiple factors contribute to this complicated and imperfect relationship. It is imperative to consider narrative comments in supporting the rationale for numerical entrustment scores in both settings and to include both simulation and workplace-based assessment in high-stakes decisions of progression. Electronic supplementary material The online version of this article (10.1186/s41077-019-0099-4) contains supplementary material, which is available to authorized users.

Starting to Think Like an Expert: An Analysis of Resident Cognitive Processes During Simulation-Based Resuscitation Examinations

“…24 PRIMARY DATA ANALYSIS Quantitative Analysis Each resident's performance was recorded with a ceilingmounted system (Kb Port, Allison Park, PA) and was then scored by an external blinded expert reviewer with no knowledge of the trainees, their training level, or the purpose of the study. The blinded external rater used an entrustment-based 25 scoring tool, termed the Resuscitation Assessment Tool 26 (Appendix E3, available online at http://www.annemergmed.com), which was developed as a modification of the previously derived Queen's Simulation Assessment Tool, which has validity evidence in a similar context. Strictly speaking, the Resuscitation Assessment Tool itself has not been directly validated, but it was deemed to be more contextually appropriate, given its widespread use in resuscitation assessment in the emergency medicine residency program at our institution.…”

Immersive Video Modeling Versus Traditional Video Modeling for Teaching Central Venous Catheter Insertion to Medical Residents

“…Thirdly, the benefit of VM in the theoretical model of deliberate practice may be rooted in the learners' ability to observe and analyze the performance of experts at key decision points, with expert guidance [ 23 ]. Entrustment scores in simulated environments may be an indicator of competence in the clinical setting [ 25 ], suggesting that the improvements seen in this workshop may translate to enhanced patient care [ 26 ]. Procedural time is also a recognized surrogate of procedural mastery, with the level of experience being inversely related to time [ 21 , 25 - 27 ].…”

“…Entrustment scores in simulated environments may be an indicator of competence in the clinical setting [ 25 ], suggesting that the improvements seen in this workshop may translate to enhanced patient care [ 26 ]. Procedural time is also a recognized surrogate of procedural mastery, with the level of experience being inversely related to time [ 21 , 25 - 27 ]. However, we did not find this association in our study.…”

Performing newborn life support in advance of neonatal advanced life support course—back to basics?

“…Therefore, we focused on pre-course practical test performance. Although there is not a direct linear correlation between demonstrating practical skills on a manikin and performance in a real-life situation, it is generally accepted that there is a certain amount of transfer of skills in real life [ 8 ].…”