The Ottawa Emergency Department Shift Observation Tool (O‐EDShOT): A New Tool for Assessing Resident Competence in the Emergency Department

Abstract

Objectives: The outcome of emergency medicine (EM) training is to produce physicians who can competently run an emergency department (ED) shift. However, there are few tools with supporting validity evidence specifically designed to assess multiple key competencies across an entire shift. The investigators developed and gathered validity evidence for a novel entrustment-based tool to assess a resident's ability to safely run an ED shift. Methods: Through a nominal group technique, local and national stakeholders identified dimensions of performance that are reflective of a competent ED physician and are required to safely manage an ED shift. These were included as items in the Ottawa Emergency Department Shift Observation Tool (O-EDShOT), and each item was scored using an entrustment-based rating scale. The tool was implemented in 2018 at the

Comparing the Ottawa Emergency Department Shift Observation Tool (O-EDShOT) to the traditional daily encounter card: measuring the quality of documented assessments

“…In competency-based medical education, programs rely heavily on data gathered through workplace-based assessments, and high-quality data is necessary to provide formative feedback and make important decisions about trainee advancement. Previous research on the O-EDShOT has demonstrated its strong psychometric properties; however, this is the first study to demonstrate that the O-EDShOT facilitates higher-quality assessments, which has pragmatic value for driving actionable feedback and for high-stakes decisions about trainee progress and promotion [5,14,25]. Our results contribute additional validity evidence for the O-EDShOT at the implications inference stage of Kane's validity framework [26].…”

“…Résultats Les scores CCERR étaient significativement plus élevés pour les O-EDShOT (moyenne (ET) = 25,6 (2,6)) par rapport aux fiches de rencontre quotidiennes (21,5 (3,9) ; t (23) = 5,2, p < 0,001, d = 1,1). Les scores A-CCERR étaient également significativement plus élevés pour les O-EDShOT (moyenne (ET) = 18,5 (1,6)) que pour les fiches de rencontre quotidiennes (15,5 (1,2) ; t (24) = 8,4, p < 0,001). Les points 1, 4 et 9 du CCERR ont été notés beaucoup plus haut pour les O-EDShOT que pour les fiches de rencontre quotidiennes.…”

“…A common form of workplace-based assessment in the emergency department (ED) is the end-of-shift assessment, traditionally in the form of a daily encounter card, which is intended to capture the performance of a trainee during a shift [2]. Daily encounter cards facilitate the simultaneous assessment of multiple competencies during an ED shift [2][3][4] and are designed to capture expert judgement of a trainee's performance after a brief supervisor-trainee interaction, thus minimizing recall bias [2][3][4][5][6][7].…”

“…With the transition to competency-based medical education occurring for all post-graduate medical education programs effectively manage an ED shift-a defining attribute of any emergency physician and a key outcome of residency training [5,10,11]. The O-EDShOT uses a rating scale that incorporates entrustment anchors that have been shown to produce more reliable ratings of performance compared to traditional anchors [5,[12][13][14][15][16]. Previous studies have demonstrated that the O-EDShOT is able to discriminate performance abilities between training levels and provide a reliable measure of a trainee's ability to safely run an ED shift independently.…”

“…Previous studies have demonstrated that the O-EDShOT is able to discriminate performance abilities between training levels and provide a reliable measure of a trainee's ability to safely run an ED shift independently. Further, preceptors and trainees perceive the O-EDShOT to be more feasible, practical, and time efficient compared to the traditional daily encounter card [5]. What remains unknown is whether the O-EDShOT provides measurable improvements in the quality of documented assessments compared to the traditional daily encounter card.…”