Development of a Rigorously Designed Procedural Checklist for Assessment of Emergency Medicine Resident Performance of Temporary Transvenous Cardiac Pacing

Abstract

Objectives: Temporary transvenous cardiac pacing (TVP) is a potentially lifesaving intervention included in the list of essential core procedures for emergency medicine (EM) training; however, opportunities to perform TVP during residency cannot be guaranteed. EM graduates report feeling subjectively underprepared for this procedure, but objective performance data are lacking. Checklist-based simulated assessment is an increasing focus of competency-based medical education, particularly for invasive procedures like TVP. The objectives of this paper were as follows: first, to enlist a multidisciplinary team of experts to create an assessment tool for TVP using best practices in checklist development; second, to determine the reliability of checklist scoring; and third, to assess EM residents' baseline ability to perform TVP using a dedicated task trainer.Methods: This study was conducted at a single 4-year EM residency. A panel of emergency physicians and cardiologists designed a TVP checklist using a modified Delphi approach. After consensus was achieved on a final checklist, EM residents were assessed using a dedicated TVP task trainer. Inter-rater reliability was determined using Cohen's kappa coefficient. Resident performance was determined by number of correctly performed checklist items. Results:The expert panel achieved consensus on a 30-item checklist after three rounds of revisions. The Cohen's kappa coefficient for the overall checklist score was 0.87, with individual checklist items ranging from 0.63 to 1.00. In total, 58 residents were assessed with a mean score of 13.5 of 30 checklist items. Scores increased with each year of training.Conclusions: This study details the rigorous development of a TVP checklist designed by a multidisciplinary team of experts. Checklist scores demonstrated strong inter-rater reliability. The overall poor performance of this cohort suggests the current approach to TVP training does not provide sufficient preparation for EM residents. Competency-based techniques, such as simulation-based mastery learning, should be explored.

Simulation-based Mastery Learning Improves Emergency Medicine Residents’ Ability to Perform Temporary Transvenous Cardiac Pacing

“…A detailed description of the checklist’s design, content, and characteristics was previously reported, along with data demonstrating strong interrater reliability for checklist scores and the ability for the checklist to distinguish learners at different levels of training. 8 …”

“…6 Checklistbased assessments conducted in a simulation setting reveal poor baseline TVP performance, 7 with both EM residents and board-certified emergency physicians correctly performing, on average, fewer than half of the necessary steps of this critical procedure. 8,9 This reported lack of confidence and objective poor performance suggest the need for more effective models of TVP training during EM residency. Simulation provides an opportunity for learners to demonstrate procedural competence in a controlled setting.…”

“…A detailed description of the checklist's design, content, and characteristics was previously reported, along with data demonstrating strong interrater reliability for checklist scores and the ability for the checklist to distinguish learners at different levels of training. 8 A panel of 12 board-certified physicians (two cardiologists and 10 emergency physicians with experience performing and teaching TVP) established an MPS for the checklist using the Mastery Angoff approach. 18 Panelists were recruited from our professional network and represent a diversity of geographic practice areas.…”

“…6 Checklist-based assessments conducted in a simulation setting reveal poor baseline TVP performance, 7 with both EM residents and board-certified emergency physicians correctly performing, on average, fewer than half of the necessary steps of this critical procedure. 8 , 9 …”