Assessment of Emergency Medicine Resident Performance in an Adult Simulation Using a Multisource Feedback Approach

Abstract

IntroductionThe Accreditation Council for Graduate Medical Education (ACGME) specifically notes multisource feedback (MSF) as a recommended means of resident assessment in the emergency medicine (EM) Milestones. High-fidelity simulation is an environment wherein residents can receive MSF from various types of healthcare professionals. Previously, the Queen’s Simulation Assessment Tool (QSAT) has been validated for faculty to assess residents in five categories: assessment; diagnostic actions; therapeutic actions; interpersonal communication, and overall assessment. We sought to determine whether the QSAT could be used to provide MSF using a standardized simulation case.MethodsProspectively after institutional review board approval, residents from a dual ACGME/osteopathic-approved postgraduate years (PGY) 1–4 EM residency were consented for participation. We developed a standardized resuscitation after overdose case with specific 1–5 Likert anchors used by the QSAT. A PGY 2–4 resident participated in the role of team leader, who completed a QSAT as self-assessment. The team consisted of a PGY-1 peer, an emergency medical services (EMS) provider, and a nurse. Two core faculty were present to administer the simulation case and assess. Demographics were gathered from all participants completing QSATs. We analyzed QSATs by each category and on cumulative score. Hypothesis testing was performed using intraclass correlation coefficients (ICC), with 95% confidence intervals. Interpretation of ICC results was based on previously published definitions.ResultsWe enrolled 34 team leader residents along with 34 nurses. A single PGY-1, a single EMS provider and two faculty were also enrolled. Faculty provided higher cumulative QSAT scores than the other sources of MSF. QSAT scores did not increase with team leader PGY level. ICC for inter-rater reliability for all sources of MSF was 0.754 (0.572–0.867). Removing the self-evaluation scores increased inter-rater reliability to 0.838 (0.733–0.910). There was lesser agreement between faculty and nurse evaluations than from the EMS or peer evaluation.ConclusionIn this single-site cohort using an internally developed simulation case, the QSAT provided MSF with excellent reliability. Self-assessment decreases the reliability of the MSF, and our data suggest self-assessment should not be a component of MSF. Use of the QSAT for MSF may be considered as a source of data for clinical competency committees.

Direct Observation Tools in Emergency Medicine: A Systematic Review of the Literature

“…Twenty studies were conducted in the United States, 14–33 13 were performed in Canada, 34–46 three took place in Australia, 47–49 and two were conducted in Taiwan 50,51 . Thirty‐three studies involved resident physicians, 15,17–23,25–32,34–47,49,50 four studies involved medical students, 14,16,24,33 and one did not report the learner level 48 .…”

“…Twenty studies were conducted in the United States, [14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33] 13 were performed in Canada, [34][35][36][37][38][39][40][41][42][43][44][45][46] three took place in Australia, [47][48][49] and two were conducted in Taiwan. 50,51 Thirty-three studies involved resident 1.…”

“…48 Twenty-two were prospective studies, 14,16,17,[22][23][24][25][26][27][28]30,31,33,37,38,[40][41][42][46][47][48]51 nine were retrospective, [18][19][20]32,35,39,44,45,50 five were qualitative studies, 21,29,34,43,49 and two were descriptive studies. 15,36 Twenty-five studies were performed in the clinical environment, 14,15,[18][19][20][21]24,[26][27][28][29][30]…”

“…Three studies specifically evaluated the use of nonphysician observers. 15,25,28 Bedy et al 15 utilized EDbased pharmacists to specifically evaluate EM resident performance of the Pharmacotherapy Milestone (PC5). They found that pharmacist input was valuable for the determination of milestone levels during the clinical competency committee meetings.…”

A contest without losers – The value of extracurricular simulation competition in undergraduate medical education

“…The use of simulation in emergency medicine and anaesthetics have been well described and continues to be incorporated into these post-graduate training schemes (Lorello et al, 2014;Jong et al, 2019;Kester-Greene, Hall and Walsh, 2019). Exposure to simulation during undergraduate medical education is demonstrably variable and we have presented a potential solution to increase technical and non-technical skill development in acute medical scenarios.…”