Feasibility and Reliability of a Multisource Feedback Tool for Emergency Medicine Residents

Abstract

Background While the Accreditation Council for Graduate Medical Education recommends multisource feedback (MSF) of resident performance, there is no uniformly accepted MSF tool for emergency medicine (EM) trainees, and the process of obtaining MSF in EM residencies is untested. Objective To determine the feasibility of an MSF program and evaluate the intraclass and interclass correlation of a previously reported resident professionalism evaluation, the Humanism Scale (HS). Methods To assess 10 third-year EM residents, we distributed an anonymous 9-item modified HS (EM-HS) to emergency department nursing staff, faculty physicians, and patients. The evaluators rated resident performance on a 1 to 9 scale (needs improvement to outstanding). Residents were asked to complete a self-evaluation of performance, using the same scale. Analysis Generalizability coefficients (Eρ2) were used to assess the reliability within evaluator classes. The mean score for each of the 9 questions provided by each evaluator class was calculated for each resident. Correlation coefficients were used to evaluate correlation between rater classes for each question on the EM-HS. Eρ2 and correlation values greater than 0.70 were deemed acceptable. Results EM-HSs were obtained from 44 nurses and 12 faculty physicians. The residents had an average of 13 evaluations by emergency department patients. Reliability within faculty and nurses was acceptable, with Eρ2 of 0.79 and 0.83, respectively. Interclass reliability was good between faculty and nurses. Conclusions An MSF program for EM residents is feasible. Intraclass reliability was acceptable for faculty and nurses. However, reliable feedback from patients requires a larger number of patient evaluations.

Developing and Implementing a Multisource Feedback Tool to Assess Competencies of Emergency Medicine Residents in the United States

“…3 In this study, we present a set of MSF instruments adapted for use in the ED setting from previously published instruments developed to measure resident behaviors across five core competencies in episodic care settings. While our pilot implementation of the instruments in eight diverse residency programs demonstrated excellent internal consistency (Cronbach's alpha = 0.96-0.97), we found that even with universal use of research assistants to facilitate data collection, it was difficult to achieve target numbers of assessments in all sites.…”

“…2 Multisource feedback (MSF), or 360-degree feedback, is a workplace-based assessment method that has been piloted in a variety of U.S. specialties, including emergency medicine (EM), internal medicine, anesthesiology, pediatrics, and family medicine, and has been in use for assessment of practicing physicians by The College of Physicians and Surgeons of Alberta (CPSA) Physician Achievement Program for the past decade. [3][4][5][6][7][8] As part of the CPSA's work, a set of MSF instruments was created for physicians who provide episodic care (including emergency physicians, locum physicians, physicians serving walk-in populations, urgent care physicians, hospitalists, and niche or specialty primary care physicians such as family planning and low-risk obstetrics). These instruments included specific questionnaires for patients, physician colleagues, nonphysician colleagues, and self-assessment.…”

Synchronous Collection of Multisource Feedback Evaluations Does Not Increase Inter‐rater Reliability

“…A previously reported instrument assessing EM resident interpersonal skills, attitudes, and behaviors, the Emergency Medicine Humanism Scale (EM‐HS), was used to obtain feedback. The EM‐HS is a MSF instrument that can be reliably administered to patients, nonphysician staff, and supervising faculty physicians 6 . The EM‐HS consists of nine questions for health care providers (Figure 1) with ratings on a nine‐point continuum from “needs improvement” to “outstanding.” The EM‐HS has previously demonstrated excellent generalizability coefficients within nurse and faculty rater groups (Eρ 2 = 0.83 and 0.79, respectively).…”

Assessing Professionalism: Summary of the Working Group on Assessment of Observable Learner Performance

“…It was found to be a feasible, reliable, and valid tool. Garra et al 32 studied an MSF tool, the EM-Humanism Scale, which evaluated U.S. EM residents' professionalism and interpersonal and communication skills. The instrument had good reliability for nurses and faculty, but only fair reliability for patient ratings.…”