Synchronous Collection of Multisource Feedback Evaluations Does Not Increase Inter‐rater Reliability

Abstract

Objectives: Most multisource feedback (MSF) evaluations are performed asynchronously, with raters reflecting on the subject's behavior. Numerous studies have demonstrated poor inter-rater reliability of MSF. This may be due to cognitive biases that are inherent in such a process. We sought to determine if within-and between-rater group reliability is increased when evaluations are gathered synchronously and relate to a specific patient interaction.Methods: This was a survey at a university emergency department (ED) of 30 emergency medicine (EM) residents. ED nurses and faculty anonymously participated in asynchronous MSF assessment of resident performance from February to April 2010 using a Web-based survey, the Emergency Medicine Humanism Scale (EM-HS). In May 2010, a second round of MSF collection was conducted in the ED. At the conclusion of patient encounters, the EM-HS was synchronously obtained from ED nurses and faculty. Evaluators were instructed to assess the resident based on the patient encounter, placing aside any preconceptions of resident performance, attitude, or behavior. Evaluators rated resident performance using a 1-9 scale (''needs improvement'' to ''outstanding''). The mean rating for each of the questions and the total score provided by each evaluator class was calculated for each EM resident. Differences between the asynchronous and synchronous ratings were compared with t-tests. Pearson correlations were used to measure agreement in scores within and between nurse and faculty rater groups. Correlations > 0.70 were deemed acceptable and are reported with 95% confidence intervals (CIs).Results: Twenty-one of 30 residents had assessments collected by both asynchronous and synchronous methods. A total of 699 Web-based (asynchronous) assessments were completed by nurses and 149 by faculty. Synchronous nurse and faculty assessments were obtained in 105 resident-patient encounters. There was no difference in faculty ratings between the MSF collection methods. Nurses assigned slightly (but significantly) higher ratings during synchronous collection. Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within the faculty rater group was poor (0.18, 95% CI = )0.22 to 0.60). Correlation of the total MSF score between asynchronous and synchronous feedback collection methods within nurse rater groups was moderate (0.63, 95% CI = 0.27 to 0.83). Correlations between faculty-nurse rater groups for the total MSF collected asynchronously and synchronously were moderate (0.39, 95% CI = )0.05 to 0.7; and 0.44, 95% CI 0.01 to 0.73, respectively).Conclusions: Synchronous collection of MSF did not provide clinically different EM-HS scores within rater groups and did not result in improved correlations. Our small, single-center study supports asynchronous collection of MSF.ACADEMIC EMERGENCY MEDICINE 2011; 18:S65-S70 ª 2011 by the Society for Academic Emergency Medicine T he traditional approach to resident assessment relies upon remote memory for performanc...

Workplace‐based Assessment Data in Emergency Medicine: A Scoping Review of the Literature

“…Six papers were conceptual pieces (e.g., narrative review, opinion, editorial, or other theory‐advancing commentary) 50–55 . Three papers were classified as justification studies 34,41,56 . Justification studies focus on the last step in the scientific method by comparing one educational intervention with another to address whether the intervention was effective.…”

“…A major challenge to collecting WBA data cited among several studies related to the feasibility of assessment methods and processes 43,51–53,55,56 . Caretta‐Weyer and Gisondi 55 emphasized the need to simplify frontline assessment to be more intuitive for assessors to complete.…”

“…Future research should expand on this and determine strategies for increasing both the quantity and the quality of WBA data provided. Moreover, given the complexity and time‐pressured nature of the emergency department environment, efforts should also be made to facilitate WBAs by streamlining forms and utilizing technology to facilitate data entry and capture 33,51–53,55,56 …”

“…[50][51][52][53][54][55] Three papers were classified as justification studies. 34,41,56 Justification studies focus on the last step in the scientific method by comparing one educational intervention with another to address whether the intervention was effective. The remaining two papers were clarification studies.…”

“…The reviewed publications consisted of six different methodologies. Nine studies were quantitative, 12,14,33,36,38,39,41,43,48,56 six were qualitative, 35,37,42,45,47,49 and two included mixed methodologies. 17,34 Many of the remaining publications were conceptual pieces, [50][51][52][53][54][55] innovation reports without results, 8,40 or consensus conference proceedings.…”