A Faculty Development Program to Reduce Rater Error on Milestone-Based Assessments

Abstract

IntroductionCompetency-based assessment of resident performance requires faculty who are prepared to provide these assessments. Studies have shown that end-of-rotation evaluations completed by faculty lack reliability and fail to identify important deficiencies in residents' performance. 1,2A number of factors account for the less-than-acceptable reliability, including variations in clinical context or patient complexity, 3 lack of longitudinal experiences with residents and patients, 4 and rater error. Types of rater error include leniency error, in which residents' abilities are rated higher than performance merits; halo or reverse halo, when performance in 1 domain affects ratings in other domains; range restriction, in which ratings fall within a narrow range; and error of undifferentiation, in which raters fail to differentiate among different performance domains. [5][6][7][8] In 2009 a Milestone Task Force of the Accreditation Council for Graduate Medical Education and American Board of Internal Medicine (ABIM) published its draft Milestone document, consisting of a list of 142 ''curricular'' Milestones (which are distinct from the 22 internal medicine reporting Milestones).9,10 These curricular Milestones represent specific, observable skills in the 6 competencies residents should demonstrate at each level of training during the 3 years of training. Although program directors have developed new assessment tools based on the Milestones, little is known about how best to educate and support faculty in using the new tools. To date, research has not examined the effect of faculty development in the use of a Milestone-based evaluation form. We sought to determine whether faculty development would reduce common rating errors among faculty completing Milestone-based end-of-rotation evaluations. Our hypothesis was that faculty who participated in faculty development AbstractBackground Rater errors, such as halo/reverse halo, range restriction, and leniency errors, are frequently cited as threats to the validity of resident assessment by faculty.

Examining the accuracy of residents’ self-assessments and faculty assessment behaviours in anesthesiology

“…Faculty development has been proposed as a way of improving rater bias, 39,40 although such interventions could still result in faculty scores that are more lenient 41 or more stringent. 42 It has been suggested that residency programs can identify a subset of faculty dedicated to learning and performance assessments, and make them responsible for the majority of resident assessments. 41,42 Yet, others point out that this strategy might be less effective since medical raters may be unresponsive to professional development (e.g., Cook et al).…”

Assessment of Emergency Medicine Residents’ Clinical Reasoning: Validation of a Script Concordance Test

“…We assessed convergent validity using the residents’ ACGME Milestones scores; however, ACGME Milestones scores have been suggested to lack inter-rater reliability, and consistency between GME programs. 38 – 39 Moreover, the ACGME Milestones are not a complete determination of residents’ abilities nor do they assess all areas essential to unsupervised practice. 40 Finally, the ACGME Milestones were not designed to be an assessment tool in this context.…”

Reflections on the First 2 Years of Milestone Implementation

“…14 Two studies determined that use of milestones was more effective than use of previous evaluation forms, and found better discrimination in ratings and a reduction in common rater errors. 15,16 On the other hand, a study of a milestone ''passport'' intervention in an emergency medicine program found only modest increases in resident satisfaction with feedback. 17 Another study reported that milestone-based assessments for end-of-shift evaluations led to grade inflation in an emergency medicine program.…”