Milestone-Based Assessments Are Superior to Likert-Type Assessments in Illustrating Trainee Progression

Abstract

Background The Pediatrics Milestone Project uses behavioral anchors, narrative descriptions of observable behaviors, to describe learner progression through the Accreditation Council for Graduate Medical Education competencies. Starting June 2014, pediatrics programs were required to submit milestone reports for their trainees semiannually. Likert-type scale assessment tools were not designed to inform milestone reporting, creating a challenge for Clinical Competency Committees. Objective To determine if milestone-based assessments better stratify trainees by training level compared to Likert-type assessments. Methods We compared assessment results for 3 subcompetencies after changing from a 5-point Likert scale to milestone-based behavioral anchors in July 2013. Program leadership evaluated the new system by (1) comparing PGY-1 mean scores on Likert-type versus milestone-based assessments; and (2) comparing mean scores on the Likert-type versus milestone-based assessments across PGY levels. Results Mean scores for PGY-1 residents were significantly higher on the prior year's Likert-type assessments than milestone-based assessments for all 3 subcompetencies (P < .01). Stratification by PGY level was not observed with Likert-type assessments (eg, interpersonal and communication skills 1 [ICS1] mean score for PGY-1, 3.99 versus PGY-3, 3.98; P = .98). In contrast, milestone-based assessments demonstrated stratification by PGY level (eg, the ICS1 mean score was 3.06 for PGY-1, 3.83 for PGY-2, and 3.99 for PGY-3; P < .01 for PGY-1 versus PGY-3). Significantly different means by trainee level were noted across 21 subcompetencies on milestone-based assessments (P < .01 for PGY-1 versus PGY-3). Conclusions Initial results indicate milestone-based assessments stratify trainee performance by level better than Likert-type assessments. Average PGY-level scores from milestone-based assessments may ultimately provide guidance for determining whether trainees are progressing at the expected pace.

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

“…6 Items on rating forms may not align with behaviors or qualities that supervisors observe or how supervisors think. [7][8][9] Ratings of trainee performance tend to be inflated, clustered at the high end of the scale, and based on a limited number of trainee characteristics. [10][11][12] To what degree milestone ratings will ameliorate these problems remains to be determined.…”

“…6 Pediatric milestone ratings within a single training program also stratified residents by training year, whereas nonmilestone-based ratings did not. 7 It is critically important to assess the validity of any new assessment. Validity evidence is needed to determine the degree to which milestone ratings measure trainees' provision of highquality patient care.…”

Genetic Counseling Milestones: A Framework for Student Competency Evaluation

“…Preliminary data on validity, feasibility, and outcomes of milestones are beginning to be reported and many more are likely to emerge in the next several years. Initial data has provided supporting evidence that milestone-based assessments are better at stratifying trainee performance than Likert-type assessments (Bartlett et al 2015). However, conceptual and logistical concerns, such as challenges in the identification of attainment and implications for length of training, continue to be explored (Carraccio et al 2014;Norman et al 2014).…”

Milestone Implementation’s Impact on Narrative Comments and Perception of Feedback for Internal Medicine Residents: a Mixed Methods Study

“…12 While the impact of milestone-aligned rotation evaluation forms (henceforth termed: milestone-based evaluation forms) on quantitative data is suggestive of reduced rater error and the ability to better differentiate level of training, the impact on narrative comments is unclear. 13,14 Comments can be valuable resources for trainees to review if they contain specific information and constructive criticism. 8,15 Narrative comments can also be sources of information for program leadership to determine how residents are performing.…”