Stages of Milestones Implementation: A Template Analysis of 16 Programs Across 4 Specialties

Abstract

Background Since 2013, US residency programs have used the competency-based framework of the Milestones to report resident progress and to provide feedback to residents. The implementation of Milestones-based assessments, clinical competency committee (CCC) meetings, and processes for providing feedback varies among programs and warrants systematic examination across specialties. Objective We sought to determine how varying assessment, CCC, and feedback implementation strategies result in different outcomes in resource expenditure and stakeholder engagement, and to explore the contextual forces that moderate these outcomes. Methods From 2017 to 2018, interviews were conducted of program directors, CCC chairs, and residents in emergency medicine (EM), internal medicine (IM), pediatrics, and family medicine (FM), querying their experiences with Milestone processes in their respective programs. Interview transcripts were coded using template analysis, with the initial template derived from previous research. The research team conducted iterative consensus meetings to ensure that the evolving template accurately represented phenomena described by interviewees. Results Forty-four individuals were interviewed across 16 programs (5 EM, 4 IM, 5 pediatrics, 3 FM). We identified 3 stages of Milestone-process implementation, including a resource-intensive early stage, an increasingly efficient transition stage, and a final stage for fine-tuning. Conclusions Residency program leaders can use these findings to place their programs along an implementation continuum and gain an understanding of the strategies that have enabled their peers to progress to improved efficiency and increased resident and faculty engagement.

Accreditation Council for Graduate Medical Education Milestone Training Ratings and Surgeons’ Early Outcomes

“…The utility of the Milestone ratings for identifying vascular surgeons who may struggle with higher complication rates following EVAR was highest in those training programs with lower Milestone mean ratings. One possible explanation is that such programs may have been wary of grade inflation and understand better the heuristic value of using the entire range of Milestone ratings, as systematic differences in Milestone implementation processes between programs have been reported . Regardless, vascular surgery programs can use the results of this study to identify trainees needing additional educational interventions.…”

“…4 Despite this initiative, program directors, institutional officials, and trainees remain concerned about the enormous cost and burden of the Milestones, in light of the fact that there is limited empirical evidence of effectiveness in improving the quality of resident education. [5][6][7][8] A key advance would be linking performance during training to patient outcomes following training. A landmark general surgery study by Birkmeyer et al 9 demonstrated a direct link between individual surgeon technical skill and complication rates.…”

“…7,8 Regardless, vascular surgery programs can use the results of this study to identify trainees needing additional educational interventions. For example, if a resident has low Milestone ratings with 18 months left in training, the program director would be welladvised to intervene.…”

The Urgency of Now: Rethinking and Improving Assessment Practices in Medical Education Programs

“…A developmental mindset requires significant rethinking of the purpose of assessment to better support professional development, feedback, and coaching. 28 This will require a greater focus on workplace-based assessment (WBA) because, unlike other assessments, WBAs can provide more timely, longitudinal feedback and occur at the point-of-care with patients and families. Technology is also enabling more efficient implementation of WBA with greater ability to capture narrative assessment using natural language processing on smart phones.…”

“…Figure 3b brings together the key concepts of professional development learning curves, programmatic assessment, and iterative learning cycles using a coproduction approach. (An expanded version of Figure 3 28 In 2017, all residencies at Queens University in Canada implemented an outcomes-based, timevariable approach and shared their early implementation results using rapid cycle evaluations. They uncovered several expected and unexpected challenges, a common phenomenon with major system changes.…”