Analysis of Milestone-based End-of-rotation Evaluations for Ten Residents Completing a Three-year Anesthesiology Residency

Abstract

IntroductionFaculty are required to assess the development of residents using educational milestones. This descriptive study examined the end-of-rotation milestone-based evaluations of anesthesiology residents by rotation faculty directors. The goals were to measure: (1) how many of the 25 Accreditation Council for Graduate Medical Education (ACGME) anesthesiology subcompetency milestones were included in each of the residency’s rotations evaluations, (2) the percentage of evaluations sent to the rotation director that were actually completed by the director, (3) the length of time between the end of the residents' rotations and completion of the evaluations, (4) the frequency of straight line scoring, defined as the resident receiving the same milestone level score for all subcompetencies on the evaluation, and (5) how often a resident received a score below a Level 4 in at least one subcompetency in the three months prior to graduating.MethodsIn 2013, the directors for each the 24 anesthesia rotations in the Stanford University School of Medicine Anesthesiology Residency Program created new milestone-based evaluations to be used at the end of rotations to evaluate residents. The directors selected the subcompetencies from the list released by the ACGME that were most appropriate for their rotation. End-of-rotation evaluations for the post-graduate year (PGY)-2 to PGY-4 from July 1, 2014 to June 30, 2017 were retrospectively analyzed for a sample of 10 residents randomly selected from 22 residents in the graduating class.ResultsThe mean number of subcompetencies evaluated by each of the 24 rotations in the residency equaled 17.88 (standard deviation (SD): 3.39, range 10–24, median 18.5) from the available possible total of 25 subcompetencies. Three subcompetencies (medical knowledge, communication with patients and families, and coordination of patient care within the healthcare system) were included in the evaluation instruments of all 24 rotations. The three least frequently listed subcompetencies were: “acute, chronic, and cancer-related pain consultation/management” (25% of rotations had this on the end-of-rotation evaluation), “triage and management of critically ill patient in non-operative setting” (33%), and “education of patient, families, students, residents, and others” (38%). Overall, 418 end-of-rotation evaluations were issued and 341 (82%) completed, with 63% completed within one month, 22% between month one and two, and 15% after two months. The frequency of straight line scoring varied, from never occurring (0%) in three rotations to always occurring (100%) in two rotations, with an overall average of 51% (SD: 33%). Sixty-one percent of straight line scoring corresponded to the residents’ postgraduate year whereby, for example, a post-graduate year two resident received an ACGME Level 2 proficiency for all subcompetencies. Thirty-one percent of the straight line scoring was higher than the resident’s year of training (e.g., a PGY-2 received Level 3 or higher for all the subcompetencies). The rema...

Trainee Perspectives on the Writing and Implementation of Milestones 2.0

“…Literature has suggested this practice is common and leads to lost assessment opportunities to provide trainees feedback on their abilities. 2,3 Following the initial creation of the Milestones, programs large and small were tasked with writing new assessment tools resulting in substantial changes to resident and fellow evaluations. Unfortunately, residents and fellows were at times left out of the process.…”

Review of Cognitive Biases in ACGME Milestones Training Assessments in Post-graduate Medical Education Programs

“…Studies have examined the milestones' impact on evaluations with narrative comments and the perception of feedback for internal medicine residents and found residents did not feel the milestones project improved these [10]. There can be a significant delay between rotation completion and timely faculty feedback, which erodes the educational utility of feedback [11].…”