Initial Validity Analysis of the Emergency Medicine Milestones

Abstract

Objectives: The Accreditation Council for Graduate Medical Education (ACGME) Milestones describe behavioral markers for the progressive acquisition of competencies during residency. As a key component of the Next Accreditation System, all residents are evaluated for the acquisition of specialtyspecific Milestones. The objective was to determine the validity and reliability of the emergency medicine (EM) Milestones. Methods:The ACGME and the American Board of Emergency Medicine performed this single-event observational study. The data included the initial EM Milestones performance ratings of all categorical EM residents submitted to the ACGME from October 31, 2013, to January 6, 2014. Mean performance ratings were determined for all 23 subcompetencies for every year of residency training. The internal consistency (reliability) of the Milestones was determined using a standardized Cronbach's alpha coefficient. Exploratory factor analysis was conducted to determine how the subcompetencies were interrelated.Results: EM Milestone performance ratings were obtained on 100% of EM residents (n = 5,805) from 162 residency programs. The mean performance ratings of the aggregate and individual subcompetency scores showed discrimination between residency years, and the factor structure further supported the validity of the EM Milestones. The reliability was a = 0.96 within each year of training. Conclusions:The EM Milestones demonstrated validity and reliability as an assessment instrument for competency acquisition. EM residents can be assured that this evaluation process has demonstrated validity and reliability; faculty can be confident that the Milestones are psychometrically sound; and stakeholders can know that the Milestones are a nationally standardized, objective measure of specialtyspecific competency acquisition.ACADEMIC EMERGENCY MEDICINE 2015;22:838-844

The Promise of Milestones: Are They Living Up to Our Expectations?

“…In contrast to the finding of a single FM competence construct, emergency medicine identified 3 constructs, 13 and internal medicine and obstetrics-gynecology identified 6 constructs aligning with the 6 ACGME competencies.…”

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

“…[1][2][3][4] Before 2014, internal medicine programs rated residents using the resident annual evaluation summary (RAES), with 9 items (ranging from unsatisfactory to superior). Subsequently, internal medicine milestones were introduced, and residency programs were required to submit milestone ratings to the Accreditation Council for Graduate Medical Education (ACGME) and American Board of Internal Medicine (ABIM).…”

Emergency Medicine Education Research Since the 2012 Consensus Conference: How Far Have We Come and What’s Next?

“…Educators at all levels were interested in finding optimal methods for assessing their learners. This was also at a time when core competencies in residency education were being modified to identify milestone‐based subcompetencies . The conference’s keynote address, “The focus on competencies and individual learner assessment as emerging themes in medical education research” set the stage for eight of the breakout consensus sessions.…”