Straight Line Scoring by Clinical Competency Committees Using Emergency Medicine Milestones

Abstract

Most programs did not submit SLS ratings. Because of the statistical improbability of SLS, any SLS ratings reduce the validity assertions of the milestone assessments.

A National Study of Longitudinal Consistency in ACGME Milestone Ratings by Clinical Competency Committees: Exploring an Aspect of Validity in the Assessment of Residents’ Competence

“…10 Therefore, we chose to focus our initial analysis on data from 3 of the phase 1 specialties—emergency medicine (EM), diagnostic radiology (DR), and urology (UR)—because of the consistency in the milestone reporting protocol between these specialties and to build on early published validity research of milestone ratings. 11–14…”

Integration of Entrustable Professional Activities with the Milestones for Emergency Medicine Residents

“…While milestones have moved us towards CBME in the U.S., the assessment of individual milestones has proven difficult, as evidenced by more programs than expected submitting straight-line scoring 33,34. This may be due to assessors having difficulty translating the level of trust they have for a trainee to perform a specific work-based activity into the multiple requisite competency domains.…”

On Rating Angels: The Halo Effect and Straight Line Scoring

“…Beeson et al 2 found that this arose in only about 6% of all assessments. However, it was not evenly distributed across programs, with 4% of programs submitting more than 50% of their assessments with SLS.…”

“…Cooper 1 reviewed a number of potential causes, although he does not estimate the relative contribution of each. Undersampling reflects an inadequate number of observations, which may well apply in the context of firsttime ratings as suggested by Beeson et al 2 Engulfing occurs when ratings of individual attributes are influenced by an overall impression. Beeson et al 2 propose that the final ratings of graduating residents may have been a result of this phenomenon, where the score was linked to predicted success of graduation.…”

“…We suspect most supervisors and program directors can recall multiple examples of the extreme in this regard, such as the long, consistent line of 8 out of 10 checks without deviation on the rating form. This phenomenon is called straight line scoring (SLS) in the study by Beeson et al 2 …”