Are USMLE Scores Valid Measures for Chief Resident Selection?

Abstract

Background The US Medical Licensing Examination (USMLE) Step 1 and Step 2 scores are often used to inform a variety of secondary medical career decisions, such as residency selection, despite the lack of validity evidence supporting their use in these contexts. Objective We compared USMLE scores between non–chief residents (non-CRs) and chief residents (CRs), selected based on performance during training, at a US academic medical center that sponsors a variety of graduate medical education programs. Methods This was a retrospective cohort study of residents' USMLE Step 1 and Step 2 Clinical Knowledge (CK) scores from 2015 to 2019. The authors used archived data to compare USMLE Step 1 and Step 2 CK scores between non-CR residents in each of the eligible programs and their CRs during the 5-year study period. Results Sixteen programs enrolled a total of 1264 non-CRs and 210 CRs over the study period. There were no significant differences overall between non-CRs' and CRs' average USMLE Step 1 (239.95 ± 14.40 versus 241.50 ± 14.29; P = .15) or Step 2 (250.51 ± 14.02 versus 251.67 ± 14.82; P = .28) scores. Conclusions There was no link between USMLE Step 1 and Step 2 CK scores and CR selection across multiple clinical specialties over a 5-year period. Reliance on USMLE Step 1 and 2 scores to predict success in residency as measured by CR selection is not recommended.

Association Between Internal Medicine Residency Applicant Characteristics and Performance on ACGME Milestones During Intern Year

“…2 Research from other clinical specialties also has suggested that commonly used metrics (such as USMLE scores and interview performance) are not strongly predictive of residency success. [28][29][30] These findings may also reflect the challenges of using the ACGME Milestones as an assessment tool. These Milestones, while reflecting the theoretical educational outcomes of a program, may also be an imperfect measure of resident performance in the real world.…”

The Association Between USMLE Step 2 Clinical Knowledge Scores and Residency Performance: A Systematic Review and Meta-Analysis

“…Twenty-five studies were unable to be pooled. 59–83 Most of these studies could not be pooled because they did not analyze a continuous Step 2 CK score or their outcome of interest was not examined by at least 2 other studies. Of these studies, 2 showed a mixed relationship between higher Step 2 CK scores and the desired outcome, 5 showed a statistically unremarkable relationship between a higher Step 2 CK score and the desired outcome, and 18 showed a statistically significant relationship between a higher Step 2 CK score and the desired outcome.…”

“…For full-text review, 82 articles were reviewed for inclusion and 14 were excluded, leaving 68 studies that met inclusion criteria. 16–83 Agreement on study inclusion achieved a κ value of 0.85.…”

Diversity and Inclusion in Anesthesiology

“…For example, overemphasis on minor differences on scholastic aptitude test scores and advantages given to “legacy” applicants may lead to a less diverse undergraduate cohort. 41 Notwithstanding the lack of evidence that clinical competence is linearly correlated with a United States Medical Licensing Examination (USMLE) score, 42 residency programs in anesthesiology and other specialties have placed a greater premium on USMLE scores than other factors such as commitment to the community, demonstrated passion for practicing medicine, or research interests. 43 An observational study suggests a range of USMLE scores compatible with passing the anesthesiology basic staged examination.…”