A Critical Disconnect: Residency Selection Factors Lack Correlation With Intern Performance

Abstract

Background Emergency medicine (EM) residency programs want to employ a selection process that will rank best possible applicants for admission into the specialty. Objective We tested if application data are associated with resident performance using EM milestone assessments. We hypothesized that a weak correlation would exist between some selection factors and milestone outcomes. Methods Utilizing data from 5 collaborating residency programs, a secondary analysis was performed on residents trained from 2013 to 2018. Factors in the model were gender, underrepresented in medicine status, United States Medical Licensing Examination Step 1 and 2 Clinical Knowledge (CK), Alpha Omega Alpha (AOA), grades (EM, medicine, surgery, pediatrics), advanced degree, Standardized Letter of Evaluation global assessment, rank list position, and controls for year assessed and program. The primary outcomes were milestone level achieved in the core competencies. Multivariate linear regression models were fitted for each of the 23 competencies with comparisons made between each model's results. Results For the most part, academic performance in medical school (Step 1, 2 CK, grades, AOA) was not associated with residency clinical performance on milestones. Isolated correlations were found between specific milestones (eg, higher surgical grade increased wound care score), but most had no correlation with residency performance. Conclusions Our study did not find consistent, meaningful correlations between the most common selection factors and milestones at any point in training. This may indicate our current selection process cannot consistently identify the medical students who are most likely to be high performers as residents.

To the Editor: Retrospective “Ranking” of Former Graduate Trainees by Senior Faculty Who Worked With Them During Their Residency

“…e read with interest the article, ''A Critical Disconnect: Residency Selection Factors Lack Correlation With Intern Performance,'' by Burkhardt et al and would like to commend the authors for their valuable investigation into the status quo of the residency match process. 1 Their conclusion corroborates our experience. All residency programs seek to identify and recruit whom they believe are the most promising medical school graduates-typically based on USMLE scores, class ranking, clinical grades, letters of recommendation, etc.…”

Medical Schools as Racialized Organizations: How Race-Neutral Structures Sustain Racial Inequality in Medical Education—a Narrative Review

“…Still, evidence suggests that USMLE Step 1 scores are not the best predictor of performance during residency, despite being the most critical selection criteria. [49][50][51] Certainly, board scores are predictive of first-time specialty board passing odds, but only at a threshold beyond which the difference between applicants in terms of success odds is negligible, [52][53][54] yet beyond using cutoffs to grant interviews, program directors cite board scores as a leading factor when ranking applicants. 55 This raises the need to deemphasize board scores above the threshold and support trainees who may have lower success odds at specialty board examinations.…”

Adaptive expertise: The optimal outcome of emergency medicine training

“…For instance, multiplechoice question exams test recall of facts and do not routinely assess conceptual understanding; they offer little predictive value about future clinical performance. 19 Similarly, competency-focused assessments for specific procedures risk a "procedural fluency trap," wherein learners focus on performing a routine skill more efficiently, but struggle when faced with task variability. 6 Both forms of assessment implicitly measure routine expertise but fail to capture adaptive expertise.…”

“…This focus on conceptual understanding addresses important flaws in our existing educational system. For instance, multiple‐choice question exams test recall of facts and do not routinely assess conceptual understanding; they offer little predictive value about future clinical performance 19 . Similarly, competency‐focused assessments for specific procedures risk a “procedural fluency trap,” wherein learners focus on performing a routine skill more efficiently, but struggle when faced with task variability 6 .…”