Does Residency Selection Criteria Predict Performance in Orthopaedic Surgery Residency?

Abstract

Background More than 1000 candidates applied for orthopaedic residency positions in 2014, and the competition is intense; approximately one-third of the candidates failed to secure a position in the match. However, the criteria used in the selection process often are subjective and studies have differed in terms of which criteria predict either objective measures or subjective ratings of resident performance by faculty. Questions/purposes Do preresidency selection factors serve as predictors of success in residency? Specifically, we asked which preresidency selection factors are associated or correlated with (1) objective measures of resident knowledge and performance; and (2) subjective ratings by faculty. Methods Charts of 60 orthopaedic residents from our institution were reviewed. Preresidency selection criteria examined included United States Medical Licensing Examination (USMLE) Step 1 and Step 2 scores, Medical College Admission Test (MCAT) scores, number of clinical clerkship honors, number of letters of recommendation, number of away rotations, Alpha Omega Alpha (AOA) honor medical society membership, fourthyear subinternship at our institution, and number of publications. Resident performance was assessed using objective measures including American Board of Orthopaedic Surgery (ABOS) Part I scores and Orthopaedics InTraining Exam (OITE) scores and subjective ratings by faculty including global evaluation scores and faculty rankings of residents. We tested associations between preresidency criteria and the subsequent objective and subjective metrics using linear correlation analysis and Mann-Whitney tests when appropriate. Results Objective measures of resident performance namely, ABOS Part I scores, had a moderate linear correlation with the USMLE Step 2 scores (r = 0.55, p \ 0.001) and number of clinical honors received in medical school (r = 0.45, p \ 0.001). OITE scores had a weak linear correlation with the number of clinical honors (r = 0.35, p = 0.009) and USMLE Step 2 scores (r = 0.29, p = 0.02). With regards to subjective outcomes, AOA membership was associated with higher scores on the global evaluation (p = 0.005). AOA membership also correlated with higher global evaluation scores (r = 0.60, p = 0.005) with the strongest correlation existing between AOA membership and the ''interpersonal and communication skills

The Current Status of Medical School Clerkship Grades in Residency Applicants

“…Thus, it seems that for specialties that prioritize an individual clerkship grade when evaluating applicants, such as orthopedic surgery does with the surgery clerkship, using an individual grade may not be a reliable criterion. Prior studies 5,10 have reported correlations between medical school grades (ie, number of honors grades) and objective outcome measures of orthopedic resident success, including OITE scores and ABOS Part I scores. However, our study demonstrates that there is a wide variation in the distribution of honors grades among all clerkships, with an average 30% of students receiving an honors grade for each clerkship across all medical schools analyzed.…”

USMLE Step 2 CK: Best Predictor of Multimodal Performance in an Internal Medicine Residency

“…Our data add to the growing body of literature suggesting that USMLE Step 2 CK may be a better predictor of resident performance. 22,24,[31][32][33][34] Reasons for these findings may be secondary to USMLE Step 2 CK being more clinically relevant or closer in time to residency graduation and board examinations. Reasons for why having an advanced degree was associated with higher patient communication scores may include residents having more life experience, more maturity, and/or completion of a previous rigorous training program.…”

“…Despite evidence for USMLE Step 2 CK, USMLE Step 1 scores continue to be one of the highest cited factors used by many residency programs in selecting applicants for interviews, although the available evidence suggests residency programs may do better by giving more weight to USMLE Step 2 CK in the application process. 22,24,[31][32][33][34] A major limitation of our study was that it was completed at a single institution and all clinical performance was measured only from a unique IM residency ambulatory long block structure. In addition, the staff, peer, and allied health assessment tools used in our program did not have significant supportive validity evidence for use.…”

“…[19][20][21] Despite the heavy reliance on USMLE Step 1 scores, recent studies suggested USMLE Step 2 Clinical Knowledge (CK) actually may be a better predictor of ITE scores and resident performance overall. [22][23][24][25] Many of these studies compare application materials to only 1 or 2 other assessment metrics, usually standardized test scores and work-based observational faculty assessments. We believe these limited forms of assessment, while valuable, are not enough to fully capture a physician's competence.…”

Correlation of Medical College Admission Test Scores and Self-assessment Materials with the United States Medical Licensing Examination Step 1 Performance

“…Step 3 is taken during the first year of residency and evaluates the application of medical knowledge to the supervised practice of medicine. Of these, Step 1 scores have been shown to play an important role in residency selection by standardizing academic achievements of students from different schools, in predicting success in clinical clerkships and Step 2 examinations, and in board examinations taken during residencies [3][4][5][6][7][8][9]. Therefore, parameters influencing student performance on Step 1 examinations such as the Medical College Admission Test (MCAT) and related preparatory materials are of interest to both students as well as medical schools.…”