The Relationship Between Internal Medicine Residency Graduate Performance on the ABIM Certifying Examination, Yearly In-Service Training Examinations, and the USMLE Step 1 Examination

Kay, Cynthia; Jackson, Jeffrey L.; Frank, Michael

doi:10.1097/acm.0000000000000500

Cited by 74 publications

(67 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior studies have shown that internal medicine in-training examination scores predict certification examination scores. [26][27][28] However, in one study, A full unit change in the resident annual evaluation summary score is equivalent to a 0.5 unit change on the milestone score. b Regression-adjusted changes mean milestone ratings across program years is a significantly different change in the mean resident annual evaluation summary (RAES) ratings (P < .001).…”

Section: Discussionmentioning

confidence: 99%

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Hauer

Vandergrift

Hess³

et al. 2016

JAMA

View full text Add to dashboard Cite

IMPORTANCE US internal medicine residency programs are now required to rate residents using milestones. Evidence of validity of milestone ratings is needed.OBJECTIVE To compare ratings of internal medicine residents using the pre-2015 resident annual evaluation summary (RAES), a nondevelopmental rating scale, with developmental milestone ratings. DESIGN, SETTING, AND PARTICIPANTSCross-sectional study of US internal medicine residency programs in the 2013-2014 academic year, including 21 284 internal medicine residents (7048 postgraduate-year 1 [PGY-1], 7233 PGY-2, and 7003 PGY-3).EXPOSURES Program director ratings on the RAES and milestone ratings. MAIN OUTCOMES AND MEASURESCorrelations of RAES and milestone ratings by training year; correlations of medical knowledge ratings with American Board of Internal Medicine (ABIM) certification examination scores; rating of unprofessional behavior using the 2 systems. RESULTSCorresponding RAES ratings and milestone ratings showed progressively higher correlations across training years, ranging among competencies from 0.31 (95% CI, 0.29 to 0.33) to 0.35 (95% CI, 0.33 to 0.37) for PGY-1 residents to 0.43 (95% CI, 0.41 to 0.45) to 0.52 (95% CI, 0.50 to 0.54) for PGY-3 residents (all P values <.05). Linear regression showed ratings differed more between PGY-1 and PGY-3 years using milestone ratings than the RAES (all P values <.001). Of the 6260 residents who attempted the certification examination, the 618 who failed had lower ratings using both systems for medical knowledge than did those who passed (RAES difference, −0.9; 95% CI, −1.0 to −0.8; P < .001; milestone medical knowledge 1 difference, −0.3; 95% CI, −0.3 to −0.3; P < .001; and medical knowledge 2 difference, −0.2; 95% CI, −0.3 to −0.2; P < .001). Of the 26 PGY-3 residents with milestone ratings indicating deficiencies on either of the 2 medical knowledge subcompetencies, 12 failed the certification examination. Correlation of RAES ratings for professionalism with residents' lowest professionalism milestone ratings was 0.44 (95% CI, 0.43 to 0.45; P < .001).CONCLUSIONS AND RELEVANCE Among US internal medicine residents in the 2013-2014 academic year, milestone-based ratings correlated with RAES ratings but with a greater difference across training years. Both rating systems for medical knowledge correlated with ABIM certification examination scores. Milestone ratings may better detect problems with professionalism. These preliminary findings may inform establishment of the validity of milestone-based assessment.

show abstract

Section: Discussionmentioning

confidence: 99%

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Hauer

Vandergrift

Hess³

et al. 2016

JAMA

View full text Add to dashboard Cite

show abstract

“…There was very high correlation between service examination scores. 26 In cases where more than one was present, we used the average score. We examined the relationships between in-service scores and the quality of feedback and between the polarity (positive, negative, neutral) of feedback in the seven domains and in-service examination scores using analysis of variance.…”

Section: Methodsmentioning

confidence: 99%

The Quality of Written Feedback by Attendings of Internal Medicine Residents

et al. 2015

Self Cite

View full text Add to dashboard Cite

BACKGROUND: Attending evaluations are commonly used to evaluate residents. OBJECTIVES: Evaluate the quality of written feedback of internal medicine residents. DESIGN: Retrospective. PARTICIPANTS: Internal medicine residents and faculty at the Medical College of Wisconsin from 2004 to 2012. MAIN MEASURES: From monthly evaluations of residents by attendings, a randomly selected sample of 500 written comments by attendings were qualitatively coded and rated as high-, moderate-, or low-quality feedback by two independent coders with good inter-rater reliability (kappa: 0.94). Small group exercises with residents and attendings also coded the utterances as high, moderate, or low quality and developed criteria for this categorization. In-service examination scores were correlated with written feedback. KEY RESULTS: There were 228 internal medicine residents who had 6,603 evaluations by 334 attendings. Among 500 randomly selected written comments, there were 2,056 unique utterances: 29 % were coded as nonspecific statements, 20 % were comments about resident personality, 16 % about patient care, 14 % interpersonal communication, 7 % medical knowledge, 6 % professionalism, and 4 % each on practice-based learning and systems-based practice. Based on criteria developed by group exercises, the majority of written comments were rated as moderate quality (65 %); 22 % were rated as high quality and 13 % as low quality. Attendings who provided high-quality feedback rated residents significantly lower in all six of the Accreditation Council for Graduate Medical Education (ACGME) competencies (p <0.0005 for all), and had a greater range of scores. Negative comments on medical knowledge were associated with lower in-service examination scores. CONCLUSIONS: Most attending written evaluation was of moderate or low quality. Attendings who provided highquality feedback appeared to be more discriminating, providing significantly lower ratings of residents in all six ACGME core competencies, and across a greater range. Attendings' negative written comments on medical knowledge correlated with lower in-service training scores.

show abstract

“…Those students meeting certain academic and professional criteria are offered an interview. USMLE Step 1 scores correlate with how well a student will do on a residency board exam (Kay, Jackson, & Frank, 2015). A primary goal of medical educators is to promote success for students and to help them fulfill their career ambitions.…”

Section: Introductionmentioning

confidence: 99%

Investigating the Impact of Preparation Strategies on USMLE Step 1 Performance

Bonasso¹,

Lucke‐Wold²,

Reed³

et al. 2015

MedEdPublish

View full text Add to dashboard Cite

Background The USMLE Step 1 score helps differentiate applicants for competitive residency programs. Students frequently ask medical educators how to prepare for this high-stakes exam. Multiple resources exist such as books, training programs, and question banks. The purpose of this study is to provide medical educators with data on which resources correlate with higher exam scores. Methods 164 medical students at West Virginia University School of Medicine were given a survey following the completion of Step 1. The survey contained questions about resources used and exam date. De-identified data was paired with class quartile rankings and Step 1 scores. Average Step 1 scores were calculated for each resource and a student's t-test was used to compare between groups. Test dates were also broken into quintiles and average Step 1 scores were compared between groups. Results Students who used the USMLE World question bank had higher Step 1 scores (M=229, SE=1.4) compared to students who did not use this resource (M=215, SE=4.2). Commercial preparation courses were not associated with improved scores. The timing of when the exam was taken was also not associated with significant differences between scores. Conclusions Interactive study options offer the best choice to improve USMLE Step 1 scores.

show abstract

The Relationship Between Internal Medicine Residency Graduate Performance on the ABIM Certifying Examination, Yearly In-Service Training Examinations, and the USMLE Step 1 Examination

Cited by 74 publications

References 25 publications

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

Correlations Between Ratings on the Resident Annual Evaluation Summary and the Internal Medicine Milestones and Association With ABIM Certification Examination Scores Among US Internal Medicine Residents, 2013-2014

The Quality of Written Feedback by Attendings of Internal Medicine Residents

Investigating the Impact of Preparation Strategies on USMLE Step 1 Performance

Contact Info

Product

Resources

About