Is it a match? a novel method of evaluating medical school success

Abstract

Background: Medical education program evaluation allows for curricular improvements to both Undergraduate (UME) and Graduate Medical Education (GME). UME programs are left with little more than match rates and self-report to evaluate success of graduates in The Match.Objective: This manuscript shares a novel method of program evaluation through a systematic assessment of Match outcomes.Design: Surveys were developed and distributed to Program Training Directors (PTDs) at our institution to classify residency programs into which our UME graduates matched using an ordinal response scale and open-ended responses. Outcomes-based measures for UME graduates were collected and analyzed. The relationship between PTD survey data and UME graduates’ outcomes were explored. Open-ended response data were qualitatively analyzed using iterative cycles of coding and identifying themes.Results: The PTD survey response rate was 100%. 71% of our graduates matched to programs ranked as ‘elite’ (36%) or ‘top’ (35%) tier. The mean total number of ‘Honors’ grades achieved by UME graduates was 2.6. Data showed that graduates entering elite and top GME programs did not consistently earn Honors in their associated clerkships. A positive correlation was identified between USMLE Step 1 score, number of honors, and residency program rankings for a majority of the programs. Qualitative analysis identified research, faculty, and clinical exposure as necessary characteristics of ‘elite’ programs:. Factors considered by PTDs in the rating of programs included reputation, faculty, research, national presence and quality of graduates.Conclusions: This study describes a novel outcomes-based method of evaluating the success of UME programs. Results provided useful feedback about the quality of our UME program and its ability to produce graduates who match in highly-regarded GME programs. The findings from this study can benefit Clerkship Directors, Student Affairs and Curriculam Deans, and residency PTDs as they help students determine their competitiveness forspecialties and specific residency programs.

Relationship between students’ perceptions of the adequacy of M1 and M2 curricula and their performance on USMLE step 1 examination

“…Although different comprehensive evaluation models [1–3] are available for curriculum evaluation, medical schools are constantly looking for innovative ways to evaluate their curricula, especially if the evaluation method responds efficiently in making recommendations and changes. For instance, Chang et al [4] utilized medical student graduates’ ability to match in highly regarded residency programs as a measure of undergraduate medical education program quality. They also reported that key factors in students finding “elite” residency programs were “clerkship grades and Step 1 scores” [4].…”

How well did our students match? A peer-validated quantitative assessment of medical school match success: the match quality score

“…While physicians are by nature quantitative [13], in one 2001 survey-driven study of internal medicine residency program directors, the determination of quality of the match was a subjective assessment of 'matched much better/somewhat better/ same/somewhat worse/much worse than previous [years]' [14]. Our study differs from another prior study [8] in the use of hypothetical match lists to determine the impact of variables on the interpretation of match success. In addition, our survey methodology included assessments made by student affairs experts from multiple institutions as opposed to a single center.…”

“…The 18 included specialties in the NRMP match plus specialties from the other matches (urology, ophthalmology) were grouped based upon their applicants/available positions ratios, with three possible outcomes: matching into a highly competitive specialty, matching into a moderately competitive specialty, and matching into a less competitive specialty (see Table 1) [9][10][11]. Reputation of the institution, adapted from methods described in other studies [8,12], was binary: whether or not the matched program was among the top 10 in terms of NIH funding. Each factitious mini-match list fulfilled one condition: Matching Status (four possible outcomes) x Competitiveness (three possible outcomes) × Reputation (two possible outcomes).…”

“…Match results from undergraduate medical institutions, when reported at all, are comprised of qualitative assessments of the match performance that are subjective and prone to bias, making year-to-year comparisons between matches difficult, if not impossible. An earlier study sought to assess the quality of match success by surveying program directors at the home intuition and was able to provide a subjective description their students' placement at various residency locations [8]. However, such a study is subjective and less useful for continuous quality improvement.…”

Student academic performance factors affecting matching into first-choice residency and competitive specialties

“…However, students have no incentive or reward to falsify this data. The Electronic Residency Application System (ERAS) which holds the information on first, second or third choice match, does not officially release this information, thus all medical schools in the United States must rely on self-reported data [27]. Additionally, in the present study, data on the specific specialty and residency program that students matched into was also self-reported.…”