Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses

Abstract

The multiple mini-interview (MMI) has become an increasingly popular admissions method for selecting prospective students into professional programs (e.g., medical school). The MMI uses a series of short, labour intensive simulation stations and scenario interviews to more effectively assess applicants' non-cognitive qualities such as empathy, critical thinking, integrity, and communication. MMI data from 455 medical school applicants were analyzed using: (1) Generalizability Theory to estimate the generalizability of the MMI and identify sources of error; and (2) the Many-Facet Rasch Model, to identify misfitting examinees, items and raters. Consistent with previous research, our results support the reliability of MMI process. However, it appears that the non-cognitive qualities are not being measured as unique constructs across stations.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

“…Sebok et al (2013) found inter-rater reliabilities of 0.41 to 0.69 for stations scored by faculty members and students.…”

“…Generalizability analyses have repeatedly indicated that MMIs with greater numbers of stations will have greater reliability (Eva et al 2004c;Roberts et al 2008;Sebok et al 2013). …”

“…They also found a non-significant difference between the mean scores assigned by faculty members (4. Sebok et al 2013). Griffin & Wilson (2010) observed that when they changed from information-based rater training to skills-based rater training that involved rating simulated interviewees the proportion of variance in their MMI scores attributable to differences between raters was reduced from 20.2% to 7.0% (t=4.42, P=0.004).…”

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

“…We also examined the reliability of separation for each facet. Notably, reliability of separation indices operate differently to conventional reliability parameters; in particular the reliability of separation for examiner cohorts describes the degree to which examiner cohorts differ from one another (Sebok, et al, 2014). Consequently it is more akin to a "Hawk-Dove reliability" (Streiner & Norman 2008, p182) than a conventional inter-rater reliability.…”

A Comparison of the Results of Many-Facet Rasch Analyses Based on Crossed and Judge Pair Designs

“…This mathematical model makes it possible to estimate the ability level and item difficulty simultaneously (Brown, O'Gorman, & Du, 1996). As the equation shows, the simple Rasch model was created for dichotomous items that can be rated as either correct or incorrect (Sebok, Luu, & Klinger, 2013). Andrich (1978) extended the use of the simple Rasch model to Likert-type data, which produced the rating scale model, and Masters (1982) created the partial credit model, which was an extension of the research by Andrich (1978).…”