Exploring sociodemographic subgroup differences in multiple mini-interview (MMI) performance based on MMI station type and the implications for the predictive fairness of the Hamburg MMI

Abstract

Background Sociodemographic subgroup differences in multiple mini-interview (MMI) performance have been extensively studied within the MMI research literature, but heterogeneous findings demand a closer look at how specific aspects of MMI design (such as station type) affect these differences. So far, it has not been investigated whether sociodemographic subgroup differences imply that an MMI is biased , particularly in terms of its predictive validity. Methods Between 2010 and 2017, the University Medical Centre Hamburg-Eppendorf (UKE) tested 1438 candidates in an MMI who also provided sociodemographic data and agreed to participate in this study. Out of these, 400 candidates were admitted and underwent a first objective structured clinical examination (OSCE) after one and a half years, including one station assessing communication skills. First, we analyzed the relationship between gender, age, native language and medical family background and MMI station performance including interaction terms with MMI station type (simulation, interview, and group) in a hierarchical linear model. Second, we tested whether the prediction of OSCE overall and communication station performance in particular differed depending on sociodemographic background by adding interaction terms between MMI performance and gender, age and medical family background in a linear regression model. Results Young female candidates performed better than young male candidates both at interview and simulation stations. The gender difference was smaller (simulation) or non-significant (interview) in older candidates. There were no gender or age effects in MMI group station performance. All effects were very small, with the overall model explaining only 0.6% of the variance. MMI performance was not related to OSCE overall performance but significantly predicted OSCE communication station performance with no differences in the prediction for sociodemographic subgroups. Conclusions The Hamburg MMI is fair in its prediction of OSCE communication scores. Differences in MMI station performance for gender and age and their interaction with MMI station type can be related to the dimensions assessed at different station types and thus support the validity of the MMI. Rather than being threats to fairness, these differences could be useful for decisions relating to the design and use of an MMI.

Transition to multiple mini interview (MMI) interviewing for medical school admissions

“…Gender differences on MMI average favored female applicants slightly, which is consistent with prior studies that have evaluated MMI scores within health professional schools. Some have shown females had significantly higher MMI scores compared with males [ 15 – 18 ], while other studies have shown no effect with gender [ 19 ]. The variable results in the different studies likely represent inherent differences in the methodology of the MMIs at the individual schools; however, it is an important consideration if these ultimately affect the demographics of a medical school class.…”

The predictive validity of admission criteria for the results of clinical competency assessment with an emphasis on family medicine in the fifth year of medical education: an observational study

“…Our results are in line with the results of earlier research on the predictive validity of selection procedures employing not only pu-GPA, but also non-cognitive characteristics. In addition to the studies discussed above [ 11 – 16 ], the multiple mini-interview (MMI)-scores of medical students were found to predict the communication skills assessed in an objective structured clinical examination after one and a half years of medical education [ 28 ]. MMIs are claimed to be a valid, cost-effective alternative to conventional admission interviews and suitable for the assessment of interpersonal skills (i.e., empathy and communication skills).…”

Are different station formats assessing different dimensions in multiple mini-interviews? Findings from the Canadian integrated French multiple mini-interviews

“…The study by Mirghani et al [ 31 ] used EFA and could adequately differentiate stations that were intended to measure visuomotor skills and soft skills, where six stations involving mainly discussion and reflection loaded in one factor, and four stations involving manual dexterity or motor tasks loaded in different factors. Considering “soft skills” stations, a German study about MMI recently hypothesized that role-play and discussion stations were assessing different constructs, thus creating small but perceptible subgroup differences [ 32 ]. No factor analysis was performed, however, in this context.…”

“…For instance, a recent German study found that some subgroup differences (e.g. male/female) would vary according to station format, suggesting that the constructs assessed in these different formats were different [ 32 ]. The authors suggested that role-play (or “simulation”) stations required more affective empathy, as opposed to discussion or interview stations, that perhaps required more cognitive skills related to perspective taking or reflection [ 32 ].…”