Validating MMI scores: are we measuring multiple attributes?

Abstract

The multiple mini-interview (MMI) used in health professional schools' admission processes is reported to assess multiple non-cognitive constructs such as ethical reasoning, oral communication, or problem evaluation. Though validation studies have been performed with total MMI scores, there is a paucity of information regarding how well MMI scores differentiate the constructs being measured, the relationship between MMI scores (construct or total) and personality characteristics, and how well MMI scores (construct or total) predict future performance in practice. Results from these studies could assist with MMI station development, rater training, score interpretation, and resource allocation. The purpose of this study was to investigate the validity of MMI construct scores (oral communication and problem evaluation), and their relationship to personality measures (emotionality and extraversion) and specific scores from standardized clinical communications interviews (building the relationship and explaining and planning). Confirmatory factor analysis results support a two factor MMI model, however the correlation between these factors was .87. Oral communication MMI scores significantly correlated with extraversion (r c = .25, p < .05), but MMI scores were not related to emotionality. Scores for building a relationship were significantly related to MMI oral communication scores, (r c = .46, p < .001) and problem evaluation scores (r c = .43, p < .001); scores for explaining and planning were significantly related to MMI problem evaluation scores (r c = .36, p < .01). The results provide validity evidence for assessing multiple non-cognitive attributes during the MMI process and reinforce the importance of developing MMI stations and scoring rubrics for attributes identified as important for future success in school and practice.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

“…Within these analyses of multiple correlations, Kulasegaram et al (2010) and Griffin & Wilson (2012) applied Bonferroni corrections, whereas Jerant et al (2012) and Oliver et al (2014) did not appear to, so the associations reported in the latter two studies are more likely to have suffered type 1 errors. Yen et al (2011) reported no correlation between a validated self-report measure of emotional intelligence, defined as 'a type of social intelligence that involves the ability to monitor one's own thinking and actions' (Salovey & Mayer 1990), and total MMI score for 196 applicants for admission to the health sciences programme at the Michener Institute for Applied Health Sciences.…”

“…Oliver et al (2014) investigated the associations between MMI and extroversion and emotionality, and reported extroversion was associated with higher MMI score (r=0.22, P<0.05) but emotionality was not (r=-0.01, n/s).…”

“…Foley & Hijazi (2013) reported a correlation between the MMI scores of students at Aberdeen dental school (n=75) and end of year assessments consisting of short answer questions and OSCES (r=0.18, P=0.001). Oliver et al (2014) demonstrated that the MMI used for selection of students to University of Calgary Veterinary School (n=60) was correlated with students' performance at building the practitioner-patient relationship (r=0.46, P<0.001, corrected for range restriction) and explaining and planning (r=0.28, P<0.05, corrected for range restriction) during a standardised clinical communication interview eight months after their MMI (see Tables 5 & 6, found in the Supplementary Materials section).…”

Interviewers' Experiences with Two Multiple Mini-Interview Scoring Methods Used for Admission to a Master of Physical Therapy Programme

“…19–21 According to the literature, the categories selected in this case—professionalism, critical thinking, and communication—are appropriate for the PT profession. 22–28 The findings of this study confirmed that the CBS categories are valid; however, they are also overlapping and interrelated, especially communication and professionalism, and this question merits further investigation.…”

Have admissions committees considered all the evidence?

“…Shouldn't there be convergent and discriminant evidence demonstrating the utility of the attributes being measured? Indeed Oliver et al (Oliver et al 2014) tried to ''draw a line in the sand'' and suggest that if we are measuring multiple attributes using an MMI, then we should attempt to show some evidence of differential prediction form the various sub-components. While some evidence of convergent and discriminant validity was provided, the correlation between communication and problem evaluation was large (.87).…”