Reliability estimates: behavioural stations and questionnaires in medical school admissions

Abstract

CONTEXT Assessment centres used in evaluating the non-cognitive attributes of medical school candidates must generate scores that reflect as accurate a measurement as possible of these attributes. Thus far, reliability coefficients for such centres have been based on limited samples and individual administrations, without reference to the error of variance that may result from retesting, or from the existence of multiple centres designed to measure the same attributes. METHODSThe National Institute for Testing and Evaluation in Israel has developed and administered two assessment centres: MOR is used by two medical schools and one dental school, and MIRKAM by another medical school. Each centre comprises eight or nine behavioural stations, a standardised biographical questionnaire, and a judgement and decision-making questionnaire. We calculated generalisability coefficients for each centre's eight or nine stations by year, composite reliability coefficients for the overall assessment centres, test-retest correlation coefficients for repeaters, and a correlation coefficient between the centres. DISCUSSION The minimal reliability desirable for high-stakes decision making (0.80) was obtained only for 14 or 15 stations with questionnaires. Nevertheless, the values obtained are considerably higher than reliability coefficients for single interviews. The questionnaires contribute significantly to the accuracy of the measurement. These reliability measures constitute an upper threshold for measures of validity.

Evidence regarding the utility of multiple mini-interview (MMI) for selection to undergraduate health programs: A BEME systematic review: BEME Guide No. 37

“…The use of selection centres to run MMIs in Israel, however, has enabled analysis of re-applicants on a considerable scale. Gafni et al (2012) Messick (1995) suggests that each of these types of validity shouldn't be considered separately, rather that different aspects of validity contribute to an overall unified validity of the assessment. It is not sufficient to have evidence of one aspect of validity, neither is it necessary to have evidence of all; rather a judgement about the overall validity of an assessment can be made based on the accumulation of evidence across the aspects.…”

“…There is wide variation between institutions as to how many distinct factors they consider their MMI process to be assessing. Roberts et al (2009) Gafni et al 2012).…”

Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study

“…These can be grouped into ''paper and pencil'' assessments of personality traits (e.g., Adams et al 2012Adams et al , 2015Bore et al 2005a, b;Dowell et al 2011;Fukui et al 2014;James et al 2013;Lumsden et al 2005;Manuel et al 2005;Nedjat et al 2013), structured multiple interview approaches (Dore et al 2010;Eva et al, 2004aEva et al, , b, 2009Hofmeister et al 2008Hofmeister et al , 2009O'Brien et al 2011;Reiter et al 2007;Roberts et al 2008;Rosenfeld et al 2008), selection centres (Gafni et al 2012;ten Cate and Smal 2002;Ziv et al 2008;Gale et al 2010;Randall et al 2006a, b) and-the ''new kid on the block''-situational judgement tests (Christian et al 2010;Koczwara et al 2012;Lievens 2013;Lievens et al 2008;Patterson et al 2009). …”

The Predictive Validity of a Two-Step Selection Process to Medical Schools

“…MOR and MIRKAM include nine or eight behavioral stations (BS), respectively, and two customized questionnaires-the Judgment and Decision-Making Questionnaire (JDQ) and a standard Biographical Questionnaire (BQ). The questionnaires used by both systems are identical, but the nature of their behavioral stations is different [23,24]. Nonetheless, MOR and MIRKAM are designed to measure the same personal attributes.…”