Practice effects in medical school entrance testing with the undergraduate medicine and health sciences admission test (UMAT)

Abstract

BackgroundThe UMAT is widely used for selection into undergraduate medical and dental courses in Australia and New Zealand (NZ). It tests aptitudes thought to be especially relevant to medical studies and consists of 3 sections – logical reasoning and problem solving (UMAT-1), understanding people (UMAT-2) and non-verbal reasoning (UMAT-3). A substantial proportion of all candidates re-sit the UMAT. Re-sitting raises the issue as to what might be the precise magnitude and determinants of any practice effects on the UMAT and their implications for equity in subsequent selection processes.MethodsBetween 2000 and 2012, 158,909 UMAT assessments were completed. From these, 135,833 cases were identified where a candidate had sat once or more during that period with 117,505 cases (86.5%) having sat once, 14,739 having sat twice (10.9%), 2,752 thrice (2%) and 837, 4 or more times (0.6%). Subsequent analyses determined predictors of multiple re-sits as well as the magnitude and socio-demographic determinants of any practice effects.ResultsIncreased likelihood of re-sitting the UMAT twice or more was predicted by being male, of younger age, being from a non-English language speaking background and being from NZ and for Australian candidates, being urban rather than rurally based. For those who sat at least twice, the total UMAT score between a first and second attempt improved by 10.7 ± 0.2 percentiles, UMAT-1 by 8.3 ± 0.2 percentiles, UMAT-2 by 8.3 ± 0.2 percentiles and UMAT-3 by 7.7 ± 0.2 percentiles. An increase in total UMAT percentile score on re-testing was predicted by a lower initial score and being a candidate from NZ rather than from Australia while a decrease was related to increased length of time since initially sitting the test, older age and non-English language background.ConclusionsRe-sitting the UMAT augments performance in each of its components together with the total UMAT percentile score. Whether this increase represents just an improvement in performance or an improvement in understanding of the variables and therefore competence needs to be further defined. If only the former, then practice effects may be introducing inequity in student selection for medical or dental schools in Australia or NZ.

The STEP model: Characterizing simultaneous time effects on practice for flight simulator performance among middle-aged and older pilots.

“…Furthermore, the number of attempts until successful admittance into the training program was positively associated with performance in the training program (Hausknecht, et al, 2002). Similarly, consistent practice effects were found among young adults who took the annual Undergraduate Medicine and Health Sciences Admission Test (UMAT) at least twice (Puddey, et al, 2014). The vast majority scored significantly better the second time than initially, and better on the third time than second time.…”

“…One study examined practice effects among aviation students; however, practice effects were assessed in paper and pencil tests rather than actual flight performance, and the range of age and flight experience among the students was very limited (Momen, 2009). The other studies investigated practice effects on admissions tests used to select candidates for entry level job positions or for the premedical track at the undergraduate level (Hausknecht, Trevor, & Farr, 2002; Puddey, Mercer, Andrich, & Styles, 2014). Hausknecht et al (2002) examined whether practice effects were evident among candidates for an entry level position in a law enforcement agency.…”

“…Yet, the above studies examining real world tasks do not illuminate whether age interacts with practice and interval effects. Participants were relatively young, with a mean age ranging from 18 years (Puddey, et al, 2014) to 32 years(Hausknecht, et al, 2002) . Therefore it is unclear as to whether these results would have been found with an older sample.…”

Selecting top candidates for medical school selection interviews- a non-compensatory approach

“…The UMAT was devised by the Australian Council for Educational Research, and consists of three sections: logical reasoning and problem solving; understanding people; and non-verbal reasoning. Most undergraduate medical programs in Australia use ATAR and UMAT outcomes to select a smaller group of applicants for additional selection processes, which may include interviews, psychological tests, or both [2,[6][7][8][9]. Most of the literature focusing on the efficacy of selection to medical programs focuses on the final stage of the selection process; that is selection from the pool of applicants who were successful in being offered an interview [2,[9][10][11].…”

Retest effects in medical selection interviews

“…However, researchers have focused predominantly on the effects of re‐sitting tests of cognitive ability (for examples in medical selection, see Andrich et al . and Puddey et al …”