Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school

Abstract

Objective: To assess the practice effects from coaching on the Undergraduate Medicine and Health Sciences Admission Test (UMAT), and the effect of both coaching and repeat testing on the Multiple Mini Interview (MMI). Design, setting and participants: Observational study based on a self‐report survey of a cohort of 287 applicants for entry in 2008 to the new School of Medicine at the University of Western Sydney. Participants were asked about whether they had attended UMAT coaching or previous medical school interviews, and about their perceptions of the relative value of UMAT coaching, attending other interviews or having a “practice run” with an MMI question. UMAT and MMI results for participants were compared with respect to earlier attempts at the test, the degree of similarity between questions from one year to the next, and prior coaching. Main outcome measures: Effect of coaching on UMAT and MMI scores; effect of repeat testing on MMI scores; candidates’ perceptions of the usefulness of coaching, previous interview experience and a practice run on the MMI. Results: 51.4% of interviewees had attended coaching. Coached candidates had slightly higher UMAT scores on one of three sections of the test (non‐verbal reasoning), but this difference was not significant after controlling for Universities Admission Index, sex and age. Coaching was ineffective in improving MMI scores, with coached candidates actually having a significantly lower score on one of the nine interview tasks (“stations”). Candidates who repeated the MMI in 2007 (having been unsuccessful at their 2006 entry attempt) did not improve their score on stations that had new content, but showed a small increase in scores on stations that were either the same as or similar to previous stations. Conclusion: A substantial number of Australian medical school applicants attend coaching before undertaking entry selection tests, but our study shows that coaching does not assist and may even hinder their performance on an MMI. Nevertheless, as practice on similar MMI tasks does improve scores, tasks should be rotated each year. Further research is required on the predictive validity of the UMAT, given that coaching appeared to have a small positive effect on the non‐verbal reasoning component of the test.

Practice effects in medical school entrance testing with the undergraduate medicine and health sciences admission test (UMAT)

“…We have not been able to measure such potential influences in this study. However, Griffin et al [15], in a study of 287 applicants to the School of Medicine at the University of Western Sydney, reported that in the 51.4% of the cohort who had prior UMAT coaching, there was no coaching effect on UMAT-1 and UMAT-2. However, there was evidence of a small coaching effect on UMAT-3 (approximately 3.8% increase in score).…”

The impact of preparatory activities on medical school selection outcomes: a cross-sectional survey of applicants to the university of Adelaide medical school in 2007

“…Their results correspond with the difficulties outlined in our study such as cost and perceived level of difficulty but only included those who had been successful in the medical school selection process. One small Australian study of 287 applicants assessed the impact of coaching on the UMAT and coaching and repeat testing on the MMI [21]. They found that coaching had a small effect on the parts of the UMAT test but was ineffective in improving the scores of the MMI.…”

“…There is little research regarding the lengths applicants go to in order to maximise their chance of receiving an offer or the difficulties they encounter during this process [21-23]. Moreover, there is little evidence on of the impact of the preparatory activities undertaken or difficulties encountered on the outcome [24].…”

Views of doctors of varying disciplines on HPAT-Ireland as a selection tool for medicine

“…Participants were also concerned that coaching could improve performance presenting an additional barrier to applicants from lower socio-economic groups. There is evidence that coaching improves performance on Section 3 of both HPAT-Ireland (O'Flynn et al 2012) and UMAT (Griffin et al 2008(Griffin et al , 2012. Hence there is a strong argument for removing this section entirely from the paper.…”