The predictive validity of a situational judgement test and multiple-mini interview for entry into postgraduate training in Australia

Abstract

BackgroundEvidence for the predictive validity of situational judgement tests (SJTs) and multiple-mini interviews (MMIs) is well-established in undergraduate selection contexts, however at present there is less evidence to support the validity of their use in postgraduate settings. More research is also required to assess the extent to which SJTs and MMIs are complementary for predicting performance in practice. This study represents the first longitudinal evaluation of the complementary roles and predictive validity of an SJT and an MMI for selection for entry into postgraduate General Practice (GP) specialty training in Australia.MethodsLongitudinal data was collected from 443 GP registrars in Australia who were selected into GP training in 2010 or 2011. All 17 Regional Training Providers in Australia were asked to participate; performance data were received from 13 of these. Data was collected for participants’ end-of-training assessment performance. Outcome measures include GP registrars’ performance on the Royal Australian College of General Practitioners (RACGP) applied knowledge test, key feature problems and an objective structured clinical exam.ResultsPerformance on the SJT, MMI and the overall selection score significantly predicted all three end-of-training assessments (r = .12 to .54), indicating that both of the selection methods, as well the overall selection score, have good predictive validity. The SJT and MMI provide incremental validity over each other for two of the three end-of-training assessments.ConclusionsThe SJT and MMI were both significant positive predictors of all end-of-training assessments. Results provide evidence that they are complementary in predicting end-of-training assessment scores. This research adds to the limited literature at present regarding the predictive validity of postgraduate medical selection methods, and their comparable effectiveness when used in a single selection system. A future research agenda is proposed.

Situational judgement tests for selection: Traditional vs construct‐driven approaches

“…Evidence for the validity of SJTs in evaluating non‐academic traits for medical selection purposes has also been sought by linking the test scores to outcomes that may require a degree of interpersonal competence, such as performance in high‐fidelity simulations of clinical practice in primary care physicians, successful completion of the first stage of postgraduate medical training or performance in the first year as a general practitioner . Cross‐sectional evidence that supports the concept of SJTs measuring constructs relevant to interpersonal functioning also exists, in the form of correlations with multiple mini‐interviews . It should be noted that a separate, systematic review of the validity evidence for SJTs in medical selection is being currently undertaken.…”

Selection into Emergency Medicine specialist training: A commentary on the science of selection

“…Recent evidence suggests that SJTs have reliable psychometric properties . However, the evidence regarding its validity is not as clear . Like any exam, a successful production of any of these tests requires a significant investment of money and time to develop and pilot a validated bank of appropriate and relevant questions along with the development of the infrastructure to run such tests.…”

Non-technical attributes in paramedicine: Is situational judgement testing the solution?

“…(3) has demonstrated this through a longitudinal study where the predictive performance of academia decreased through the student's transition from study and into practice. (4) Non-technical attributes have previously been identified as an important skill set within paramedicine with international sources referring to their importance for practicing as a paramedic. (2) Despite this, there is little evidence within paramedicine on the best way to embed or assess for these attributes.…”