Situational judgement test validity for selection: A systematic review and meta‐analysis

Abstract

Context Situational judgement tests (SJTs) are widely used to evaluate ‘non‐academic’ abilities in medical applicants. However, there is a lack of understanding of how their predictive validity may vary across contexts. We conducted a systematic review and meta‐analysis to synthesise existing evidence relating to the validity of such tools for predicting outcomes relevant to interpersonal workplace performance. Methods Searches were conducted in relevant databases to June 2019. Study quality and risk of bias were assessed using the Quality In Prognosis Studies (QUIPS) tool. Results were pooled using random effects meta‐analysis and meta‐regressions. Results Initially, 470 articles were identified, 218 title or abstracts were reviewed, and 44 full text articles were assessed with 30 studies meeting the final inclusion criteria and were judged, overall, to be at moderate risk of bias. Of these, 26 reported correlation coefficients relating to validity, with a pooled estimate of 0.32 (95% confidence interval 0.26 to 0.39, P < .0001). Considerable heterogeneity was observed (I2 = 96.5%) with the largest validity coefficients tending to be observed for postgraduate, rather than undergraduate, selection studies (β = 0.23, 0.11 to 0.36, P < .001). The correction of validity coefficients for attenuation was also independently associated with larger effects (β = 0.13, 0.03 to 0.23, P = .01). No significant associations with test medium (video vs text format), cross‐sectional study design, or period of assessment (one‐off vs longer‐term) were observed. Where reported, the scores generally demonstrated incremental predictive validity, over and above tests of knowledge and cognitive ability. Conclusions The use of SJTs in medical selection is supported by the evidence. The observed trend relating to training stage requires investigation. Further research should focus on developing robust criterion‐relevant outcome measures that, ideally, capture interpersonal aspects of typical workplace performance. This will facilitate additional work identifying the optimal place of SJTs within particular selection contexts and further enhancing their effectiveness.

The hidden curriculum and limitations of situational judgement tests for selection

“…In our view, the most significant limitation of SJTs, as highlighted in an earlier systematic review 12 and the current paper, 11 is the lack of strong outcome measures in evaluations of their predictive validity. The primary studies in Webster et al’s review 11 often used artificial, maximal performance one‐off assessments, such as interviews, simulations or objective structured clinical examinations (OSCEs). Others relied upon evaluations from tutors and clinical supervisors or grades for courses with interpersonal components.…”

“…In this issue, Webster et al report a rigorous systematic review and meta‐analysis of SJTs’ predictive validity for selection for medical education 11 . The authors find support for SJTs in general and greater validity for postgraduate than undergraduate medical education in predicting outcomes related to interpersonal performance 11 .…”

“…In this issue, Webster et al report a rigorous systematic review and meta‐analysis of SJTs’ predictive validity for selection for medical education 11 . The authors find support for SJTs in general and greater validity for postgraduate than undergraduate medical education in predicting outcomes related to interpersonal performance 11 . Although the pooled predictor‐outcome correlation estimate of 0.32 (and incremental validity of 5.0%–10.0% of additional variance in the outcome accounted for) perhaps seems initially unimpressive, the authors argue a relatively homogenous population, restriction of range and similar correlations for cognitive assessments in medical school bolster the strength of their findings 11 …”

“…As high‐stakes examinations, however, they are an important way for medical schools to communicate their values and create a hidden curriculum. In making the 160‐minute United Kingdom (UK) Foundation Programme SJT carry equal weight to academic performance from 5‐6 years of medical school, 11 the message is clear that non‐academic attributes are important. This SJT, similar to most medical SJTs, explicitly describes uncontroversial attributes being assessed and thus valued, such as commitment to professionalism and effective teamwork abilities 6 .…”

Different Approaches to Selection of Surgical Trainees in the European Union&nbsp;

“…Situational judgment tests (SJTs) have been found to re ect job performance and tend to discriminate accurately between average and low performing participants (44,45). Situational judgment tests could be used as an initial screening method before conducting more costly and time-consuming assessments, especially in situations with a low applicant to vacancy ratio (19,20,46).…”