Evaluating cognitive ability, knowledge tests and situational judgement tests for postgraduate selection

Abstract

OBJECTIVES This study aimed to evaluate the validity and utility of and candidate reactions towards cognitive ability tests, and current selection methods, including a clinical problemsolving test (CPST) and a situational judgement test (SJT), for postgraduate selection.METHODS This was an exploratory, longitudinal study to evaluate the validities of two cognitive ability tests (measuring general intelligence) compared with current selection tests, including a CPST and an SJT, in predicting performance at a subsequent selection centre (SC). Candidate reactions were evaluated immediately after test administration to examine face validity. Data were collected from candidates applying for entry into training in UK general practice (GP) during the 2009 recruitment process. Participants were junior doctors (n = 260). The mean age of participants was 30.9 years and 53.1% were female. Outcome measures were participants' scores on three job simulation exercises at the SC.RESULTS Findings indicate that all tests measure overlapping constructs. Both the CPST and SJT independently predicted more variance than the cognitive ability test measuring non-verbal mental ability. The other cognitive ability test (measuring verbal, numerical and diagrammatic reasoning) had a predictive value similar to that of the CPST and added significant incremental validity in predicting performance on job simulations in an SC. The best single predictor of performance at the SC was the SJT. Candidate reactions were more positive towards the CPST and SJT than the cognitive ability tests.CONCLUSIONS In terms of operational validity and candidate acceptance, the combination of the current CPST and SJT proved to be the most effective administration of tests in predicting selection outcomes. In terms of construct validity, the SJT measures procedural knowledge in addition to aspects of declarative knowledge and fluid abilities and is the best single predictor of performance in the SC. Further research should consider the validity of the tests in this study in predicting subsequent performance in training.change management

Do personality traits assessed on medical school admission predict exit performance? A UK-wide longitudinal cohort study

“…Although this seems a conceptual gap to us, there is some evidence that SJTs predict performance in one medical training context, that of UK general practice training (Lievens and Patterson 2011) (and the wider literature also suggests that the way an individual responds to an SJT question does predict actual behaviour and performance once in a role (e.g., McDaniel et al 2001)). Validity studies have also shown that SJTs add incremental validity when used in combination with other predictors of job performance such as structured interviews, tests of IQ and personality questionnaires (O'Connell et al 2007;McDaniel et al 2007;Koczwara et al 2012). While the focus of this paper is not to analyse the conceptual and theoretical frameworks of personality tools, it is essential that these are critically examined in order to develop, evaluate and compare medical selection tools and how these are used in admissions/selection processes.…”

“…These can be grouped into ''paper and pencil'' assessments of personality traits (e.g., Adams et al 2012Adams et al , 2015Bore et al 2005a, b;Dowell et al 2011;Fukui et al 2014;James et al 2013;Lumsden et al 2005;Manuel et al 2005;Nedjat et al 2013), structured multiple interview approaches (Dore et al 2010;Eva et al, 2004aEva et al, , b, 2009Hofmeister et al 2008Hofmeister et al , 2009O'Brien et al 2011;Reiter et al 2007;Roberts et al 2008;Rosenfeld et al 2008), selection centres (Gafni et al 2012;ten Cate and Smal 2002;Ziv et al 2008;Gale et al 2010;Randall et al 2006a, b) and-the ''new kid on the block''-situational judgement tests (Christian et al 2010;Koczwara et al 2012;Lievens 2013;Lievens et al 2008;Patterson et al 2009). …”

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

“…Previous studies on the use of SJTs for selection into medical school have shown that these beneficial characteristics of SJTs also apply in a medical school context (Koczwara et al 2012;Lievens 2013;Lievens et al 2005;Lievens and Sackett 2012;Patterson et al 2009Patterson et al , 2011Patterson et al , 2015.…”

The Distinguishing Personality Traits of Intelligent Fresh Graduates. A Comparison Study of High and Low GMA Individuals

“…While the dominant traits of intelligent females are innovation and responsibility, the dominant traits of males are risk taking and innovation. Such results may help in selecting the right candidates as many studies indicated that the most predictive variables of future performance of recent college graduates are personality and mental ability (Koczwara, 2012;Sjoberg et al, 2012;Marcus et al, 2009;Cook, 2009;Schmidt, Shaffer, & OH, 2008;Rode et al, 2008;Schmidt & Hunter, 2004;Gottfredson, 1997).…”