Retest Effects in Operational Selection Settings: Development and Test of a Framework

Lievens, Filip; Buyse, Tine; Sackett, Paul R.

doi:10.1111/j.1744-6570.2005.00713.x

Cited by 105 publications

(166 citation statements)

References 39 publications

Supporting

Mentioning

149

Contrasting

Unclassified

Order By: Relevance

“…Again, research is scarce. Lievens et al (2005b) demonstrated that retest effects of SJTs were not larger than those of more traditional tests (cognitive ability). An important moderator seems to be the approach of constructing alternate SJT forms (Clause et al, 1998;Oswald et al, 2005).…”

Section: Fakabilitymentioning

confidence: 99%

Situational judgment tests: a review of recent research

2008

Self Cite

View full text Add to dashboard Cite

Purpose -The purpose of this paper is to give an empirically-based review of the strengths and weaknesses of situational judgment tests (SJTs). Design/methodology/approach -The features, history, and development of SJTs are discussed. Next, a computerized search (from 1990-2007) is conducted to retrieve empirical studies related to SJTs. The effectiveness of SJTs is discussed in terms of reliability, criterion-related validity, incremental validity, construct-related validity, utility, adverse impact, applicant perceptions, fakability, and susceptibility to practice and coaching effects. Findings -Strengths of SJTs are that they show criterion-related validity and incremental validity above cognitive ability and personality tests. SJTs have also less adverse impact towards minorities (especially if the cognitive loading of the SJT is low). Furthermore, applicant reactions towards SJTs are positive and SJTs enable to test large applicant groups at once (through the Internet). In terms of weaknesses, SJTs might be prone to faking, practice, and coaching effects. There is also debate about what constructs are measured by SJTs. Research limitations/implications -Five avenues for future research are discussed: construct-related validity of SJTs, utility of SJTs vis-à -vis other predictors, impact of SJT features on validity and adverse impact, examination of alternative stimulus and response formats, and cross-cultural transportability of SJTs. Practical implications -Practitioners receive evidence-based information about the features, development, and strengths and weaknesses of SJTs. Originality/value -Apart from the USA, SJTs have not made strong inroads in selection practice in Europe and other parts of the world. This evidence-based paper might highlight the value of SJTs.

show abstract

Section: Fakabilitymentioning

confidence: 99%

Situational judgment tests: a review of recent research

2008

Self Cite

View full text Add to dashboard Cite

show abstract

“…Kline (2000) recommends that parallel forms reliability should be above r=.70. On balance, the research evidence shows good levels of parallel forms reliability for SJTs, for example: r=.76 (Chan & Schmitt 2002) and r=.66 (Lievens et al 2005b). …”

Section: How Reliable Are Sjts?mentioning

confidence: 99%

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

Patterson¹,

Zibarras²,

Ashworth³

2015

Medical Teacher

169

228

View full text Add to dashboard Cite

This is the accepted version of the paper.This version of the publication may differ from the final published version. Vicki specialises in the fields of assessment and selection, evaluation and organisational development, delivering bespoke solutions to public and private sector clients in the UK and internationally. Permanent repository link Acknowledgements:We especially would like to thank Anna Rosselli and Fran Cousans for their support in preparing the final manuscript. reactions to a number of hypothetical role-relevant scenarios, which reflect situations candidates are likely to encounter in the target role. These scenarios are based on a detailed analysis of the role and should be developed in collaboration with subject matter experts, in order to accurately assess the key attributes that are associated with competent performance.From a theoretical perspective, SJTs are believed to measure prosocial Implicit Trait Policies (ITPs), which are shaped by socialisation processes that teach the utility of expressing certain traits in different settings such as agreeable expressions (e.g. helping others in need), or disagreeable actions (e.g. advancing ones own interest at others expense).

show abstract

“…And improvement will vary with the type of test. Lievens et al (2005) reported retest effect sizes of .30, .40, and .46 for a situational judgment, knowledge, and cognitive ability test, respectively. Hypothesis 3: Tests at lower ranks would correlate with performance on tests at higher ranks.…”

mentioning

confidence: 94%

An Exploratory Study of Two Decades of Promotional Testing in a Metropolitan Police Department

Topp

2010

J Police Crim Psych

View full text Add to dashboard Cite

This study explored the career progression of police officers in a police department by examining the relation among 13 promotional tests at four ranks administered to 1,960 candidates over two decades. It was hypothesized that performance on promotional tests would be relatively stable during a law enforcement career, consistent with research demonstrating consistency in general mental ability. Overall, 28% of subjects retested one or more times and their retest scores on non-equivalent forms correlated about .40 with their initial scores. On average, subjects increased their score by one-fifth of a standard deviation by retesting. With some exceptions, candidate performance on promotional tests significantly correlated with tests for the immediately superior rank as well as tests for higher ranks as much as 15 years later. However, using multiple regression to predict performance on higher rank tests by combining lower rank tests was unsuccessful. Implications for the management of police promotional testing programs are discussed as well as suggestions for future research.In 1989, a promotional examination for sergeant was conducted, resulting in a rank-order list with ten pages and 747 names. Of interest was the first page consisting of candidates ranked 1 through 78. Many of these individuals later promoted to leadership roles, including 15 lieutenants, 11 captains, four deputy chiefs, two assistant chiefs, and a chief. In essence, a substantial portion of a department's executive leadership in 2010 appeared at the top of an eligible list from a test conducted 20 years before.This outcome is familiar to HR analysts and managers working in civil service testing. Most of us have observed the advancement of talented candidates and the rank and file often predict who will appear at the top of eligible lists from which promotions are made. Some coworkers attribute the success to "good test taking" or "book knowledge" while others acknowledge the intense motivation of these rising stars who may study for a year or more. Not surprisingly, a recent study found selfreported motivation to correlate with test performance (McCarthy et al. 2009).Most law enforcement agencies have a well-defined rank hierarchy and therefore predefined paths for career progression. And to varying degrees, most conduct promotional examinations which result in public lists, often with ranks and scores. These eligible lists provide a rich historical record of employee career progression. Despite the availability of volumes of raw data, no one appears to have explored the relationship among sequential promotional tests administered over an entire career. That was the purpose of this study. Of particular interest was the consistency of candidate performance on tests at different ranks and the prevalence and outcome of retesting.To begin, it is clear that assessment centers (AC) are valid predictors of performance during a career. In a metaanalysis integrating the results of 34 validation studies, Arthur et al. (2003) regressed assessor ratings ...

show abstract

Retest Effects in Operational Selection Settings: Development and Test of a Framework

Cited by 105 publications

References 39 publications

Situational judgment tests: a review of recent research

Situational judgment tests: a review of recent research

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

An Exploratory Study of Two Decades of Promotional Testing in a Metropolitan Police Department

Contact Info

Product

Resources

About