Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.

Abstract

This paper systematically revisits prior meta-analytic conclusions about the criterion-related validity of personnel selection procedures, and particularly the effect of range restriction corrections on those validity estimates. Corrections for range restriction in meta-analyses of predictor-criterion relationships in personnel selection contexts typically involve the use of an artifact distribution. After outlining and critiquing five approaches that have commonly been used to create and apply range restriction artifact distributions, we conclude that each has significant issues that often result in substantial overcorrection and that therefore the validity of many selection procedures for predicting job performance has been substantially overestimated. Revisiting prior meta-analytic conclusions produces revised validity estimates. Key findings are that most of the same selection procedures that ranked high in prior summaries remain high in rank, but with mean validity estimates reduced by .10-.20 points. Structured interviews emerged as the top-ranked selection procedure. We also pair validity estimates with information about mean Black-White subgroup differences per selection procedure, providing information about validity-diversity tradeoffs. We conclude that our selection procedures remain useful, but selection predictor-criterion relationships are considerably lower than previously thought.

Factors leading to interview question decisions: Introducing the Model of Interviewer Question Preferences

“…Interviews are one of the most used hiring assessments given to job applicants, and interviews are especially predictive of job performance when those interviews are structured (e.g., Huffcutt & Arthur, 1994; McDaniel et al, 1994; Sacket et al, 2021; Schmidt & Hunter, 1998). However, despite the benefits of interview structure such as asking the same questions and evaluating all applicants consistently, unstructured or semistructured interviews are still commonly used in practice (Chapman & Zweig, 2005; Dipboye, 1992;1994; Huffcutt & Arthur, 1994; Levashina et al, 2014; Lievens & De Paepe, 2004; Ryan et al, 1999; Tsai et al, 2016; Van der Zee et al, 2002).…”

“…However, despite the benefits of interview structure such as asking the same questions and evaluating all applicants consistently, unstructured or semistructured interviews are still commonly used in practice (Chapman & Zweig, 2005; Dipboye, 1992;1994; Huffcutt & Arthur, 1994; Levashina et al, 2014; Lievens & De Paepe, 2004; Ryan et al, 1999; Tsai et al, 2016; Van der Zee et al, 2002). Although unstructured interviews might allow for easier rapport building and interpersonal exchanges, the use of unstructured interviews is concerning for selection‐oriented interviews because they have lower validity than interviews that are structured (Sackett et al, 2021). Because unstructured interviews are commonly used, it is important to investigate factors that affect the quality of unstructured interviews, and specific to this study, to investigate how interviewers determine what questions to ask applicants.…”

Analysis of a situational judgment test for teamwork as a preselection tool for an assessment center: A construct‐based approach

“…Contrarily, a larger group of candidates can simultaneously perform SJTs; SJTs can also be presented digitally and are less costly than high-fidelity tools. Consequently, it would be economically reasonable to use SJTs as a preselection tool in personnel selection contexts (Lievens & De Soete, 2015;Lievens et al, 2021). Taking into account these considerations, the main focus of this paper is to analyze whether an SJT can be used as a preselection tool for an AC in a multistage selection process.…”

“…Generally, SJTs are aimed at measuring job-realistic performance by covering a broad range of job-specific situations and a variety of constructs . This often leads to heterogeneous item content and rather low internal consistencies (Lievens et al, 2021). Therefore, a common practice is to aggregate results to an overall SJT score (Christian et al, 2010).…”

“…Revisiting meta‐analytical validities in personnel selection, Sackett et al (2021) showed that SJTs are valid predictors of job performance, with operational validities that are comparable to those of ACs, based on several meta‐analytical studies. They reported an operational validity of 0.26 for SJTs and an average validity estimate across different meta‐analyses of 0.29 for ACs.…”