Situational Judgment Tests: Constructs Assessed and a Meta-Analysis of Their Criterion-Related Validities

Abstract

Situational judgment tests (SJTs) are a measurement method that may be designed to assess a variety of constructs. Nevertheless, many studies fail to report the constructs measured by the situational judgment tests in the extant literature. Consequently, a construct‐level focus in the situational judgment test literature is lacking, and researchers and practitioners know little about the specific constructs typically measured. Our objective was to extend the efforts of previous researchers (e.g., McDaniel, Hartman, Whetzel, & Grubb, 2007; McDaniel & Ngyuen, 2001; Schmitt & Chan, 2006) by highlighting the need for a construct focus in situational judgment test research. We identified and classified the construct domains assessed by situational judgment tests in the literature into a content‐based typology. We then conducted a meta‐analysis to determine the criterion‐related validity of each construct domain and to test for moderators. We found that situational judgment tests most often assess leadership and interpersonal skills and those situational judgment tests measuring teamwork skills and leadership have relatively high validities for overall job performance. Although based on a small number of studies, we found evidence that (a) matching the predictor constructs with criterion facets improved criterion‐related validity; and (b) video‐based situational judgment tests tended to have stronger criterion‐related validity than pencil‐and‐paper situational judgment tests, holding constructs constant. Implications for practice and research are discussed.

How “situational” is judgment in situational judgment tests?

“…Three different SJTs were administered (either with or without situation descriptions) to cover the three broad content domains of SJTs (cf. Christian et al, 2010): (a) job knowledge and skills, (b) applied social skills, and (c) basic personality tendencies. For each of the domains, a representative SJT was chosen for inclusion in the current study.…”

“…The Pilot Judgment Inventory was developed to capture general aviation pilots' decision making in realistic flight scenarios. Therefore, Christian et al (2010) coded this SJT into the domain of job knowledge and skills. The 39 items of this test confront participants with short descriptions of flight scenarios, which are followed by four response alternatives.…”

“…That is, we posited that the content domain captured by the SJT might be a key moderator of whether SJT performance is determined by context-independent knowledge versus contextdependent knowledge. SJTs are measurement methods that have the potential to assess a variety of constructs (Christian et al, 2010). As already mentioned, SJTs might therefore be placed on a continuum, with some SJTs measuring rather context-independent knowledge and others being situated on the context-dependent knowledge side.…”

“…As already mentioned, SJTs might therefore be placed on a continuum, with some SJTs measuring rather context-independent knowledge and others being situated on the context-dependent knowledge side. More generally, Christian et al (2010) classified the various constructs assessed by SJTs in a taxonomy consisting of three broad categories: job knowledge and skills (e.g., SJTs assessing pilot judgment or knowledge relevant for firefighters), applied social skills (e.g., teamwork or leadership SJTs), and basic personality tendencies (e.g., SJTs gauging Big Five personality dimensions or facets such as integrity).…”

“…The most frequently employed SJT format consists of written job-related situations that are presented with multiple-choice response options (McDaniel, Hartman, Whetzel, & Grubb, 2007;Motowidlo, Dunnette, & Carter, 1990). As evidenced by a recent meta-analysis (Christian, Edwards, & Bradley, 2010), SJT situations most often pertain to the construct domain of applied social skills.…”