A Meta-Analytical Multilevel Reliability Generalization of 					Situational Judgment Tests (SJTs)

Kasten, Nadine; Freund, Philipp Alexander

doi:10.1027/1015-5759/a000250

Cited by 32 publications

(36 citation statements)

References 31 publications

(46 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…However, we sought to obtain a total of 240 participants following general guidelines for logistic regression analysis (Peduzzi, Concato, Kemper, Holford, & Feinstein, 1996). 4 Meta-analyses have revealed that SJTs' internal consistencies are generally low to moderate (Catano, Brochu, & Lamerson, 2012;Kasten & Freund, 2016). 5 For all studies, we report categorical (Green & Yang, 2008) for SJTs.…”

Section: Endnotesmentioning

confidence: 99%

See 1 more Smart Citation

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

Freudenstein

Schäpers

Roemer

et al. 2020

Personnel Psychology

View full text Add to dashboard Cite

Recent research challenges the importance of situation descriptions for situational judgment test (SJT) performance. This study contributes to resolving the ongoing debate on whether SJTs are situational measures, by incorporating findings on person × situation interactions into SJT research. Specifically, across three studies (N Total = 1,239), we first tested whether situation construal (i.e., the individual perception of situations in SJTs) predicts responses to SJT items. Second, we assessed whether the relevance of situation construal for SJT performance depends on test elements (i.e., situation descriptions and response options) and item features (i.e., description-dependent vs. description-independent SJT items). Lastly, we determined whether situation construal has incremental validity for job-related criteria over and above SJT performance. The results showed that, for most SJT items, situation construal significantly contributed to SJT performance, even if only response options were available. This was also true for SJT items that are significantly more difficult to solve when situation descriptions are omitted (i.e., description-dependent SJT items). Finally, situation construal explained variance in relevant criteria over and above SJT performance. Despite recent efforts to reconceptualize SJTs, our results suggest that they can still be viewed as situational measures. However, situation descriptions may be less crucial for these underlying This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.

show abstract

Section: Endnotesmentioning

confidence: 99%

“…44Meta‐analyses have revealed that SJTs’ internal consistencies are generally low to moderate (Catano, Brochu, & Lamerson, 2012; Kasten & Freund, 2016). …”

mentioning

confidence: 99%

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

Freudenstein

Schäpers

Roemer

et al. 2020

Personnel Psychology

View full text Add to dashboard Cite

show abstract

“…Psychometric examinations of measurement instruments using meta-analytical methods primarily focus on issues of reliability (e.g., Gnambs, 2014Gnambs, , 2015Kasten and Freund, 2015) or selected aspects of criterion validity (e.g., Van Iddekinge et al, 2012;Salgado and Táuriz, 2014); matters of factorial validity are rarely addressed (e.g., Worley et al, 2008). The latter is typically impeded by a lack of appropriate meta-analytic methods for the pooling of published factor loading matrices.…”

mentioning

confidence: 99%

Parameter accuracy in meta‐analyses of factor structures

Gnambs

Staufenbiel

2016

Research Synthesis Methods

View full text Add to dashboard Cite

Two new methods for the meta-analysis of factor loadings are introduced and evaluated by Monte Carlo simulations. The direct method pools each factor loading individually, whereas the indirect method synthesizes correlation matrices reproduced from factor loadings. The results of the two simulations demonstrated that the accuracy of meta-analytical derived factor loadings is primarily affected by characteristics of the pooled factor structures (e.g., model error, communality) and to a lesser degree by the sample size of the primary studies and the number of included samples. The choice of the meta-analytical method had a minor impact. In general, the indirect method produced somewhat less biased estimates, particularly for small-sample studies. Thus, the indirect method presents a viable alternative for the meta-analysis of factor structures that could also address moderator hypotheses. Copyright © 2016 John Wiley & Sons, Ltd.

show abstract

“…The reliability of the SJT scores in the six conditions ranged from .63 to .87 (Cronbach's α) and from .58 to .87 (McDonald's ω total). These reliability estimates are above those reported in meta‐analyses on SJTs in general (Catano, Brochu, & Lamerson, ; Kasten & Freund, ; McDaniel, Morgeson, Finnegan, Campion, & Braverman, ). SJT scores for one condition (video situation description/text responses) exhibited a lower internal consistency (α = .63, ω = .58) than the others.…”

Section: Resultsmentioning

confidence: 46%

Removing situation descriptions from situational judgment test items: Does the impact differ for video‐based versus text‐based formats?

Schäpers

Lievens

Freudenstein

et al. 2019

J Occupat & Organ Psyc

View full text Add to dashboard Cite

Recent research has shown that many text-based situational judgment test (SJT) items can be solved even when the situational descriptions in the item stems are not presented to test takers. This finding challenges the traditional view of SJTs as low-fidelity simulations that rely on 'situational' (context-dependent) judgment. However, media richness theory and construal level theory suggest that situation descriptions presented in a richer and more concrete format (video format) will reduce uncertainty about inherent requirements and facilitate the perception that the situation is taking place in the here and now. Therefore, we hypothesized that situational judgment would be more important in video situation descriptions than in text situation descriptions. We adapted a leadership SJT to realize a 3 (situation description in the item stem: video vs. text vs. none) 9 2 (response format: video response options vs. text response options) between-subjects design (N = 279). Participants were randomly assigned to one of the six conditions. The removal of videobased situation descriptions in item stems led to an equivalent decrease in SJT scores as the removal of text-based situation descriptions in item stems (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). SJT scores were also contingent on the presentation format of both situation descriptions and response options: The highest scores were observed when situation descriptions and response options were presented in the same format. Implications for SJT theory and research are discussed. Practitioner pointsThe presentation format did not moderate the effect of omitting situation descriptions in SJTsthat is, the context dependency of SJT performance did not increase when the SJT was administered in a videobased rather than a text-based format. The elimination of situation descriptions in item stems had a medium effect on overall test scores: SJT scores were significant lower without situation descriptions in comparison to SJT scores with situation descriptions (video-based version: Cohen's d = 0.535 vs. text-based version: Cohen's d = 0.531). It is important to match the stimulus and response formats in SJTs.

show abstract

A Meta-Analytical Multilevel Reliability Generalization of Situational Judgment Tests (SJTs)

Cited by 32 publications

References 31 publications

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

Is it all in the eye of the beholder? The importance of situation construal for situational judgment test performance

Parameter accuracy in meta‐analyses of factor structures

Removing situation descriptions from situational judgment test items: Does the impact differ for video‐based versus text‐based formats?

Contact Info

Product

Resources

About