Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Abstract

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.

Situational judgement tests in medical education and training: Research, theory and practice: AMEE Guide No. 100

“…SJTs have been described as "construct heterogeneous" at the item level ) since any one item may target several performance dimensions or constructs (in others words, SJTs rarely measure one single dimension), which implies an inherent difficulty in accurately estimating the internal consistency of an SJT, and a risk of underestimating their reliability , Catano et al 2012. Indeed, a meta-analysis of 39 different studies, found Cronbach's alpha coefficients ranging from α=.43 to .94 .…”

“…Some researchers argue that test-retest or parallel forms reliability may be a more accurate approach to examining the reliability of an SJT since internal consistency is more appropriate for uni-dimensional (construct homogenous) tests (McDaniel & Nguyen 2001, Lievens et al 2008, Catano et al 2012. Research shows a range of values for test-retest (Ployhart & Ehrhart 2003, dependent upon the quality of the test construction.…”

Scoring method of a Situational Judgment Test: influence on internal consistency reliability, adverse impact and correlation with personality?

“…The appropriateness of internal consistency as a reliability estimate for SJT scores is often called into question (Catano et al 2012). Internal consistency reliability estimates, such as coefficient alpha, are based on the assumption that all items measure the same latent trait on the same scale, i.e., that the same latent trait equally contributes to all item scores (Yang and Green 2011).…”

“…A more accurate reliability estimate might be obtained by a combination of a more thoroughly construct-based SJT development (Christian et al 2010) and a reliability estimate that takes into account the imposed factor structure of the SJT, for example a structural equation modeling (SEM) reliability estimate (Yang and Green 2011) or stratified alpha (Catano et al 2012). Future research is required on the application of constructbased development methods and alternative internal consistency estimates for SJTs.…”

Development and Construct Validation of a Situational Judgment Test of Strategic Knowledge of Classroom Management in Elementary Schools

“…Test developers should take this point into account for future SJTs. However, the average internal consistency of SJTs in a meta-analysis of Catano, Brochu, and Lamerson (2012) was .46 (CI^.05), which is comparable to the scale monitoring with the lowest internal consistency of the three. Some researchers in the field of SJTs regard alpha, as used in the aforementioned meta-analysis, as an inappropriate reliability coefficient for SJTs, due to heterogeneity within one response (McDaniel et al, 2007;Motowidlo, Dunnette, & Carter, 1990;Schmitt & Chan, 2006;Whetzel & McDaniel, 2009).…”