The effects of response instructions on situational judgment test performance and validity in a high-stakes context.

Assessing reliability of situational judgment tests (SJTs) in high‐stakes situations is problematic with reliability inappropriately measured by Cronbach's alpha when test items are heterogeneous. We computed the corrected, weighted mean alpha from 56 alpha coefficients, which produced a value of α = .46 and reviewed appropriate types of reliability to use with SJTs. In the current longitudinal study, SJT test–retest reliability was r = .82, compared with internal consistency, α = .46, and stratified alpha, α = .45 at Time 1 and α = .52 and stratified α = .51 at Time 2. We used a student sample (Time 1: n = 185; Time 2: n = 132) with items from a credentialing exam with ‘should do’ instructions. The SJT correlated significantly with cognitive ability, r = .30, and agreeableness, r = .24. In Study 2, we assessed test–retest reliability with Human Resource professionals (Time 1: n = 94; Time 2: n = 32) who had been recently credentialed and who participated in a pilot test of new SJT items with ‘most likely/least likely do’ response options. The SJT test–retest reliability was r = .66 compared with internal consistency, α = .43 and stratified α = .47 at Time 1 and α = .61 and stratified α = .67 at Time 2. We discuss the theoretical implications of the Study 1 results as well as the practical implications for use of SJTs in credentialing examinations.

Section: Discussionsupporting

confidence: 86%

Section: Introductionmentioning

confidence: 87%

Assessing the Reliability of Situational Judgment Tests Used in High‐Stakes Situations

Catano

Brochu

Lamerson

2012

“…All SJT questions were of the multiple‐choice type, with four response alternatives. A knowledge‐based response format (‘What is the most effective response?’) was used because faking is not an issue for this response format (Lievens, Buyse and Sackett, ; McDaniel et al., ). The scoring key was developed via consensus among medical experts (a panel of experienced physicians and professors in general medicine).…”

Section: Methodsmentioning

confidence: 99%

“…A knowledgebased response format ('What is the most effective response?') was used because faking is not an issue for this response format (Lievens, Buyse and Sackett, 2009;McDaniel et al, 2007).The scoring key was developed via consensus among medical experts (a panel of experienced physicians and professors in general medicine). This scoring key indicated which response alternative was correct for each item (+1 point).…”

Section: Sjtmentioning

confidence: 99%

The Effects of Coaching on Situational Judgment Tests in High‐stakes Selection

Lievens

Buyse

Sackett

et al. 2012

Self Cite

Although the evidence for the use of situational judgment tests (SJTs) in high‐stakes testing has been generally promising, questions have been raised regarding the potential coachability of SJTs. This study reports the first examination of the effects of coaching on SJT scores in an operational high‐stakes setting. We contrast findings from a simple comparison of SJT scores for coached and uncoached participants (posttest only) with three different approaches to deal with the effects of self‐selection into coaching programs, namely using a pretest as a covariate and using two different forms of propensity score‐based matching using a wide range of variables as covariates. Coaching effects were estimated at about 0.5 SDs. The implications for the use of SJTs in high‐stakes settings and for coaching research in general are discussed.

“…In addition, the meta‐analysis of McDaniel et al () indicated that SJTs with knowledge instructions still have noncognitive correlates, although to a lesser extent than SJTs with behavioral tendency instructions. Moreover, the differences between both types of response instructions are not replicated in high‐stakes settings, like a medical school selection setting (Lievens, Sackett, & Buyse, ). Finally, due to the higher susceptibility to faking, behavioral tendency instructions are of limited practical value in high‐stakes medical school selection and examining faking effects on SJTs using these instructions would, therefore, have little ecological validity.…”

Section: Introductionmentioning

confidence: 99%

Faking on a situational judgment test in a medical school selection setting: Effect of different scoring methods?

Leng

Stegers‐Jager

Born

et al. 2019

We examined the occurrence of faking on a rating situational judgment test (SJT) by comparing SJT scores and response styles of the same individuals across two naturally occurring situations. An SJT for medical school selection was administered twice to the same group of applicants (N = 317) under low‐stakes (T1) and high‐stakes (T2) circumstances. The SJT was scored using three different methods that were differentially affected by response tendencies. Applicants used significantly more extreme responding on T2 than T1. Faking (higher SJT score on T2) was only observed for scoring methods that controlled for response tendencies. Scoring methods that do not control for response tendencies introduce systematic error into the SJT score, which may lead to inaccurate conclusions about the existence of faking.