Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Abstract

As a testing method, the efficacy of situational judgment tests (SJTs) is a function of a number of design features. One such design feature is the response format. However, despite the considerable interest in SJT design features, there is little guidance in the extant literature as to which response format is superior or the conditions under which one might be preferable to others. Using an integrity-based SJT measure administered to 31,194 job applicants, we present a comparative evaluation of 3 response formats (rate, rank, and most/least) in terms of construct-related validity, subgroup differences, and score reliability. The results indicate that the rate-SJT displayed stronger correlations with the hypothesized personality traits; weaker correlations with general mental ability and, consequently, lower levels of subgroup differences; and higher levels of internal consistency reliability. A follow-up study with 492 college students (Study 2; details of which are presented in the online supplemental materials) also indicates that the rate response format displayed higher levels of internal consistency and retest reliability as well as favorable reactions from test takers. However, it displayed the strongest relationships with a measure of response distortion, suggesting that it is more susceptible to this threat. Although there were a few exceptions, the rank and most/least response formats were generally quite similar in terms of several of the study outcomes. The results suggest that in the context of SJTs designed to measure noncognitive constructs, the rate response format appears to be the superior, preferred response format, with its main drawback being that it is susceptible to response distortion, although not any more so than the rank response format.

The internal structure of situational judgement tests reflects candidate main effects: Not dimensions or situations

“…In support of the SJTs-as-methods perspective, several meta-analyses have found that SJT scores indeed relate to general mental ability and personality variables (Arthur et al, 2014;McDaniel & Nguyen, 2001;McDaniel et al, , 2007. However, under this perspective, the internal measurement structure specific to SJTs is essentially sidestepped and, as a result, there is no way of knowing what it is about SJTs that is reliable and, thus, leads to observed correlations with externally measured constructs.…”

The Use of Mobile Devices in High‐stakes Remotely Delivered Assessments and Testing

“…As these results indicate, it took on average 8.55 more minutes to complete the personality measure on a mobile device (d = −0.49, p < .05). Whereas it is impossible to specify the exact reasons for this difference, the pattern of results is consonant with the fact that longer response latencies are associated with activities and tasks that have higher cognitive demands (Arthur et al, 2014;Bassili & Scott, 1996;Yan & Tourangeau, 2008). The higher cognitive demands arise from the structural differences (and challenges) such as increased scrolling time, interface manipulation, and comprehension time associated with using a small screen mobile device.…”

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

“…However, past research demonstrated that design variations in SJTs appear to influence subgroup differences. For example, knowledge‐based response instructions showed slightly higher ethnic and sex group differences (Whetzel et al, ) which seems to be due to the higher cognitive load of knowledge‐based response instructions (Whetzel et al, ; see also McDaniel et al, ) (for other examples, see Arthur et al, ; McDaniel, Psotka, Legree, Yost, & Weekley, ; Weng, Yang, Lievens, & McDaniel, ).…”