<scp>O</scp>rder <scp>E</scp>ffects on <scp>S</scp>ituational <scp>J</scp>udgment <scp>T</scp>est <scp>I</scp>tems: <scp>A</scp> case of construct‐irrelevant difficulty

Marentette, Brian J.; Meyers, Lawrence S.; Hurtz, Gregory M.; Kuang, Daniel

doi:10.1111/j.1468-2389.2012.00603.x

Cited by 13 publications

(8 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The low answer variance for less difficult tasks – in the present case, the questions in the framework of the SJT – influences the mean indices of discrimination. As a further factor that influences the discriminatory power and, ultimately, the validity of, e.g., SJT results, the positioning of the SJT in the whole test is discussed in the literature [ 30 , 31 ]. In this context, Marentette et al [ 31 ] describe construct-irrelevant order effects which occur when longer SJT items and SJT items presented in written form have to be answered at the end of an admission process [ 31 ].…”

Section: Resultsmentioning

confidence: 99%

“…As a further factor that influences the discriminatory power and, ultimately, the validity of, e.g., SJT results, the positioning of the SJT in the whole test is discussed in the literature [ 30 , 31 ]. In this context, Marentette et al [ 31 ] describe construct-irrelevant order effects which occur when longer SJT items and SJT items presented in written form have to be answered at the end of an admission process [ 31 ]. Nevertheless, in any case all single test indices of any of the test parts were positive, indicating that participants with higher abilities on average performed better on each single test item.…”

Section: Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Situational judgment test as an additional tool in a medical admission test: an observational investigation

et al. 2015

View full text Add to dashboard Cite

BackgroundIn the framework of medical university admission procedures the assessment of non-cognitive abilities is increasingly demanded. As tool for assessing personal qualities or the ability to handle theoretical social constructs in complex situations, the Situational Judgment Test (SJT), among other measurement instruments, is discussed in the literature. This study focuses on the development and the results of the SJT as part of the admission test for the study of human medicine and dentistry at one medical university in Austria.MethodsObservational investigation focusing on the results of the SJT. 4741 applicants were included in the study. To yield comparable results for the different test parts, “relative scores” for each test part were calculated. Performance differences between women and men in the various test parts are analyzed using effect sizes based on comparison of mean values (Cohen’s d). The associations between the relative scores achieved in the various test parts were assessed by computing pairwise linear correlation coefficients between all test parts and visualized by bivariate scatterplots.ResultsAmong successful candidates, men consistently outperform women. Men perform better in physics and mathematics. Women perform better in the SJT part. The least discriminatory test part was the SJT. A strong correlation between biology and chemistry and moderate correlations between the other test parts except SJT is obvious. The relative scores are not symmetrically distributed.ConclusionsThe cognitive loading of the performed SJTs points to the low correlation between the SJTs and cognitive abilities. Adding the SJT part into the admission test, in order to cover more than only knowledge and understanding of natural sciences among the applicants has been quite successful.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Situational judgment test as an additional tool in a medical admission test: an observational investigation

et al. 2015

View full text Add to dashboard Cite

show abstract

“…Constructed response multimedia tests might require less cognitive resources to perform well because they allow to respond if at least the core of the situation is understood (Hakel, ; Ryan & Greguras, ). In contrast, closed‐ended response formats appear to involve higher cognitive demands because they require to evaluate the appropriateness of all given response options on a high level of detail or to detect differences between response options and to make comparative judgments between them (Marentette, Meyers, Hurtz, & Kuang, ). However, multimedia presentations do not always change d values when cognitive saturation of the test itself is high (Roth, Buster, & Bobko, ).…”

Section: Discussionmentioning

confidence: 99%

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

Herde

Lievens

Jackson

et al. 2019

Int J Selection Assessment

View full text Add to dashboard Cite

To promote diversity in organizations it is important to have accurate knowledge about subgroup differences associated with selection procedures. However, current estimates of subgroup differences in situational judgment tests (SJTs) are overwhelmingly based on range‐restricted incumbent samples that are downwardly biased. This study provides much‐needed applicant level estimates of SJT subgroup differences (N = 37,530). As a key finding, Black‐White differences (d = 0.66) were higher than in incumbent samples (d = 0.38). Overall, sex differences were small. Females scored higher for management jobs (d = −0.13) and males scored higher for administrative jobs (d = 0.15). By analyzing applicant samples that do not suffer from range restriction, this study adds knowledge about subgroup differences in SJTs.

show abstract

“…with GMA-the preferred response format would be one that displays the weakest relationship with GMA. A zero relationship is not expected because SJTs by their very nature place unique cognitive demands on test takers (Marentette, Meyers, Hurtz, & Kuang, 2012). However, based on the response process model, it was posited that the rank-SJT requires comparatively higher levels of cognitive and information processing than the most/least-SJT, which in turn requires more cognitive and information processing than the rate-SJT.…”

Section: Construct-related Validitymentioning

confidence: 99%

Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Arthur¹,

Glaze²,

Jarrett³

et al. 2014

Journal of Applied Psychology

View full text Add to dashboard Cite

As a testing method, the efficacy of situational judgment tests (SJTs) is a function of a number of design features. One such design feature is the response format. However, despite the considerable interest in SJT design features, there is little guidance in the extant literature as to which response format is superior or the conditions under which one might be preferable to others. Using an integrity-based SJT measure administered to 31,194 job applicants, we present a comparative evaluation of 3 response formats (rate, rank, and most/least) in terms of construct-related validity, subgroup differences, and score reliability. The results indicate that the rate-SJT displayed stronger correlations with the hypothesized personality traits; weaker correlations with general mental ability and, consequently, lower levels of subgroup differences; and higher levels of internal consistency reliability. A follow-up study with 492 college students (Study 2; details of which are presented in the online supplemental materials) also indicates that the rate response format displayed higher levels of internal consistency and retest reliability as well as favorable reactions from test takers. However, it displayed the strongest relationships with a measure of response distortion, suggesting that it is more susceptible to this threat. Although there were a few exceptions, the rank and most/least response formats were generally quite similar in terms of several of the study outcomes. The results suggest that in the context of SJTs designed to measure noncognitive constructs, the rate response format appears to be the superior, preferred response format, with its main drawback being that it is susceptible to response distortion, although not any more so than the rank response format.

show abstract

Order Effects on Situational Judgment Test Items: A case of construct‐irrelevant difficulty

Cited by 13 publications

References 33 publications

Situational judgment test as an additional tool in a medical admission test: an observational investigation

Situational judgment test as an additional tool in a medical admission test: an observational investigation

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

Comparative evaluation of three situational judgment test response formats in terms of construct-related validity, subgroup differences, and susceptibility to response distortion.

Contact Info

Product

Resources

About