Subgroup differences in situational judgment test scores: Evidence from large applicant samples

Herde, Christoph N.; Lievens, Filip; Jackson, Duncan J. R.; Shalfrooshan, Ali; Roth, Philip L.

doi:10.1111/ijsa.12269

Cited by 9 publications

(7 citation statements)

References 54 publications

(87 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…SJTs are widely assumed to yield weaker group score differences relative to general mental ability (GMA) tests, and yet interpersonal skill SJTs exhibit moderate group score differences favoring White test takers (e.g., Bobko & Roth, 2013;Whetzel et al, 2008). Similar group mean score differences have been found among applicants who completed SJTs designed for one of four different occupational groups (Herde et al, 2020). Likewise, female test takers are commonly found to outperform males on SJTs across a variety of domains and settings (Lievens & Motowidlo, 2016;Weekley et al, 2015).…”

Section: Present Studymentioning

confidence: 92%

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

Brown

Speer

Tenbrink

et al. 2022

Int J Selection Assessment

View full text Add to dashboard Cite

This study introduces a novel, game-like method for measuring social intelligence: the Social Shapes Test. Unlike other existing video or game-based tests, the Shapes Test uses animations of abstract shapes to represent social interactions.We explore demographic differences in Shapes Test scores compared to a written situational judgment test (SJT). Gender and race/ethnicity only had meaningful effects on written SJT, scores while no effects were found for Shapes Test scores. This pattern of results remained after controlling for general mental ability and English language exposure. We also found metric invariance between demographic groups for both tests. Our results demonstrate the potential for using animated shape tasks as an alternative to written SJTs when designing future game-based assessments.

show abstract

Section: Present Studymentioning

confidence: 92%

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

Brown

Speer

Tenbrink

et al. 2022

Int J Selection Assessment

View full text Add to dashboard Cite

show abstract

“…Consequently, applicant samples are more likely to contain lower scoring applicants, whereas incumbent samples are more likely to consist of higher scoring applicants. When incumbent samples are analysed, direct or indirect range restriction can thus give raise to substantially underestimated subgroup difference estimates (Bobko & Roth, 2013 ; Herde, Lievens, Jackson, Shalfrooshan, & Roth, 2019 ; Roth, Le, Oh, Van Iddekinge, & Robbins, 2017 ).…”

Section: Methodsmentioning

confidence: 99%

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio‐economic status on situational judgement test performance

Bardach

Rushby

Klassen

2021

Brit J of Edu Psychol

View full text Add to dashboard Cite

Background. Situational judgement tests (SJTs) measure non-cognitive attributes and have recently drawn attention as a selection method for initial teacher education programmes. To date, very little is known about adverse impact in teacher selection SJT performance.Aims. This study aimed to shed light on adverse effects of gender, ethnicity, and socioeconomic status (SES) on SJT scores, by exploring both main effects and interactions, and considering both overall SJT performance and separate SJT domain scores (mindset, emotion regulation, and conscientiousness).Sample. A total of 2,808 prospective teachers from the United Kingdom completed the SJTs as part of the initial stage of selection into a teacher education programme.Methods. In addition to SJT scores, the variables gender (female vs. male), ethnicity (majority group vs. minority group), and home SES background (higher SES status vs. lower SES status) were used in the analyses. Regression models and moderated regression models were employed.Results and conclusions. Results from the regression models revealed that gender effects (females scoring higher than males) were restricted to emotion regulation, while ethnicity effects (ethnic majority group members scoring higher than ethnic minority group members) emerged for SJT overall scores and all three domains. Moderated regression modelling results furthermore showed significant interactions (gender and ethnicity) for SJT overall scores and two domains. Considering the importance of reducing subgroup differences in selection test scores to ensure equal access to teacher education, this study's findings are a critical contribution. The partially differentiated results for overall vs. domain-specific scores point towards the promise of applying a domain-level perspective in research on teacher selection SJTs.Diversifying the teacher workforce has long been a concern of educational policy (e.g., Kirby, Berands, & Naftel, 1999). However, limited progress has been made to reach this goal (e.g., Albert Shanker Institute, 2015; OECD, 2016), as indicated by the relative scarcity ofThis is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited and is not used for commercial purposes.

show abstract

“…However, it is important to note that there now are increasing numbers of female medical students in many countries, so this is a consideration when using an SJT in how scores are weighted and combined with other selection methods in practice. One possible explanation for why females tend to outperform males on SJTs is that SJTs measure (to some extent) aspects of personality where females typically report greater conscientiousness, agreeableness and sensitivity when compared to males [ 26 , 27 ]. Similarly, in Weekley and Jones’s study [ 28 ] where females tended to perform better on an SJT, they argued that this was due to many SJT items in their study relating to interpersonal conflicts or interactions.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

et al. 2022

View full text Add to dashboard Cite

Background Internationally, medical selection relies heavily on prior academic attainment which has an adverse impact on the diversity of selected students. Since non-academic attributes are also important, this study aims to evaluate the use of a Situational Judgment Test (SJT) for selection and the impact on student diversity relating to gender, ethnicity and socio-economic status. Previous SJT research has almost entirely originated from a Western context and this study focuses on new evidence in a South East Asian context with a different demographic profile. Methods Thirty faculty members developed 112 SJT scenarios assessing professionalism, communication and self-awareness domains. The scenarios underwent a concordance stage where stakeholder input was sought on the content appropriateness, to define the item scoring key, followed by an initial psychometric evaluation with first and second year medical students (N = 436). Based on these results, 30 scenarios, consisting of 128 nested items, were selected for pilot testing and evaluation regarding diversity issues with two cohorts of applicants in 2017 (N = 446) and 2018 (N = 508). Results The SJT demonstrated good internal consistency (Cronbach’s alpha of 0.80 and 0.81 respectively). There were significant differences in SJT scores based on gender in both years, where females consistently outperformed males (p = .0001). However, no significant differences were found based on high school origin, parental educational background or ethnicity. Conclusions This is the first study to evaluate the use of an SJT in Indonesia, which has a unique diversity profile compared to Western countries. Largely, the preliminary results replicate previous studies of the potential diversity benefits of using an SJT as a tool for medical student selection and has the potential to level the playing field regarding socio-economic status and ethnicity. Further studies exploring more variables representing diversity are warranted to confirm the early results in this study.

show abstract

Subgroup differences in situational judgment test scores: Evidence from large applicant samples

Cited by 9 publications

References 54 publications

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

Using game‐like animations of geometric shapes to simulate social interactions: An evaluation of group score differences

The selection gap in teacher education: Adverse effects of ethnicity, gender, and socio‐economic status on situational judgement test performance

Evaluation of Situational Judgment Tests in student selection in Indonesia and the impact on diversity issues

Contact Info

Product

Resources

About