Effects of response instructions on situational judgment test performance in operational selection and developmental contexts

Zhang, Charlene; Cullen, Michael J.; Sackett, Paul R.

doi:10.1111/ijsa.12324

Cited by 4 publications

(3 citation statements)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further research is also required to investigate the fakability and coachability of SJTs in selection contexts. 56,57 Finally, since SJTs typically have lower subgroup differences than measures with higher cognitive loading (e.g., the United States Medical Licensing Examination [USMLE]), 31,32 future research should examine how combining and differentially weighting SJT scores, USMLE scores, and other selection criteria might help to increase the diversity of minority applicants in programs’ rank pools. 58,59…”

Section: Discussionmentioning

confidence: 99%

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

et al. 2022

Self Cite

View full text Add to dashboard Cite

Purpose To determine whether overall situational judgment test (SJT) scores are associated with programs’ clinical competency committee (CCC) ratings of trainee professionalism, any concerning behavior, and concerning behavior requiring active remediation at 2 time periods. Method In fall 2019, trainees from 17 U.S. programs (16 residency, 1 fellowship) took an online 15-scenario SJT developed to measure 7 dimensions of professionalism. CCC midyear and year-end (6 months and 1 year following SJT completion, respectively) professionalism scores and concern ratings were gathered for academic year 2019–2020. Analyses were conducted to determine whether overall SJT scores related to overall professionalism ratings, trainees displaying any concerns, and trainees requiring active remediation at both time periods. Results Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings (r = .21 and .14, P < .001 and = .03, respectively). Holding gender and race/ethnicity constant, a 1 standard deviation (SD) increase in overall SJT score was associated with a .20 SD increase in overall professionalism ratings at midyear (P = .005) and a .22 SD increase at year-end (P = .001). Holding gender and race/ethnicity constant, a 1 SD increase in overall SJT score decreased the odds of a trainee displaying any concerns by 37% (odds ratio [OR] 95% confidence interval [CI]: [.44, .87], P = .006) at midyear and 34% (OR 95% CI: [.46, .95], P = .025) at year-end and decreased the odds of a trainee requiring active remediation by 51% (OR 95% CI: [.25, .90], P = .02) at midyear. Conclusions Overall SJT scores correlated positively with midyear and year-end overall professionalism ratings and were associated with whether trainees exhibited any concerning behavior at midyear and year-end and whether trainees needed active remediation at midyear. Future research should investigate whether other potential professionalism measures are associated with concerning trainee behavior.

show abstract

Section: Discussionmentioning

confidence: 99%

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

et al. 2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…Due to the selection context, SJTs are mostly conducted in a high-stakes setting. There is a lot of research comparing different aspects of high-stakes vs. low-stakes settings for SJTs (e.g., [ 29 , 30 ]). However, when conducting SJTs in a high-stakes setting, there can be various factors in addition to the actual processing of the test material that place additional demands on cognitive resources, such as a time limit.…”

Section: Introductionmentioning

confidence: 99%

The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests

Rabe,

Gröne,

von Bernstorff

et al. 2024

BMC Med Educ

View full text Add to dashboard Cite

Background Situational Judgment Tests (SJTs) are commonly used in medical school admissions. However, it has been consistently found that native speakers tend to score higher on SJTs than non-native speakers, which can be particularly problematic in the admission context due to the potential risk of limited fairness. Besides type of SJT, awareness of time limit may play a role in subgroup differences in the context of cognitive load theory. This study examined the influence of SJT type and awareness of time limit against the background of language proficiency in a quasi high-stakes setting. Methods Participants (N = 875), applicants and students in healthcare-related study programs, completed an online study that involved two SJTs: one with a text-based stimulus and response format (HAM-SJT) and another with a video-animated stimulus and media-supported response format (Social Shapes Test, SST). They were randomly assigned to a test condition in which they were either informed about a time limit or not. In a multilevel model analysis, we examined the main effects and interactions of the predictors (test type, language proficiency and awareness of time limit) on test performance (overall, response percentage). Results There were significant main effects on overall test performance for language proficiency in favor of native speakers and for awareness of time limit in favor of being aware of the time limit. Furthermore, an interaction between language proficiency and test type was found, indicating that subgroup differences are smaller for the animated SJT than for the text-based SJT. No interaction effects on overall test performance were found that included awareness of time limit. Conclusion A SJT with video-animated stimuli and a media-supported response format can reduce subgroup differences in overall test performance between native and non-native speakers in a quasi high-stakes setting. Awareness of time limit is equally important for high and low performance, regardless of language proficiency or test type.

show abstract

“…In their meta‐analysis, McDaniel et al (2007) showed that SJTs with knowledge instructions correlate higher with cognitive ability, while SJTs with behavioral instructions correlate higher with personality. Furthermore, SJTs with knowledge instructions were less prone to faking in (high‐stakes) selection contexts (Nguyen et al, 2005; Zhang et al, 2021) due to their focus on maximum performance.…”

Section: Introductionmentioning

confidence: 99%

Analysis of a situational judgment test for teamwork as a preselection tool for an assessment center: A construct‐based approach

Goerke

Maier

2022

Int J Selection Assessment

View full text Add to dashboard Cite

Situational judgment tests (SJTs) constitute an efficient and well‐established method for predicting professional success. However, more information is needed regarding their relationship to applicants' behavior as observed in an assessment center (AC) at the construct level. In the present study, an SJT for teamwork (SJT‐TW) was assessed in terms of its suitability as a preselection tool for ACs in the context of a multistage selection. Data were collected from 276 applicants for ab initio pilot training during their selection process. Results from the regression analyses showed that the SJT‐TW test performance is a significant predictor for the AC result and it contributed more than knowledge, cognitive performance tests, and personality scales thereto. The SJT‐TW also caused significant increments for the prediction of single AC dimensions. Therefore, it was concluded that construct‐based SJTs are useful for the assessment of behavioral constructs and can complement selection processes as a preselection tool.

show abstract

Effects of response instructions on situational judgment test performance in operational selection and developmental contexts

Cited by 4 publications

References 52 publications

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

Can a Situational Judgment Test Identify Trainees at Risk of Professionalism Issues? A Multi-Institutional, Prospective Cohort Study

The effects of language proficiency and awareness of time limit in animated vs. text-based situational judgment tests

Analysis of a situational judgment test for teamwork as a preselection tool for an assessment center: A construct‐based approach

Contact Info

Product

Resources

About