Effects of Situational Judgment Test Format on Reliability and Validity

Martín‐Raugh, Michelle P.; Anguiano-Carrsaco, Cristina; Jackson, Teresa; Brenneman, Meghan W.; Carney, Lauren; Barnwell, Patrick; Kochert, Jonathan F.

doi:10.1080/15305058.2018.1428981

Cited by 2 publications

(8 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Reliability and validity evidence for the Military SJT used in this study were previously reported by Martin-Raugh et al (2018).…”

Section: Military Sjtsupporting

confidence: 54%

“…Reliability and validity evidence for the Military SJT used in this study were previously reported by Martin-Raugh et al (2018). This SJT was designed based on 203 critical incidents obtained from U.S. soldiers and response options were created by an independent sample of SMEs.…”

Section: Sjtsmentioning

confidence: 99%

“…One limitation is that each of the SJTs that were used in the present study were all newly developed and had little pre-existing criterion-related validation data. Although the Communication SJT was only developed as part of a pilot project, we report construct validity evidence for the Medical and Military SJTs which have been presented as part of conference presentations (May & Prewett, 2016) or published in peer-reviewed journals (Martin-Raugh et al, 2018). It would be especially insightful to observe whether the within-item agreement between SMEs and crowd estimates has any relationship with the validity of the item (e.g., a point-biserial correlation with a criterion measure).…”

Section: Study Limitationsmentioning

confidence: 99%

See 2 more Smart Citations

Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Brown

Grossenbacher²,

Martín‐Raugh

et al. 2021

Int J Selection Assessment

Self Cite

View full text Add to dashboard Cite

This study seeks to answer a simple question: Is it possible to develop a scoring key for a situational judgment test (SJT) without a pool of subject matter experts (SMEs)? The SJT method is widely studied and used for selection in both occupational and educational settings (Oswald et al., 2004;Lievens & Sackett, 2012). SJTs are typically designed to measure procedural knowledge about how to behave effectively in a particular job (Motowidlo et al., 2006). Along these lines, SJT items are typically scored using rational keys based on ratings gathered from SMEs (Whetzel et al., 2020). According to the U.S. Office of Personnel Management, SMEs are defined as a "person with bona-fide expert knowledge about what it takes to do a particular job" (Office of Personnel Management, 2016). Their judgments determine how each item will be scored. This poses a challenge when creating a new assessment because test developers typically need an item pool at least twice as large as the desired length of the final test (Hinkin, 1998;Murphy & Davidshofer, 1998). Each of these items needs a scoring key in order to be evaluated during an initial round of testing. This means that SMEs would likely spend most of their time rating items that will not be used in the final version of the test.Researchers and test developers face several challenges when using SMEs. The criteria for determining who is considered a SME can be vague or inconsistent across studies. When SJTs are developed for a specific occupation, SMEs have included job incumbents, supervisors, customers, and even novices (Weekley et al., 2006). In many cases, these SMEs work in prestigious occupations (e.g., experienced physicians or medical professionals, Lievens & Sackett, 2012;Patterson et al., 2009) which can make them expensive and difficult to recruit. For more construct-driven SJTs (e.g., applied social skills), academic researchers or graduate

show abstract

“…Reliability and validity evidence for the Military SJT used in this study were previously reported by Martin-Raugh et al (2018).…”

Section: Military Sjtsupporting

confidence: 54%

Section: Sjtsmentioning

confidence: 99%

Section: Study Limitationsmentioning

confidence: 99%

See 1 more Smart Citation

Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Brown

Grossenbacher²,

Martín‐Raugh

et al. 2021

Int J Selection Assessment

Self Cite

View full text Add to dashboard Cite

show abstract

“…No âmbito da Psicologia Organizacional e do Trabalho (POT) e do Comportamento Organizacional (CO), Testes de Julgamento Situacional (TJSs) e Liderança Autêntica (LA) têm recebido atenção de pesquisadores e de profissionais da prática, desde a última década do século XX e com interesse crescente desde o início deste século. Restam, entretanto, lacunas em ambos os campos de pesquisa (Ambiel, Campos, Alves & Silva, 2015;Avolio, Wernsing, & Gardner, 2018;Campos & Rueda, 2018a, 2018bCorstjens, Lievens, & Krumm, 2017;De Leng et al, 2017;Kaminski, Felfe, Schäpers, & Krumm, 2019;Martin-Raugh et al, 2018). Acompanhando esse contexto, o objetivo deste artigo é investigar a confiabilidade de um TJS desenvolvido para avaliar a LA, endereçando duas das lacunas existentes: a confiabilidade de TJSs e a possibilidade de avaliar a LA em processos inerentes à prática organizacional, que não se beneficiam da aplicação de instrumentos de heterorrelato, tais como seleção, e treinamento e desenvolvimento.…”

unclassified

“…Normalmente, TJSs são bem aceitos pelos respondentes, que vêm neles uma proximidade maior com sua realidade laboral ou profissional. Ainda, sua validade preditiva e incremental têm alcançado bons resultados com relação a desempenho e outros critérios relacionados ao trabalho (Corstjens et al, 2017;Goss et al, 2017;Kaminski et al, 2019;Martin-Raugh et al, 2018;Tiffin et al, 2019).…”

unclassified

Escala de Avaliação do Líder Autêntico: Confiabilidade e Diferenças Entre Grupos

Campos¹,

Rueda²

2021

rPOT

View full text Add to dashboard Cite

Effects of Situational Judgment Test Format on Reliability and Validity

Cited by 2 publications

References 34 publications

Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Can you crowdsource expertise? Comparing expert and crowd‐based scoring keys for three situational judgment tests

Escala de Avaliação do Líder Autêntico: Confiabilidade e Diferenças Entre Grupos

Contact Info

Product

Resources

About