Developing and validating a serious game‐based assessment for cyber occupations in the US Air Force

Wiernik, Brenton M.; Raghavan, Mukhunth; Caretta, Thomas R.; Coovert, Michael D.

doi:10.1111/ijsa.12378

Cited by 5 publications

(15 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“… Wiernik et al’s (2022) study is the only one on this type of validity. Their article documents the creation of the game-based assessment Virus Slayer.…”

Section: Resultsmentioning

confidence: 99%

“…Within game-based assessment, we could also differentiate between theory-driven games, designed to evaluate constructs that are related to job performance, and data-driven games, where game scores are related to the criterion instead of the constructs’ psychological entity ( Landers et al, 2021 ; Auer et al, 2022 ). An example of game-based assessment is Virus Slayer ( Wiernik et al, 2022 ), a serious game to assess candidates for cyber occupations in the United States Air Force (USAF).…”

Section: Introductionmentioning

confidence: 99%

“… Landers and Sanchez (2022) focus on games developed to evaluate what other classifications have called serious games ( Wiernik et al, 2022 ). However, conceptually, we can also include the possibility of using conventional games to gather information about specific abilities, such as general cognitive ability ( Quiroga et al, 2019 ; Peters et al, 2021 ).…”

Section: Introductionmentioning

confidence: 99%

“…As shown in Figure 1 , the classification begins on a continuum: at one end are the traditional assessments (e.g., tests, simulations), and at the other end, the playful games created for fun. Thus, we distinguish at least four types of GRA: (1) gamified assessment (e.g., Hommel et al, 2022 ); (2) gamefully designed assessment (e.g., Georgiou, 2021 ); (3) game-based assessments (e.g., Wiernik et al, 2022 ); and (4) playful games used for assessment purposes (e.g., Sanchez et al, 2022 ).…”

Section: Introductionmentioning

confidence: 99%

“…Concerning the rigor of the assessments, GRA used for personnel selection must meet psychometric standards ( Salgado et al, 2017 ; Landers et al, 2021 ; Wiernik et al, 2022 ): (1) acceptable reliability to ensure consistency in the measure; (2) construct validity, verifying that GRA measure what is meant to be measured; (3) predictive validity, to predict the criterion (e.g., job performance); (4) freedom from bias, so that applicants’ scores are not influenced by their personal characteristics (e.g., sex, experience with video games) or, if this occurs, knowing how to correct this effect when estimating the scores. Moreover, GRA should promote positive applicant reactions.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Game-related assessments for personnel selection: A systematic review

Ramos-Villagrasa

Río²,

Castro³

2022

Front. Psychol.

View full text Add to dashboard Cite

Industrial development in recent decades has led to using information and communication technologies (ICT) to support personnel selection processes. One of the most notable examples is game-related assessments (GRA), supposedly as accurate as conventional tests but which generate better applicant reactions and reduce the likelihood of adverse impact and faking. However, such claims still lack scientific support. Given practitioners’ increasing use of GRA, this article reviews the scientific literature on gamification applied to personnel selection to determine whether the current state of the art supports their use in professional practice and identify specific aspects on which future research should focus. Following the PRISMA model, a search was carried out in the Web of Science and Scopus databases, identifying 34 valid articles, of which 85.3% are empirical studies that analyze five areas: (1) validity; (2) applicant reactions; (3) design of GRA; (4) personal characteristics and GRA; and (5) adverse impact and faking. Together, these studies show that GRA can be used in personnel selection but that the supposed advantages of GRA over conventional tests are fewer than imagined. The results also suggest several aspects on which research should focus (e.g., construct validity, differences depending on the type of game, prediction of different job performance dimensions), which could help define the situations in which the use of GRA may be recommended.

show abstract

“… Wiernik et al’s (2022) study is the only one on this type of validity. Their article documents the creation of the game-based assessment Virus Slayer.…”

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Game-related assessments for personnel selection: A systematic review

Ramos-Villagrasa

Río²,

Castro³

2022

Front. Psychol.

View full text Add to dashboard Cite

show abstract

Cross‐national applicability of a game‐based cognitive assessment

Lill

McColl²,

Neale³

2023

Int J Selection Assessment

View full text Add to dashboard Cite

New technology has had a discernable impact on how organizations recruit and select potential employees. Game-based assessment has emerged as a potential technology that can be used to enhance the assessment of individual differences and applicants' views of the selection process. However, studies investigating the psychometric properties and predictive validity of game-based assessments are still lacking. This study investigated the structural equivalence of a game-based assessment of cognitive ability across 228 Australians and 239 South Africans. A smaller sample of 115 South Africans also received work performance ratings to investigate the predictive validity of the cognitive assessment. Results of factor analysis supported a strong general factor of cognitive ability across the entire sample but only partial metric and scalar invariance across the two nations. The general factor of the game-based assessment further revealed promising results in terms of its predictive validity for five broad dimensions of individual work performance. K E Y W O R D S cognitive ability, cross-cultural psychology, game-based assessments, individual work performance Key points• This study investigated the measurement invariance of an Australian developed game-based cognitive ability assessment (GBCA) in South Africa.• Partial metric and scalar invariance were obtained for a general factor of the GBCA among South African and Australian samples.• A general factor based on the GBCA displays promising criterion validity for performance in South Africa, highlighting its utility as a selection instrument.

show abstract

Are serious games an alternative to traditional personality questionnaires? Initial analysis of a gamified assessment

Ramos-Villagrasa,

Fernández-del-Río,

Hermoso

et al. 2024

PLoS ONE

View full text Add to dashboard Cite

Personality questionnaires stand as crucial instruments in personnel selection but their limitations turn the interest towards alternatives like game-related assessments (GRAs). GRAs developed for goals other than fun are called serious games. Within them, gamified assessments are serious games that share similarities with traditional assessments (questionnaires, situational judgment tests, etc.) but they incorporate game elements like story, music, and game dynamics. This paper aims to contribute to the research on serious games as an alternative to traditional personality questionnaires by analyzing the characteristics of a gamified assessment called VASSIP. This gamified assessment, based on an existing measure of the Big Five personality traits, incorporates game elements such as storyfication, immersion, and non-evaluable gamified dynamics. The study performed included 98 university students (77.6% with job experience) as participants. They completed the original personality measure (BFI-2-S), the gamified evaluation of personality (VASSIP), a self-report measure of the main dimensions of job performance (task performance, contextual performance, and counterproductive work behaviors), and measures of applicant reactions to BFI-2-S and VASSIP. Results showed that the gamified assessment behaved similarly to the original personality measure in terms of reliability and participants’ scores, although the scores in Conscientiousness were substantially higher in VASSIP. Focusing on self-reports of the three dimensions of job performance, regression models showed that the gamified assessment could explain all of them. Regarding applicant reactions, the gamified assessment obtained higher scores in perceptions of comfort, predictive validity, and attractiveness, although the effect size was small except for the latter. Finally, all applicant reactions except for attractiveness were related to age and personality traits. In conclusion, gamified assessments have the potential to be an alternative to traditional personality questionnaires but VASSIP needs more research before its application in actual selection processes.

show abstract

Developing and validating a serious game‐based assessment for cyber occupations in the US Air Force

Cited by 5 publications

References 21 publications

Game-related assessments for personnel selection: A systematic review

Game-related assessments for personnel selection: A systematic review

Cross‐national applicability of a game‐based cognitive assessment

Are serious games an alternative to traditional personality questionnaires? Initial analysis of a gamified assessment

Contact Info

Product

Resources

About