A review of online reactions to game‐based assessment mobile applications

Abstract

This paper takes an exploratory approach to analyze reactions to game-based assessments (GBAs) by examining users' reviews of GBA mobile applications. In this study, we explore 3146 user reviews and 1253 comments from 10 GBA applications found on the two most popular mobile application distribution platforms using a natural language processing tool. Findings suggest that candidates generally perceive GBAs as novel and have varying reactions to specific game, assessment, and application elements. As this study contributes to the limited body of research available on candidates' reactions to GBA mobile applications, findings and research directions are discussed to expand our understanding of this growing area of assessment.game-based assessment, mobile applications, natural language processing, reactions Practitioner points• Game-based assessment (GBA) mobile applications are generally viewed by applicants as novel and interesting, and as a favorable method of assessment.• However, some users are hesitant to place full trust in GBA mobile applications, perceiving them as less face-valid than traditional assessment methods.• Users also react negatively to technical issues such as mobile application glitches or crashes.• Organizations seeking to use GBA mobile applications in their employee selection system should carefully design the assessment by selecting psychometrically valid and useful game elements, and should also incorporate user-tested feedback. | INTRODUCTIONEmployee selection has experienced several changes over its more than 100-year history (Ryan & Ployhart, 2014). Technology has driven this evolution in many ways and is often considered one of the biggest influences on how we recruit, select, train, and evaluate employees (Tippins, 2015). While validating and assessing employee selection systems is critical, many organizations have been increasingly concerned with the usability of employee selection procedures and applicants' engagement with the process. Applicant reactions are important to employers because of influences on the applicants' perceptions of the organization, validity, and utility of the procedures, influences on the employee selection process, and potential ethical and legal issues (Smither et al., 1993).Recently, game-based assessments (GBAs) have been promoted for use in employee selection as a potential method to improve the user experience (Chamorro-Premuzic et al., 2017). A scan of the literature reveals that research on applicant reactions to GBAs is

Game-related assessments for personnel selection: A systematic review

“…In fact, more than half of the empirical research on GRA deals with this issue. Thus, we can confirm that GRA promote positive reactions in the applicants (Georgiou, 2021;al-Qallawi and Raghavan, 2022;Georgiou and Lievens, 2022) and tend to be better valued than conventional selection methods (Collmus and Landers, 2019;Georgiou and Nikolaou, 2020;Landers et al, 2021;Harman and Brown, 2022;Hommel et al, 2022). It is noteworthy that this result seems consistent across the different types of GRA, as these investigations have been conducted with very diverse games, starting with traditional assessment and going on to include serious games.…”

“…On the other hand, we must acknowledge that not all reactions are positive. Thus, al-Qallawi and Raghavan (2022) study the reactions generated by nine serious games published in mobile application stores (App Store and Google Play). Through a qualitative approach, using natural language processing, they identify a general tendency to value GRA positively.…”

“…Bearing in mind both issues, rigor in the evaluation and the particular characteristics of each GRA, we propose to review the developing scientific literature on GRA applied to personnel selection with two objectives: (1) to determine whether the current state of the art supports their use in professional practice; (2) to identify specific aspects on which future research should focus. This will mitigate the general public’s misgivings concerning this new form of evaluation ( al-Qallawi and Raghavan, 2022 ) and, at the same time, it will help to clarify the incipient research on games-related assessment, which so far has shown some inconsistency, for example with the use of terms ( Landers and Sanchez, 2022 ).…”

Exploring Digital Narratives: A Comprehensive Dataset of Household Energy App Reviews

“…1. For data collection, the Python libraries "google-play-scraper " (available at https://pypi.org/ project/google-play-scraper/ ) [8] and the "app-store-scraper " (available at https://pypi.org/ project/app-store-scraper/ ) [9] were used [10] . The reviews were merged into a unified dataset and exported as a single Excel file.…”

Can we playfully measure cognitive ability? Construct‐related validity and applicant reactions

“…Additionally, Landers et al (2022) found that their GBA was perceived more positively in terms of procedural fairness, distributive fairness, job relatedness, and test propriety compared with a paper‐pencil test. Furthermore, al‐Qallawi and Raghavan (2022) examined user reviews and comments from 10 GBA applications on two mobile distribution platforms. Generally, they found many positive evaluations from applicants, who particularly perceived GBAs as innovative.…”