Construct Development and Validation in Game-Based Research

Abstract

Problem. Teams or groups of individuals working together to achieve a shared goal, make up today’s world of work. Although the literature is rife with issues concerning teams, there is no coherent structure to guide researchers wishing to gain a deeper understanding into those factors leading to positive team outcomes. Question. This is due in part to two factors; one being the methods (e.g., observation; self report) typically employed to study teams often do not apply rigorous standards for reliability and validity. The second is it is difficult to construct data gathering situations that realistically approach the context in which teams operate. Approach. Addressing the first issue, we present a framework for the type of data that should be gathered to reliably and validly evaluate team performance. We believe the second issue is best addressed through the application of serious games, where realistic scenarios are delivered representing problems similar to those faced by teams in the real world. Finally, we describe a study whereby we demonstrate the approach utilizing a serious game to gather the data, which is then analyzed to assess the reliability and validity of the measures. Conclusion. With reliability and validity being established, a latent change score model is presented to illustrate how rich models of team interaction can be stated, investigated, and statistically assessed; since the serious game ensures the quality of the team interaction and the superior quality of the data.

Serious Games are a Serious Tool for Team Research

“…Many of the real-world measures employed for individual and team performance assessment can be directly implemented in a serious game --leading to data that is both highly reliable and internally/externally valid. The reader is referred to [60] for a recent discussion and an example employing confirmatory factor analysis subsequent to testing a latent change score structural equation model. Examples employing graph theory with teams are described in [9][10][11].…”

“…As can be gleaned from our review, there is a plethora of available methods for serious games. These range from guiding principles [58] through traditional experimental designs [59] and emerging techniques such as confirmatory factor analysis and structural equation modeling [60], graph theory and Petri nets [9], and Bayesian networks [57]. Using the appropriate methodological and statistical tool for the research task at hand, we argue it is important to assess the reliability and validity of serious games.…”

A Quantitative Approach for Developing Serious Games for Aptitude and Trait Assessment

“…The key potential strength of serious games as a modality for employee assessment is their ability to create an immersive experience with high psychological fidelity to the onthe-job work environment [9,10]. That is, the game can place the player into cognitive and emotional states that resemble those experienced on the job [11].…”

“…After specifying the aptitudes and skills to be measured, the next step is to select the specific metrics that will be used to quantify these competences. Assessment in a serious game typically occurs via three primary methods-( 1) direct assessment by the game engine during gameplay, (2) ratings by trained observers, or (3) player responses to questions during or after play [10,20]. Some constructs, such as decision-making ability, may be amenable to direct assessment by the game engine (e.g., by achievement of game objectives), whereas others (e.g., team cohesion, emotional states) may be better-assessed using observer ratings or player responses to post-game reflection questions.…”

Simulation and Gaming can be Used to Determine Validity While Engaging in Collaborative Environments

“…In 1975, C. H. Lawshe described a formula that could be used by professionals to reach concurrence when exploring content validity. In this issue, we find two uses for simulation and gaming for accomplishing the sometimes-elusive task of validity: using serious games and team performance (Coovert, Winner, & Bennett, 2017) and using training sessions with questionnaires (van Lankveld, Sehic, Lo, & Meijer, 2017). In their research, both sets of researchers seek innovative ways of determining validity to confront real issues.…”