A methodology for assessing the effectiveness of serious games and for inferring player learning outcomes

Abstract

Although serious games are proven educational tools in many educational domains, they lack reliable, automated and repeatable methodologies to measure their effectiveness: what do players know after playing a serious game? Did they learn with it? Literature research shows that the vast majority of serious games are assessed through questionnaires, which strikes a stark contrast with current trends in the video game industry. Commercial videogames have been learning from their players through Game Analytics for years, using non-disruptive game tracking. In this paper, we propose a methodology to assess serious games effectiveness using non-disruptive in-game tracking. The methodology proposes a design pattern that structures the delivery of educational goals within the game. This structure also allows inferring learning outcomes for each individual player, which, when aggregated, would determine the effectiveness of the serious game. We have tested this methodology with a serious game that was played by 320 students. The proposed methodology allowed us to infer players' learning outcomes and assess game effectiveness and to spot issues in the game design.

An interactive E-book with an educational game for children with developmental disorders: A pilot user study

“…Although most of the games provide some output on user performance, usually it does not directly indicate if the learning goals have been met. Recently, a methodology for effectiveness of serious games as educational tools has been proposed by Serrano-Laguna et al [47]. In addition, Chaudy et al developed a serious games assessment engine [48].…”

A Comparative Study of In-Game and Out-Game Assessment for Storyline-Based Games

“…Games are rarely adopted in the classroom as teachers perceived them as black boxes [13] through which the assurance of learning is difficult to determine. Currently, there is a lack of methodologies on how to assess learning outcomes in games [13].…”

“…Games are rarely adopted in the classroom as teachers perceived them as black boxes [13] through which the assurance of learning is difficult to determine. Currently, there is a lack of methodologies on how to assess learning outcomes in games [13]. Learning analytics suffer from high complexity and costs [17] and there is little known of what learning analytics could tell the stakeholders [18].…”

“…Existing in-game assessment methods use a combination of questionnaires integrated into the educational game and data generated through the educational gameplay [11], [12], [13], [14] to provide an overview of the student learning outcome. Although none of these studies reported user experience being affected, few studies [11], [2] have actually assessed the effect of the way the assessment was integrated into the player experience.…”

“…Existing research that focused on using learning analytics for assessment purposes [11], [12], [13], [14], [15] have not assessed whether the use of learning analytics would provide similar results with the traditional assessment methods or have not focused on determining how changes done in the design of the game to collect existing analytics have affected the user experience. In this paper, we focus on addressing this gap.…”