Specification and evaluation of an assessment engine for educational games: Integrating learning analytics and providing an assessment authoring tool

Abstract

Educational games are highly engaging, motivating and they offer many advantages as a supplementary tool for education. However, the development of educational games is very complex, essentially because of its multidisciplinary aspect. Fully integrating assessment is challenging and the games created are too often distributed as blackboxes; unmodifiable by the teachers and not providing much insight about the gameplays. We propose an assessment engine, EngAGe, to overcome these issues. EngAGe is used by both developers and educators. It is designed to separate game and assessment. Developers use it to easily integrate assessment into educational games and teachers can then modify the assessment and visualise learning analytics via an online interface. This paper focuses on EngAGe's benefits for games developers. It presents a quantitative evaluation carried out with 36 developers (7 experienced and 29 students). Findings were very positive: every feature of the engine was rated useful and EngAGe received a usability score of 64 using the System Usability Scale. A Mann-Whitney U test showed a significant difference in usability (Z=-3.34, p<0.002) between novice developers (mean=56) and experienced developers (mean=71) but none in terms of usefulness. This paper concludes that developers can use EngAGe effectively to integrate assessment and learning analytics in educational games.

Use of Learning Analytics Interactive Dashboards in Serious Games

“…In the context of serious games, learning analytics is an area of emerging knowledge that provides detailed reports on the use of a game; gameplay data and information visualization, based on data mining processes (Chaudy & Connolly, 2019). The combination of learning analytics techniques and serious games results in learning analytics in serious games .…”

“…The study showed satisfactory results and classified users with the probability of belonging to one of the three classes: high, medium and low performance (Villagrá-Arnedo et al, 2016). Studies in computational technologies also stand out, for addressing the development and application of assessment tools in serious games, such as the EngAGe tool, which is an integrated assessment tool for teachers and developers designed to separate the serious game from the system evaluation (Chaudy & Connolly, 2019). Developers use the tool to integrate assessment with teaching using games.…”

“…Developers use the tool to integrate assessment with teaching using games. In addition, teachers have the possibility to modify the assessment and view the learning analysis through a data control dashboard (Chaudy & Connolly, 2019). In order to test EngAGe, a qualitative assessment was carried out with the developers, where 7 experienced developers and 29 students were selected (Chaudy & Connolly, 2019).…”

“…In addition, teachers have the possibility to modify the assessment and view the learning analysis through a data control dashboard (Chaudy & Connolly, 2019). In order to test EngAGe, a qualitative assessment was carried out with the developers, where 7 experienced developers and 29 students were selected (Chaudy & Connolly, 2019). Qualitative evaluation was also carried out by 31 educators, and it was observed that with access to EngAGe, it is more likely that educators trust the evaluation of the game, since EngAGe can be used by educators effectively to modify educational games, evaluate, and view gameplay data .…”

“…With the breakthrough in digital technologies, the educational field has been seeking to update itself with regard to active teaching methodologies such as serious games (Chaudy & Connolly, 2019). Serious games prove to be very effective in the teaching-learning process and in retaining users, in addition to providing a wide variety of possibilities that can be difficult to obtain in a traditional classroom (Chaudy & Connolly, (Massa & Kuhn, 2019).…”