Specification and evaluation of an assessment engine for educational games: Empowering educators with an assessment editor and a learning analytics dashboard

Abstract

Assessment is a crucial aspect of any teaching and learning process. Educational games offer promising advantages for assessment; personalised feedback to students and automated assessment process. However, while many teachers agree that educational games increase motivation, learning and retention, few of them are ready to fully trust them as an assessment tool. We believe there are two main reasons for this lack of trust: educators are not given sufficient information about the gameplays, and many educational games are distributed as black-boxes, unmodifiable by teachers. This paper presents an assessment engine designed to separate a game and its assessment. It allows teachers to modify a game's assessment after distribution and visualise gameplay data via a learning analytics dashboard. The engine was evaluated quantitatively by 31 educators. Findings were overall very positive: both the assessment editor and the learning analytics dashboard were rated useful and easy to use. The evaluation also indicates that, having access to EngAGe, educators would be more likely to trust a game's assessment. This paper concludes that EngAGe can be used by educators effectively to modify educational games' assessment and visualise gameplay data, and that it contributes to increasing their trust in educational games as an assessment tool.

Specification and evaluation of an assessment engine for educational games: Integrating learning analytics and providing an assessment authoring tool

“…Another study was carried out with educators (Chaudy & Connolly, 2018) to evaluate the learning analytics dashboard of the engine and the visual editor that educators can use to modify a game's assessment. A tutorial was also be given to them and a post-questionnaire collected information about usability and usefulness.…”

“…EngAGe was designed to be used by two target users: developers creating the games and educators using and modifying them. The authors evaluated EngAGe quantitatively with 31 educators (Chaudy & Connolly, 2018). The study shows encouraging results with both the assessment editor and the LA dashboard rated useful and easy to use.…”

“…process-oriented vs. product-oriented or generic vs. detailed). As discussed in another paper (Chaudy & Connolly, 2018), we believe educators should be able to customise the feedback so any engine should only be able to provide a trigger event for feedback without having to influence its content.…”

A Customizable and Incremental Processing Approach for Learning Analytics

“…A recent work has presented the EngAge engine which separates the assessment from the educational game itself [10], [18]. A DSL is offered to developers to configure the assessment of any educational game available in the engine, and a set of web services is in charge of performing the assessment.…”

“…Firstly, a textual DSL was created that enables teachers to customize the LA dashboards. Other works have recently presented proposal to customize LA solutions, but the mechanisms provided to teachers are more limited, as discussed in detail in Section VI: wizards to choose predefined indicators [10], [12], [16] or DSLs that offer a more simple expression language to define indicators [11], [17], or DSLs targeted to developers instead of teachers [18].…”