Information Literacy Skills Assessment in Digital Crisis Management Training for the Safety Domain: Developing an Unobtrusive Method

Abstract

This study aims to develop an unobtrusive assessment method for information literacy in the context of crisis management decision making in a digital serious game. The goal is to only employ in-game indicators to assess the players' skill level on different facets of information literacy. In crisis management decision making it is crucial to combine an intuitive approach to decision making, build up by experience, with an analytical approach to decision making, taking into account contextual information about the crisis situation. Situations like these have to be trained frequently, for example by using serious games. Adaptivity can improve the effectiveness and efficiency of serious games. Unobtrusive assessment can enable game developers to make the game adapt to the players current skill level without breaking the flow of gameplay. Participants played a gameplay scenario in the Dilemma Game. Additionally, participants completed a questionnaire that was used as a validation measure for the in-game information literacy assessment. Using latent profile analyses, unobtrusive assessment models could be identified, most of which correlate significantly to the validation measure scores. Although inconsistencies in correlations between the information literacy standards, which call for broader testing of the identified unobtrusive assessment models, have been observed, the results display a good starting point for an unobtrusive assessment method and a first step in the development of an adaptive serious game for information literacy in crisis management decision making.

Evaluation of Online Information in University Students: Development and Scaling of the Screening Instrument EVON

“…They constructed a test of 28 situational judgment tasks that provided students with a scenario description and several possible procedures to solve the scenario and requested them to rate each procedure according to its usefulness. Also worth mentioning is the serious game of Steinrücke et al (2020). They measured information literacy by classifying the in-game behavior of individuals playing a crisis situation manager game.…”

Digital game-based learning in crisis management decision-making

“…For example, van de Ven et al (2014) used a former version of the Dilemma Game to provide insight into different leadership styles. In a more recent study, Steinrücke et al (2020) discuss, that the Dilemma Game can be used to train and assess information literacy. The authors found a clear correlation between styles of in-game behavior and performance on specific facets of information literacy.…”

“…Given the close interaction between information and its context, training IL in practical domains as crisis management decision-making is a complex task to achieve efficiently (see Steinrücke et al, 2020). Training sessions in crisis management decision-making should be realistic, which means that they should be designed similar to real life crisis situations.…”

“…Furthermore, serious games can mimic realistic scenarios in various contexts , one of which is decisionmaking in crisis management . For example, Steinrücke et al (2020) showed that a serious game, called the Dilemma Game (T-Xchange, 2018), can be used to infer IL skill levels of crisis management decision-makers based on only their in-game behavior. Another example is described in the context of a serious game about decision-making in triage situations by van der Spek et al (2011), who found a positive effect on verbal knowledge of experts, supporting the potential that serious games offer even for experts.…”

“…To compare the IL skill level from before and after the intervention between the conditions we adapt the unobtrusive assessment models by Steinrücke et al (2020). In Table 4.2 we display which in-game indicators were used by the authors to assess which IL standard.…”