Putting Theory-Oriented Evaluation Into Practice

Abstract

Evaluations of gaming simulations and business games as teaching devices are typically end-state driven. This emphasis fails to detect how the simulation being evaluated does or does not bring about its desired consequences. This paper advances the use of a logic model approach, which possesses a holistic perspective that aims at including all elements associated with the situation created by a game. The use of the logic model approach is illustrated as applied to SIMGAME, a board game created for secondary school level business education in six European Union countries. Keywords: business games; evaluation methods; logic model; secondary educationO ne can look back at a very long history of gaming simulations and their use for educational purposes. The history of efforts to study and prove their effectiveness and efficiency as a learning tool for business education purposes is also long (Wolfe & Crookall, 1998) and has been accompanied by changes in what has been accepted as valid evaluation criteria and methodologies (Feinstein & Cannon, 2002). Accordingly, a frequent critique in the gaming simulation literature concerns the conduct and results of its evaluation activities. It has been argued that past evaluations have been based on anecdotal evidence (Greenblat, 1989), lacked methodological rigor (Dorn, 1989;Remus, 1981), failed to use valid outcome criteria (Anderson & Lawton, 1992;Feinstein & Cannon, 2002;Wolfe, 1990), or used psychometrically unproven measurement techniques (Gosen & Washbush, 2004;Spector, 2000;Spector, Christensen, Sioutine, & McCormack, 2001).Even when the picture is confined to research considered methodologically sound, traditional evaluation shortcomings become apparent. These failings have resulted in widely varying appraisals of the value of games and simulations for teaching business subjects. This can be exemplified by reviews conducted by those (2001), Gosen and Washbush (2004), Keys and Wolfe (1990), Wolfe (1997), and Greenlaw and Wyman (1973). Although the overall picture suggests gaming simulation is an effective learning tool, the results of the relevant studies are far from ambiguous and are often contradictory.Such heterogeneous results indicate that mediating factors may exist, and that these factors exert an important influence on what makes for an effective experiential learning environment. For practitioners, such as simulation designers and trainers who use them, it would be of value to learn more about these influences. However, because most studies concentrate on showing whether simulations are effective as compared with an alternative teaching method (Wolfe, 1997), they fall short of divulging the specific conditions and factors that must be met to make simulation an effective learning tool. Thus, the problem of knowing whether an educational method such as simulation gaming works is relevant only if we also know how it works. Despite the considerable efforts that have been spent on evaluating the simulation gaming method, too little attention has been paid to gene...

Creating a Modified Monopoly Game for Promoting Students' Higher-Order Thinking Skills and Knowledge Retention

“…Similarly, Bransford et al (2000) contend that the degree of mastery of the initial learning influences successful transfer of learning. The empirical findings support the importance of prior knowledge in determining the effectiveness of simulation game-based learning (e.g., Bartholomew et al, 2000;Cheng et al, 2014;Hense et al, 2009;Ruggiero, 2015).…”

“…Empirical findings on factors relevant for learning with simulation games were derived from studies discussed in Section 2.6 and focused on this issue (e.g., Fraas, 1982;Hense, Kriz, & Wolfe, 2009). This developed model is illustrated in Figure 3.…”

“…Age is included in the framework because it might be expected to correlate significantly with learning from the developmental psychology point of view (Hense et al, 2009). For instance, Bartholomew et al, (2000) reported that among children aged six to 17, the older one benefited most from participating in a simulation game teaching about self-asthmamanagement skills.…”

Mobile Learning Game Effectiveness in Cognitive Learning by Adults: A Comparative Study

“…Thus, very strict and limited time and subject scope have been introduced, also the type of the test method have been selected based on the more behavioristic rationale. The other trade-off was removing the debriefing part, which is essential part of the learning process in simulation and gaming (Crookall, 2010; Kriz, 2003; Kriz & Hense, 2006; Hense, Kriz & Wolfe, 2009). Minimizing the influence of the teacher/moderator was one of the elements requiring attention, as the human factor was a disturbing element affecting the relationship between the control group and the test group.…”

Process-oriented evaluation of user interactions in integrated system analysis tools

“…Thus, a process-oriented method such as user observation is necessary to analyze user interactions and behavior in a greater detail [13][14]. Such investigation of process is important not only in usability evaluation, but also in analyzing other complex systems as well, especially when human interactions are involved [15][16].…”