The Mystery Method Reconsidered—A Tool for Assessing Systems Thinking in Education for Sustainable Development

Abstract

Influence diagrams, derived from the mystery method as its learning output, represent an externalization of systems thinking and are, therefore, valid to research; so far they have not been conceptualized in the research literature for teaching systems thinking in education for sustainable development. In this study, 31 of those diagrams are confronted with (1) three different expert references, in (2) two different ways, by (3) three different scoring systems to determine which evaluation option is both valid and easy to implement. As a benchmark, the diagrams’ diameters are used, which allows statements about the quality of the maps/diagrams in general. The results show that, depending on the combination of variables that play a role in the evaluation (1, 2, 3), the quality of the influence diagram becomes measurable. However, strong differences appear in the various evaluation schemes, which can be explained by each variable’s peculiarities. Overall, the tested methodology is effective, but will need to be sharpened in the future. The results also offer starting points for future research to further deepen the path taken here.

Systems thinking assessments in engineering: A systematic literature review

“…Ten of the 27 assessments utilized environmental topics ranging from sustainability to resource and energy use to ecology (Benninghaus et al, 2019b;Brandstädter et al, 2012;Hu & Shealy, 2018;Keynan et al, 2014;Meilinda et al, 2018;Rehmann et al, 2011;Zoller & Scholz, 2004). No other contexts frequently occurred; rather, there was a wide variety of content areas, such as information systems (Lavi et al, 2020), an export management company (Jaradat, 2014), and heating expenses in a village (Grohs et al, 2018).…”

“…Ten of 13 Group 3 assessments included identifying individual aspects of the problem, which we defined as attending to elements. Elements included objects and processes (Lavi et al, 2020(Lavi et al, , 2021, components (Meilinda et al, 2018), the system's structure (Brandstädter et al, 2012), key variables (Rehmann et al, 2011), terms (Keynan et al, 2014), or information cards (Benninghaus et al, 2019b). One example of an assessment that emphasized elements was Hrin, Milenkovi c, Segedinac, and Horvat's Assessment, which rated participants in part based on identifying concepts to fill in the Systemic Synthesis Questions [SSynQs] (Hrin et al, 2017).…”

“…Their inclusion of criteria about the identification of individual concepts in their scoring rubric showed that the element identification was foundational to their definition of systems thinking. In contrast, Mystery Maps provided participants with information cards, a proxy for the problem's main elements; thus, it did not emphasize element identification (Benninghaus et al, 2019b).…”

“…Twelve of the 13 assessments included identifying and/or analysing relationships between elements, such as objects (Lavi et al, 2020(Lavi et al, , 2021 or information cards (Benninghaus et al, 2019b). One example was the Systems Thinking Assessment Rubric (STAR), which rated participants partially on their understanding of structural and procedural relations (Lavi et al, 2020(Lavi et al, , 2021.…”

“…The inclusion of different types of links in the STAR showed that a consideration of relationships was a key aspect of the authors' definition of systems thinking. Mystery Maps also emphasized relationships, where all the evaluation variants rate participants based on whether they create the appropriate direct or indirect connections (Benninghaus et al, 2019b).…”