Using a Complex Adaptive Systems Perspective to Illuminate the Concept of Evaluation Capacity Building in a Network

Abstract

While evaluation capacity building (ECB) may hold promise for fostering evaluation, little is known about how it is operationalized within a network. This article presents initial findings from a National Science Foundation-funded research project (Complex Adaptive Systems as a Model for Network Evaluations) that used concepts from complex adaptive systems theory to develop case studies of ECB within the Nanoscale Informal Science Education Network. The project used a multiple case study approach to explore ECB within four Network workgroups. Cross-case themes documented characteristics of the system and ECB within it. Evaluation capacity was evident in several ways, including people's comfort with evaluation, evaluation-related skills, evaluation processes used, and the value placed on evaluation. Ultimately, the study identified several complex adaptive system features that fostered Network ECB: massive entanglement and neighbor interactions, information

Including Diverse Stakeholder Voices in Youth Character Program Evaluation

“…A more recent definition, based on a research synthesis of the ECB literature, describes ECB as "an intentional process to increase individual motivation, knowledge, and skills, and to enhance a group or organization's ability to conduct or use evaluation" (Labin et al, 2012, p. 308). ECB practitioners and scholars agree that the ultimate purpose of ECB is to improve program outcomes (Labin, 2014;Nelson et al, 2019;Suarez-Balcazar & Taylor-Ritzler, 2014;Wandersman, 2014). In the PACE project, ECB was taught using elements of relational systems evaluation (RSE), including evaluative thinking, and evolutionary evaluation.…”

Grounding evaluation capacity development in systems theory

“…Nevertheless, a previous review of the published evaluation capacity development literature shows that a variety of researchers have acknowledged the potential usefulness of a systemic perspective (Bourgeois, 2016; Danseco, 2013; Haarich and del Castillo Hermosa, 2004; Preskill and Boyle, 2008; Stockmann, 2018). Indeed, systems theory is noted as a valuable framework for dealing with the complexities of evaluation capacity development (Danseco, 2013), and has even been used as an orientation tool for better understanding evaluation capacity development efforts, both within general evaluation systems (Haarich and del Castillo Hermosa, 2004) and educational networks (Grack Nelson et al, 2019). Previous studies have invoked systems language, and referenced important systems concepts such as levels, interactions, feedback loops (Haarich and del Castillo Hermosa, 2004), and control (Grack Nelson et al, 2019).…”

Applying a systems and complexity lens to building evaluation capacity: Learning from a multicountry donor's experience

“…Nylen and Sridharan (2020) examined the added value of collaborations and highlighted how knowledge had diffused externally to spread beyond the program's boundaries. Another set of articles (e.g., Grack Nelson et al., 2019; Lawrenz et al., 2018) used a complex adaptive system perspective to examine ECB in a network of science education organizations. Grack Nelson et al.…”

“…Grack Nelson et al. (2019) found that using a STCS lens facilitated, and later made visible, the complex flows and impact of information across participants and groups within the network; these findings may not have been as evident using traditional ECB and evaluation theories.…”