An examination of evaluation users’ preferences for program- and actor-oriented theories of change

Abstract

As programs and policies have grown in complexity, theory of change evaluation approaches and methods have become increasingly popular. While the academic literature often adheres to program-oriented theories of change, it is gradually being exposed to actor-oriented theories of change, which aim to enhance the flexibility of evaluations in order to adapt to growing complexity. Aside from the methodological considerations of when to use which approach, it is generally accepted that to increase their influence, evaluation approaches and methods should be suited both to the evaluated program and the preferences of intended users. But do we really know what approaches intended users prefer? Using a simulation-based survey and interviews with 148 public-sector employees in Israel, this article aims to provide an initial answer to this question. The results suggest that there is room to reconsider current perceptions of what intended users think. The findings have implications for the types of theories of change evaluators can suggest to intended users, with the ultimate goal of increasing the influence of evaluations.

Participatory systems mapping for complex energy policy evaluation

“…Knox, 1995) is also a popular and sometimes conflated approach, but does not use any sense of causality in its approach, so we do not cover it here. ToC maps first emerged in the 1990s in the United States (see Weiss, 1995), and have become popular both within and beyond the evaluation community (Coryn et al, 2011; Ofek, 2017). The actual use of ToC maps varies among different practitioners (Mason and Barnes, 2007); some use them in relatively linear and policy-focussed ways, whereas others include feedbacks and wider contexts, some use programme-oriented maps, whereas others also use actor-oriented maps (Ofek, 2017).…”

“…ToC maps first emerged in the 1990s in the United States (see Weiss, 1995), and have become popular both within and beyond the evaluation community (Coryn et al, 2011; Ofek, 2017). The actual use of ToC maps varies among different practitioners (Mason and Barnes, 2007); some use them in relatively linear and policy-focussed ways, whereas others include feedbacks and wider contexts, some use programme-oriented maps, whereas others also use actor-oriented maps (Ofek, 2017). Despite its popularity, ToC is not beyond criticism, perhaps one of the most common being that they are too inflexible and linear, meaning they struggle to deal with complex contexts (Ofek, 2017).…”

“…The actual use of ToC maps varies among different practitioners (Mason and Barnes, 2007); some use them in relatively linear and policy-focussed ways, whereas others include feedbacks and wider contexts, some use programme-oriented maps, whereas others also use actor-oriented maps (Ofek, 2017). Despite its popularity, ToC is not beyond criticism, perhaps one of the most common being that they are too inflexible and linear, meaning they struggle to deal with complex contexts (Ofek, 2017). The main response to this has been to construct ToCs that focus more on actors, rather than variables, and to develop maps that show webs of actors, rather than chains of events and actors (e.g.…”

Cased-based modelling and scenario simulation for ex-post evaluation

“…Finally, several methodological points about CBMs were discussed including the issue of deciding on appropriate cases, (i.e. casing, see Ragin, 2009; Ragin and Becker, 1992), conducting validity checks on an evaluators model (Castellani and Haffery, 2009: 79–80) and incorporating theories of change (Coryn et al, 2011; Ofek, 2017; Rolfe, 2019), system maps (Barbrook-Johnson, 2020; Barbrook-Johnson, 2019; Barbrook-Johnson and Penn, 2021) or other theories to inform case construction and scenario exploration.…”

“…Under these circumstances, it may be that elements in the case profiles poorly capture critical causal factors that influence case dynamics and differences between cases behaviours. One way to address this challenge is to draw on theories of change (Coryn et al, 2011; Ofek, 2017; Rolfe, 2019; Wilkinson et al, 2021), systems maps (Barbrook-Johnson, 2020; Barbrook-Johnson, 2019; Barbrook-Johnson and Penn, 2021) or other theories that demarcate the major causal forces in and around the cases. These theories can then be used to inform how cases are built and what factors are manipulated in scenarios.…”