Theory-based evaluations have helped open the 'black box' of programmes. An account is offered of the evolution of this persuasion, through the works of Chen and Rossi, Weiss, and Pawson and Tilley. In the same way as the 'theory of change' approach to evaluation has tackled the complexity of integrated and comprehensive programmes at the community level, it is suggested that a theory-oriented approach based on the practice of realistic cumulation be developed for dealing with the vertical complexity of multi-level governance.
K E Y WO R D S : complexity; institutional evaluation hierarchies; integrated programmes; multi-level governance; theory
Black Boxes and Evaluation DeficitsSince its inception with the 'War on Poverty' programmes in the US, evaluation has been plagued with the 'black box' problem. The black box is the space between the actual input and the expected output of a programme. Moved by the need to tackle serious social problems, programme designers often gloss over what is expected to happen, the how and why, when an input is put in place; and evaluations do the same concentrating on measuring outputs, whilst attributing the observed difference to the input. All this is hardly informative for a policy design wishing to build upon previous experience. However, for a long time evaluations have coexisted with black box programmes, and have tried to cope with similar shortcomings by developing sophisticated methods for measuring the distance between objectives and results.Nowadays the evaluation community has become more and more concerned with the challenge of how to understand 'what works better for whom in what circumstances, and why ' (Pawson and Tilley, 1997) to improve policy decisions and public sector practice. Theory-oriented approaches reproach the previous, method-oriented approaches for being ineffective, given their inability (or unwillingness) to 'open the black box'. In recent times, theory-oriented approaches have flourished, most notably of the 'theories of change' kind , with respect to the evaluation of complex programmes.