Transition governance for energy efficiency - insights from a systematic review of Swedish policy evaluation practices

Abstract

Background: The transition towards a more sustainable energy system is urgent for addressing global environmental and social challenges, and will require transformative changes including improved energy efficiency in the built environment. To reach identified efficiency potentials, various policy instruments have been introduced but their effects are often unclear. In this paper, we argue that the outline of transformative policy strategies will require well-designed evaluations. The objective is to present a theory-based evaluation framework that can be used to assess existing evaluations, in order to support transformative policy strategies. The framework is also applied to provide insights from current Swedish evaluation practices. Methods: The theory-based evaluation framework presented builds on evaluation theory, policy analysis and transition research and is arranged around methods, value judgements and use of the evaluations. Moreover, key aspects from transition research are included to provide guidance for transformative efforts in the evaluations. The systematic review presented in the paper is qualitative, covering 33 policy evaluations for energy efficiency in buildings in Sweden, commissioned by Swedish governmental authorities over a decade. Results: The results of the review reveal a wide range of evaluations undertaken, using a sound methodological evaluation base that builds on a variety of methods for analysis, and application of multi-criteria analyses. Commonly, however, a rather narrow scope was applied and we note a missed opportunity for triangulation of findings. Key aspects for capturing transformative efforts, such as system-, scale-and multi-actor approaches, as well as visioning, experimentation and learning, were considered to varying extents, but could be more explicit and elaborate. Conclusion: In all, we find the proposed theory-based evaluation framework useful for assessing and discussing both robustness and transformative efforts of current policy and evaluation practices. The review of the Swedish policy evaluations further indicates sound evaluation practices, and a foundational structure for identifying and analysing transformative efforts. To fully support transformative changes, we suggest a wider system perspective and a more thorough multi-actor approach and actor involvement in the evaluations. We also stress the need to further link evaluation theory with transition research to design evaluations that can support transformative changes in society.

A theory-based approach to evaluations intended to inform transitions toward sustainability

“…Empirical studies applying the presented framework on new Swedish data (Neij et al, 2021;Sandin et al, 2019) have reconfirmed several previous results. Among these results are the predominance of effectiveness and impact as criteria, the quite limited use of theory, the assessment of the contrafactual development mainly limited to the perceptions of the interviewees, and the rare use of triangulation.…”

“…This does not imply that all evaluations should be focused merely on transitions, but that their role should clearly increase. Many studies have confirmed that the perception that evaluations largely focus on incremental impacts and effectiveness can clearly be established empirically (Huitema et al, 2011;Sandin et al, 2019;Schoenefeld, 2018). For an evaluation to have a focus on transitions, it implies that the evaluation should evaluate key mechanisms for transitions, that is, experimentation, visioning, and learning for acceleration; they should also be based on a system and multi-actor perspective.…”

“…Behavior can be studied from many different perspectives: sociology, economics, and psychology. We have here chosen to focus on psychology, mainly because it is a perspective that has been sparsely utilized in relation to sustainability transitions but also because it is particularly relevant for energy efficiency in buildings, which was the empirical case our framework first was utilized for (Neij et al, 2021;Sandin et al, 2019).…”

Making Use of Evaluations to Support a Transition towards a More Sustainable Energy System and Society—An Assessment of Current and Potential Use among Swedish State Agencies

“…While this may seem fairly obvious, it does raise further questions regarding whether the purpose, the main users, and the channels used for communication are indeed prepared to support a transition. A previous study of current Swedish evaluation practices implied that there is rarely a stated objective within an evaluation to provide knowledge to support a transition [38], and the focus of evaluations is often geared towards a program level. Therefore, while the purpose and reported uses are still seemingly not yet fully aligned for conducting evaluations with a transformative focus, this study indicates a rather robust and varied approach to disseminating results.…”

“…Neij et al [37] propose that such a transformative evaluation approach needs to build on insights drawn from various evaluations to deliberately synthesize a broad, holistic, transformative perspective of how different research initiatives and policy instruments are building momentum towards a sustainability transition together. A framework that accounts for the key aspects to consider when performing such transformative evaluations includes drivers for a transition, such as visioning, experimentation and learning, and accounts for a systems and multi-actor perspective, as well as considering reflexivity and the construction of value judgements [37,38]. How such evaluations can actually be facilitated and integrated into current evaluation commissioning and use practices is, however, largely unknown, particularly in the context of facilitating a transition towards a more efficient energy system and society.…”