Four Methodological Guidelines to Evaluate the Research Impact of Co-produced Climate Services

Abstract

As climate change impacts unfold across the globe, growing attention is paid toward producing climate services that support adaptation decision-making. Academia, funding agencies, and decision-makers generally agree that stakeholder engagement in co-producing knowledge is key to ensure effective decision support. However, co-production processes remain challenging to evaluate, given their many intangible effects, long time horizons, and inherent complexity. Moreover, how such evaluation should look like is understudied. In this paper, we therefore propose four methodological guidelines designed to evaluate co-produced climate services: (i) engaging in adaptive learning by applying developmental evaluation practices, (ii) building and refining a theory of change, (iii) involving stakeholders using participatory evaluation methods, and (iv) combining different data collection methods that incorporate visual products. These methodological guidelines offset previously identified evaluation challenges and shortcomings, and can be used to help stakeholders rethink research impact evaluation through their complementary properties to identify complex change pathways, external factors, intangible effects, and unexpected outcomes.

Improving stakeholder engagement in climate change risk assessments: insights from six co-production initiatives in Europe

“…The framework used synthesized knowledge gaps found in the literature (Leander et al, 2020) as a baseline to construct both general and specific research questions and related indicators. These were validated against a review of 25 peer-reviewed articles suggesting co-production evaluation practices (Englund et al, 2022).…”

“…To this end, we found the Wall et al (2017) framework for evaluating co-produced climate science particularly useful, acknowledging multiple components-including internal, external and process related factors-of relevance. This framework also corresponds to key factors identified in the wider literature on evaluating co-production processes (Englund et al, 2022). In brief, we applied three overarching categories centered on: (1) the knowledge co-production process, (2) co-production effects, and (3) contextual factors (Table 3).…”

“…The ToC fits well in the initial module of the impact chain method, scoping, as it allows stakeholders and researchers to co-explore issues and context in depth and formulate joint learning objectives and expected outcomes. Thereafter, the ToC can be used to monitor and evaluate the co-production process and encourage reflection and learning as new insights emerge (Englund et al, 2022).…”

“…Further, co-production processes are rarely evaluated (Lemos et al, 2018), yet a growing body of research suggests that monitoring and evaluation can support iterative learning and adaptive management in complex endeavors (Patton, 2010;van Tulder and Keen, 2018;Englund et al, 2022;Visman et al, 2022). A monitoring framework allows stakeholders and researchers to reflect whether learning outcomes are achieved and adjust the implementation process accordingly, hence stimulating a continuous real-time feedback loop that connects evaluation findings and decision-making.…”

Thermal stress information as a tourism-oriented climate product: Performance analysis for selected urban destinations in Romania and Italy

“…In this line, the efforts to define an evaluation framework (e.g., Refs. [ [32] , [33] , [34] ]) and to evaluate climate services with regard to their uptake [ 35 ], technical aspects [ 36 ] or economic/societal benefits (e.g., Refs. [ [37] , [38] ]) have been intensified.…”