Evaluation and monitoring of transdisciplinary collaborations

Abstract

In this paper we focus on the governance, in particular evaluation and monitoring, of the growing number of transdisciplinary collaborations (TDC's). Researchers and a variety of stakeholders collaborate in such TDC's, the purpose of which is to address societal challenges, like renewable energy, healthy aging or better language teaching in schools. Commonly used practices for evaluation of scientific research (accountability, rankings and benchmarking, dedicated to scientific excellence) do not fit the goals of TDC's. A bottom up or stakeholder oriented approach is better suited; one that stimulates mutual learning as well as the development of socially robust knowledge. We introduce the participatory impact pathways analysis (PIPA), as a method that suits the requirements. It has been developed in the context of development research. Two crucial features are the involvement of stakeholders from the start, and the joint development of a theory of change. This narrates what one wants to achieve and how that will be achieved. From this, stakeholders construct a logical frame that serves as a source for indicators. These indicators enable monitoring ex durante, during the TDC. We present evidence of the use of PIPA for a TDC. From this empirical evidence a number of issues with regard to evaluation, monitoring and indicators can be identified that require attention. Most prominent is the change of function of indicators. Instead of looking back and a focus on past performance, indicators look forward, in the short, intermediate and more distant future.

Handbuch Transdisziplinäre Didaktik

“…Die Wirkungspfade können im Projektverlauf iterativ im Rahmen einer begleitenden Evaluation zum Zweck der adaptiven Projektsteuerung diskutiert und revidiert werden. Eine Methode, die die Idee der Wirkungspfade aufgreift, ist der Theory-of-Change-Ansatz, der im Kontext der Entwicklungszusammenarbeit entwickelt und auf die transdisziplinäre Forschung übertragen wurde (van Drooge und Spaapen 2017, Belcher et al 2019, Fritz et al 2019.…”

Rethinking research impact assessment: a multidimensional approach

“…Secondly, the "productive interactions" concept (Spaapen & van Drooge, 2011) arises as an alternative for overcoming the difficulties of measuring and evaluating the social impact of research, focusing on the personal, indirect (through texts or artefacts) and financial (through money or 'in kind' contributions) interactions between researchers and other actors as a transparent proxy of the process from research to impact. Such a concept has been further developed, bringing more attention to the governance, evaluation and monitoring of transdisciplinary collaborations (TDCs) addressing societal challenges, as a fruitful -bottom-up or stakeholder oriented -approach for valorising socially robust knowledge (van Drooge & Spaapen, 2017). Thirdly, and in line with the approaches previously mentioned, the need for a more holistic view in the observation and monitoring of interdisciplinary research (Anzai et al, 2012) has been addressed in Japan as an attempt towards research valorisation.…”

“…Given the increasing pressure on considering research impact when assessing research activity, it is important to put forward systems which achieve this in broader and accurate way, going beyond (but without dismissing) the measurable effects of research. In alignment with 1) the context-based perspective of research assessment (Spaapen et al, 2007), 2) the transdisciplinary collaborations and "productive interactions" concepts in research evaluation and monitoring (Spaapen & van Drooge, 2011;van Drooge & Spaapen, 2017), 3) the need for a more holistic view in the observation and monitoring of interdisciplinary research (Anzai et al, 2012), and 4) the need for a fairer treatment of Social Sciences, Arts and Humanities in research impact assessment (Benneworth et al, 2016), this paper is an effort for joining and contributing to the ongoing learning process in research impact agenda, by proposing a multidimensional and flexible approach towards this issue.…”

Towards the Construction of Productive Interactions for Social Impact

“…The relevance of discussing contributions rather than attribution is considered because research is only one influencing factor that supports the results [11,[13][14][15]. The different meanings that the "social benefits of research" have for the various actors-higher education institutions, funding agencies, and government bodies-deriving from their own perspectives, views, and interests, represent another difficulty when assessing the impact of research [11], in addition to the segmentation of public policies and the need to establish a more comprehensive evaluative approach [16,17].…”