The production of scientific and societal value in research evaluation: a review of societal impact assessment methods

Abstract

Over the past two decades, several methods have been developed to evaluate the societal impact of research. Compared to the practical development of the field, the conceptual development is relatively weak. This review article contributes to the latter by elucidating the theoretical aspects of the dominant methods for evaluating societal impact of research, in particular, their presuppositions about the relationship between scientific and societal value of research. We analyse 10 approaches to the assessment of the societal impact of research from a constructivist perspective. The methods represent different understandings of knowledge exchange, which can be understood in terms of linear, cyclical, and co-production models. In addition, the evaluation methods use a variety of concepts for the societal value of research, which suggest different relationships with scientific value. While some methods rely on a clear and explicit distinction between the two types of value, other methods, in particular Evaluative Inquiry, ASIRPA, Contribution Mapping, Public Value Mapping, and SIAMPI, consider the mechanisms for producing societal value integral to the research process. We conclude that evaluation methods must balance between demarcating societal value as a separate performance indicator for practical purposes and doing justice to the (constructivist) science studies’ findings about the integration of scientific and societal value of research. Our analytic comparison of assessment methods can assist research evaluators in the conscious and responsible selection of an approach that fits with the object under evaluation. As evaluation actively shapes knowledge production, it is important not to use oversimplified concepts of societal value.

Assessing the societal value of a service-learning project in information studies during the COVID-19 pandemic

“…Even if the interest in societal impact was initially conceived as a strategy to assess the returns of government R&D investments (Miettinen et al, 2015), it also covers the societal benefits of research in social, cultural, and environmental dimensions and also sustainability (Bornmann, 2013). Smit and Hessels (2021) of the university. Assessing impact from an economic and quantitative perspective causes such problems as causation and attribution, as it is difficult to detect causeand-effect relationships and attribute certain results to specific actors or interventions (Temple et al, 2018).…”

“…On the other hand, Heyeres et al's (2019) review of case studies as impact assessment tools reveals that little attention is usually given to the needs of target populations and to the economic value of research, claiming major transparency in the way in which resources are used. Finally, the 10 assessment methods that Smit and Hessels (2021) gather differ in the very concept of societal value, and especially in the types and roles of supporting actors in knowledge production processes and in the mechanisms of interaction that promote the transfer of knowledge. With its limitations, case studies have the advantage of adjusting to the great diversity of scenarios and social actors that can intervene in generating impact, even in the same field of knowledge (Janker & Mann, 2018).…”

“…In previous editions of the project, it had been impossible to collect data on societal outreach, among the other things, due to difficulties in engaging the elderly in the assessment process (Montesi et al, 2021). However, this edition has emphasized precisely the "societal value" of the project, as it is conceived in (Smit & Hessels, 2021), encompassing intangible results not necessarily embodied in behavioral or practical changes. According to the literature mentioned in the introduction and with the intention of emphasizing intangibility, the collaborative character and the liminality of societal value, we have designed a case study that allows to see the project from different perspectives and contributions.…”

A value creation model from science-society interconnections: Archetypal analysis combining publications, survey and altmetric data

“…It is through the intertwined processes of knowledge production, translation, communication, and utilization that efforts to 'valorise' research by and for different research users occurs. Similar to Smit and Hessels (Smit & Hessels, 2021), our conceptualisation of value from research is open, inclusive, and does not grant a priori ascendency to science or society. Based on this understanding, the paper conducts an exploratory empirical study of interconnections among research-related knowledge processes.…”

“…Such approaches are useful for reconstructing pathways between research activities, research outputs, and linked outcomes at the level of specific research projects, programmes, or organisations (Matt et al, 2017). Each of these approaches treats the conceptualisation of societal value from research differently (Smit & Hessels, 2021) and focuses on different configurations of connections between 'science and non-science' entities (Costas et al, 2021). This paper seeks to contribute to this discussion and to empirical analysis of the generation of value from research in society, but in a different way to that found in the 'research impact' literature.…”