Changing research on research evaluation: A critical literature review to revisit the agenda

Abstract

The current range and volume of research evaluation-related literature is extensive and incorporates scholarly and policy/practice-related perspectives. This reflects academic and practical interest over many decades and trails the changing funding and reputational modalities for universities, namely increased selectivity applied to institutional research funding streams and the perceived importance of university rankings and other reputational devices. To make sense of this highly diverse body of literature, we undertake a critical review of over 350 works constituting, in our view, the ‘state-of-the-art’ on institutional performance-based research evaluation arrangements (PREAs). We focus on PREAs because they are becoming the predominant means world-wide to allocate research funds and accrue reputation for universities. We highlight the themes addressed in the literature and offer critical commentary on the balance of scholarly and policy/practice-related orientations. We then reflect on five limitations to the state-of-the-art and propose a new agenda, and a change of perspective, to progress this area of research in future studies.

Overton -- A bibliometric database of policy document citations

“…Research evaluation also developed to support a greater need for accountability (Thomas et al, 2020). Initially, by peer review (Gibbons & Georghiou, 1987), then strategic reorientation (Georghiou, 1995), and recently using more data driven approaches that incorporate bibliometric components (Adams et al, 2007;Hicks, 2010;Hicks & Melkers, 2013;Martin, 1996).…”

Understanding Conceptual Impact of Scientific Knowledge on Policy: The Role of Policymaking Conditions

“…As the increasingly widespread use of accountability tools in science governance places the onus of creating, tracking and reporting impacts on researchers, there is a need to refocus on the users of research in policymaking. Various approaches to evaluating research and their effects on science and scientists are widely debated (Donovan 2011;Penfield et al 2013;Thomas et al 2020;de Rijcke et al 2016;Watermeyer 2016). Typically, instruments cover both scientific excellence and societal impact (Bornmann 2013;Martin 2011).…”

Interpreting Bibliometric Data

“…As a consequence, research evaluation using, to a very variable degree, some form of publication and/or citation data is now widespread and present in different forms and at various levels in for example: European programs ( European Science Foundation, 2012 ), in Australia ( ARC, 2019 ), Finland ( Lahtinen et al, 2005 ), Italy ( CIVR, 2006 ; Abramo and D’Angelo, 2015 ), New Zealand ( Buckle and Creedy, 2019 ; PBRF, 2020 ), Sweden ( Karolinska Institute, 2010 ), Spain ( Jiménez-Contreras et al, 2003 ), Norway ( Sivertsen, 2018 ), the United Kingdom ( REF, 2020 ) and the United States ( National Institutes of Health, 2008 ). Thomas et al (2020) recently reviewed 350 research papers on performance-based research evaluation arrangements and discuss important limitations in applying and using such research.…”