Assessing research excellence: Evaluating the Research Excellence Framework

Abstract

Performance-based research funding systems have been extensively used around the globe to allocate funds across higher education institutes (HEIs), which led to an increased amount of literature examining their use. The UK’s Research Excellence Framework (REF) uses a peer-review process to evaluate the research environment, research outputs and non-academic impact of research produced by HEIs to produce a more accountable distribution of public funds. However, carrying out such a research evaluation is costly. Given the cost and that it is suggested that the evaluation of each component is subject to bias and has received other criticisms, this article uses correlation and principal component analysis to evaluate REF’s usefulness as a composite evaluation index. As the three elements of the evaluation—environment, impact and output—are highly and positively correlated, the effect of the removal of an element from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings of HEIs. As a result, future evaluations may consider the removal of some elements of the REF or reconsider a new way of evaluating different elements to capture organizational achievement rather than individual achievements.

Evaluating Research for Development: Innovation to Navigate Complexity

“…In today’s context of performance-based research funding, demand for evaluation of the impact of academic research has increased (Zacharewicz et al 2019 ; Bornmann 2013 ) leading to ever more sophisticated metrics for assessing the excellence of research (Pinar and Horne 2022 ). The focus of these assessments, however, is premised on a linear pathway starting from new knowledge produced by excellent academic research, communicated through engagement activities and subsequently leading to changes in policy (see Georgalakis and Rose 2019 for broader debates on understanding how research leads to policy impact).…”

Business and Management Studies in the United Kingdom's 2021 Research Excellence Framework: Implications for Research Quality Assessment

“…Among the measures used were PhD student numbers, amount of competitive research funding, level of internationalization and scientific productivity (the latter typically assessing the number and quality of research publications, and measured using quantitative/bibliometric approaches, peer review or both; Zacharewicz et al, 2019). 5 For more information, see recent reviews of PRFs by Dougherty et al (2016), Hicks et al (2015), Pinar and Horne (2022), Sivertsen (2017) and Zacharewicz et al (2019).…”

“…The REF requirements for submitting institutions to demonstrate clear and appropriate research strategies, following the three elements, has undoubtedly influenced the strategic direction, internal processes and investment priorities of universities, but this has come at a cost. However, evidence suggesting that the scores of these three elements are highly correlated has generated debate about whether the inclusion of all three elements unnecessarily adds to the cost and complexity of the evaluation exercise (Pinar and Horne, 2022).…”

“…The REF requirements for submitting institutions to demonstrate clear and appropriate research strategies, following the three elements, has undoubtedly influenced the strategic direction, internal processes and investment priorities of universities, but this has come at a cost. However, evidence suggesting that the scores of these three elements are highly correlated has generated debate about whether the inclusion of all three elements unnecessarily adds to the cost and complexity of the evaluation exercise (Pinar and Horne, 2022). 3 In arguing for the efficient allocation of funding for research, the European Commission (in 2012) called for Member States to 'introduce or enhance competitive funding through calls for proposals and organisational assessments as the main modes of allocating public funds to research and innovation, introducing legislative reforms if necessary'.…”

Investigating the relationship between research income and research excellence in education: Evidence from the REF2021‐UoA23 data

“…These types of analyses also add to the growing knowledge base on university metrics and evaluation systems (see e.g. Basso & di Tollo, 2022;Pinar & Horne, 2022;Tymms & Higgins, 2018;Uslu, 2020). REF-type assessments were initiated to audit universities' research quality and to use this as a mechanism to distribute Government Quality Research (QR) funding (Kelly, 2016(Kelly, , 2023.…”