Assessing (for) impact: future assessment of the societal impact of research

Abstract

National research assessments play a role in providing accountability-to funders, government and civil society-for the activities of largely autonomous research systems. Increasingly, an element of such assessments is the assessment of societal impact. In this article, I review the development of impact assessment, with a focus on the UK's Research Excellence Framework, and consider implications and challenges for the future. Notwithstanding these challenges, I further argue that the assessment of societal impact needs to become a more central aspect of research evaluation. This article is published as part of a collection on the future of research assessment.

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

“…in terms of across sections "children have benefited in terms of enhanced […] awareness" "research into […] is demonstrated in terms of its reach by citation in" "The impact in terms of awareness-raising" the way(s) in which across sections "[researcher's] work was significant in the way in which the […] were devised" "evidence for the way in which coaches influence" in relation to 2 and 4 "In relation to (i) participants disclosed that" "This is important in relation to two approaches" "[Researcher's] work in relation to [research topic] has led to" • Correspond to structure that may be signposted in Section 1 (or at start of relevant Section) • One or two levels of subheadings • There is a danger of breaking the text up too much at the expense of a coherent narrative • Headings which are titles of research projects or names of researchers can give the impression that these are the focus of the case study, rather than the impact Bullet points, lists • List of testimonials • Details of impact by beneficiary • Highlighting the central research questions of projects • In Section 2 breaking down research findings • Bullets announce a list that is then not fully elaborated on • Points don't link together • Danger of highlighting irrelevant details and therefore weakening the claim for reach and significance Bold or italics • Bold is used for impacts, beneficiaries, researcher names, dates, references to Section 3/5 • Italics for testimonial quotes • Italics are less effective for impacts/beneficiaries • Testimonials as block quotations can give the impression of taking over from the main narrative the basis of their underpinning research. However, there is evidence that units whose research outputs scored well in REF2014 also performed well on impact (unpublished Research England analysis cited in Hill, 2016). This observation only shows that high-quality research and impact were co-located, rather than demonstrating a causal relationship between high-quality research and highly rated impacts.…”

“…Critics argue that a focus of academic performativity can be seen to "destabilise" professional identities (Chubb and Watermeyer, 2017), which in the context of research impact evaluation can further "dehumanise and deprofessionalise" academic performance (Watermeyer, 2019), whilst leading to negative unintended consequences (which Derrick et al, 2018, called "grimpact"). MacDonald (2017), Chubb and Reed (2018) and Weinstein et al (2019) reported concerns from researchers that the impact agenda may be distorting research priorities, "encourag[ing] less discovery-led research" (Weinstein et al, 2019, p. 94), though these concerns were questioned by University managers in the same study who were reported to "not have enough evidence to support that REF was driving specific research agendas in either direction" (p. 94), and further questioned by Hill (2016).…”

“…For example, Weinstein et al (2019) found that "a focus on changing the culture outside of academia is broadly valued" by academics and managers. The impact agenda might enhance stakeholder engagement (Hill, 2016) and give "new currency" to applied research (Chubb, 2017;Watermeyer, 2019). Others have highlighted the long-term benefits for society of incentivising research impact, including increased public support and funding for a more accountable, outward-facing research system (Chubb and Reed, 2017;Hill, 2016;Nesta, 2018;Oancea, 2010Oancea, , 2014Wilsdon et al, 2015).…”

Learning from the UK’s research impact assessment exercise: a case study of a retrospective impact assessment exercise and questions for the future

“…As has been pointed out elsewhere (Hill 2016), the REF was a new and unprecedented attempt at incorporating impact into a research assessment exercise whose primary purpose is to allocate research funding. In doing this, the UK was, to some extent, conducting an experiment as to whether universities could demonstrate their impact, and how this would be done.…”

“…Indeed, Hill, a Higher Education Funding Council for England (HEFCE) policymaker who was deeply involved in the development and delivery of REF 2014, wrote a review arguing this point (Hill 2016). Hill traces some of the first codifications of the idea that publicly funded research can, and should, have a practical usage for society to the words of Vannevar Bush in his treaty, Science The Endless Frontier:…”