Governing by narratives: REF impact case studies and restrictive storytelling in performance measurement

Abstract

Performance assessment is permeating increasingly diverse domains of higher education, even in areas previously perceived to be too complex and idiosyncratic to quantify. The UK's attempts to assess 'research impact' within the Research Excellence Framework (REF) are illustrative of this trend and are being closely monitored by several other countries. A fundamental rationale for employing narrative case studies to assess impact within REF, rather than taking a (less resource intensive) quantified approach, was that this would allow for the variation, complexity and idiosyncrasy inherent in research impact. This paper considers whether this promise of narrative flexibility has been realised, by analysing a combination of REF impact case study reports and interviews and focus group discussions with actors involved in case study production. Informed by this analysis, our central argument is that the very quality which allows narratives to govern is their ability to standardise performance (albeit whilst retaining a degree of flexibility). The paper proposes that REF impact case studies position narratives of impact as technologies of governance in ways that restrict the 'plot line' and belie the far more complex accounts held by those working to achieve research impact. This is partly because, as research impact becomes institutionalised within universities' measurement infrastructures, higher education institutions become impact gatekeepers, filtering out narratives that are deemed overly complex or insufficiently persuasive, while perpetuating particular approaches to recounting tales of impact that are deemed likely to perform well. Crucially, these narratives not only describe impact but actively construct it as an auditable phenomenon.

Talking methods, talking about methods: Invoking the transformative potential of social methods through animals, objects and how‐to instructions

“…However, moments of silence or hesitancy triggered by these feelings of discomfort (Chadwick, 2021) allowed us to generate deeper understanding of participants’ decision making around methods. Attuning to pauses, silences and tone of voice enabled us, with participants, to unravel some of the more complex, and at times self‐deprecatory narratives of the impacts emanating from social science research and methods that REF impact case studies are less able to capture (Bandola‐Gill & Smith 2021). While this paper, and the broader Methods for Change project are informed by a commitment to creative innovation with method, we maintain that interviews, particularly when carried out sensitively and collaboratively, have an important role to play (Burrell, 2014; Hitchings 2012).…”

“…Gently pushing respondents to go beyond familiar interpretations of research impact (Bandola‐Gill & Smith, 2021) helped elicit discussion of individual, “ therapeutic” or subtle change. Taking the time to talk about methodological detail exposes method itself as a space for the transformation of ideas, practices and knowledge (Law, 2004, 2009).…”

“…Some participants, and particularly those with recent experience of completing REF impact case studies, 2 could immediately highlight how their work had led to tangible, operational or policy changes. Yet, as Bandola‐Gill and Smith (2021, p. 1) observe, the particular form of narrative demanded by REF impact case studies works to ‘restrict the “plot line” and belie the far more complex accounts held by those working to achieve research impact’. As the authors note, this is significant because REF narratives “not only describe but also construct a very specific vision of impact, as streamlined, linear, and preferably economic” (p. 12; see also Demeritt, 2010).…”

Knowledge brokering organisations: a new way of governing evidence

“…Because we depoliticised this issue and explained it" (Mowat 4). This more informal impact is far from the clear and usually quantifiable examples of impact that knowledge brokers usually discuss (the three KBOs also part-took in traditional, more quantified evaluations of their impact with annual and mid-term reviews, external evaluations, altmetrics, and stories of impact (Bandola-Gill and Smith, 2021). In fact, this informal impact can be pictured as the blend of their multiple sources of legitimacies -academic rigour, knowledge brokering networks, evidence, trust and closeness to government.…”

Research funding and academics' scholarly performance

“…Such assessments tend to narrow the definition of 'impact' and cause academics to form a particular notion of the desired output, such as 'instrumental impacts' with tangible benefits to non-academic stakeholders, rather than other forms including 'conceptual impacts' or 'capacity building impacts' (Ma et al, 2020;Meagher & Martin, 2017). This leads academics to propose research that is closer to real-life applications (Stern, 2016) and to avoid long-term or risky research agendas that are less likely to be regarded as impactful by review panels (Bandola-Gill & Smith, 2021). Although some funding schemes have been designed to exclusively encourage blue-sky and innovative research during the past few years, it was recently reported that such schemes still fail to understand the dynamics of scientific progress and devaluate some forms of effort towards scientific innovation (Falkenberg, 2021).…”