Evaluating ‘impact’ in the UK’s Research Excellence Framework (REF): liminality, looseness and new modalities of scholarly distinction

Watermeyer, Richard; Chubb, Jennifer

doi:10.1080/03075079.2018.1455082

Cited by 62 publications

(41 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…These findings suggest that aspects of written style may have contributed towards or compromised the scores of some case studies in REF2014, in line with previous research emphasising the role of implicit and subjective factors in determining the outcomes of impact evaluation Watermeyer and Chubb, 2018). If this were the case, it may raise questions about whether case studies are an appropriate way to evaluate impact.…”

supporting

confidence: 82%

“…Highly-rated case studies were easy to understand and well written. In preparation for the REF, many universities invested heavily in writing assistance (Coleman, 2019) to ensure that impact case studies were "easy to understand and evaluationfriendly" (Watermeyer and Chubb, 2018) for the assessment panels, which comprised academics and experts from other sectors (HEFCE, 2011, p. 6). With this in mind, we investigated readability and style, both in the quantitative linguistic and in the qualitative thematic analysis.…”

Section: Search Term Appears In Section Examplementioning

confidence: 99%

“…For example, Pidd and Broadbent (2015) emphasised the implicit role a "strong narrative" plays in high-scoring case studies (p. 575). This was echoed by the fears of one REF2014 panellist interviewed by Watermeyer and Chubb (2018) who said, "I think with impact it is literally so many words of persuasive narrative" as opposed to "giving any kind of substance" (p. 9). Similarly, Watermeyer and Hedgecoe (2016), reporting on an internal exercise at Cardiff University to evaluate case studies prior to submission, emphasised that "style and structure" were essential to "sell impact", and that "case studies that best sold impact were those rewarded with the highest evaluative scores" (p. 651).…”

Section: Introductionmentioning

confidence: 99%

“…research findings based on panellist interviews and participant observation of REF2014 sub-panels argued that scores were strongly influenced by who the evaluators were and how the group assessed impact together. Indeed, a panellist interviewed by Watermeyer and Chubb (2018) concurred that "the panel had quite an influence on the criteria" (p. 7), including an admission that some types of (more intangible) evidence were more likely to be overlooked than other (more concrete) forms of evidence, "privileg[ing] certain kinds of impact". Other panellists interviewed spoke of their emotional and intellectual vulnerability in making judgements about an impact criterion that they had little prior experience of assessing (Watermeyer and Chubb, 2018).…”

Section: Introductionmentioning

confidence: 99%

“…Indeed, a panellist interviewed by Watermeyer and Chubb (2018) concurred that "the panel had quite an influence on the criteria" (p. 7), including an admission that some types of (more intangible) evidence were more likely to be overlooked than other (more concrete) forms of evidence, "privileg[ing] certain kinds of impact". Other panellists interviewed spoke of their emotional and intellectual vulnerability in making judgements about an impact criterion that they had little prior experience of assessing (Watermeyer and Chubb, 2018). argued that this led many evaluators to base their assessments on more familiar proxies for excellence linked to scientific excellence, which led to biased interpretations and shortcuts that mimicked "groupthink" (p. 193).…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

et al. 2020

View full text Add to dashboard Cite

This paper reports on two studies that used qualitative thematic and quantitative linguistic analysis, respectively, to assess the content and language of the largest ever sample of graded research impact case studies, from the UK Research Excellence Framework 2014 (REF). The paper provides the first empirical evidence across disciplinary main panels of statistically significant linguistic differences between high-versus low-scoring case studies, suggesting that implicit rules linked to written style may have contributed to scores alongside the published criteria on the significance, reach and attribution of impact. High-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach than low-scoring cases. High-scoring case studies contained attributional phrases which were more likely to attribute research and/or pathways to impact, and they were written more coherently (containing more explicit causal connections between ideas and more logical connectives) than low-scoring cases. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, and often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than typically associated with academic writing. Highscoring case studies in two Main Panels were significantly easier to read than low-scoring cases on the Flesch Reading Ease measure, although both high-scoring and low-scoring cases tended to be of "graduate" reading difficulty. The findings of our work enable impact case study authors to better understand the genre and make content and language choices that communicate their impact as effectively as possible. While directly relevant to the assessment of impact in the UK's Research Excellence Framework, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

show abstract

supporting

confidence: 82%

Section: Search Term Appears In Section Examplementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

et al. 2020

View full text Add to dashboard Cite

show abstract

Visegrád countries' scientific productivity in the European context: A 10‐year perspective using Web of Science and Scopus

Szuflita-Żurawska

Basińska

2021

Learned Publishing

View full text Add to dashboard Cite

Measuring the growth of research productivity is a core element of performance in the higher education sector. This paper aims to analyse the scientific productivity of the Visegrád Group countries (2010–2019) based on data from the Web of Scopus and Scopus databases, as well as data from secondary sources (demographic and socio‐economic factors). Quantitatively, although Poland has the highest output, this is due to its comparative size, whereas output per researcher in Poland is lower than in other Visegrád Four (V4) countries. As regards the qualitative approach, Hungary and the Czech Republic are significantly more effective in terms of scientific collaboration and receive a greater number of European Research Council grants. Thus, bibliometric data in relation to cross‐country characteristic indicators (socio‐economic variables including expenditure on science and having prestigious universities) are related to the positions of the V4 countries. Practical implications suggest that university management within the Visegrád Group must develop and continue strategies to strengthen international collaboration between researchers in order to accelerate change in the dissemination of scientific output at the global level. Academic scholarly publishers may benefit from publishing research on specific issues important to different regions of the world in order to expand their audience and gain new potential resources.

show abstract

Rethinking the Role of Productive Interactions in Explaining SSH Research Societal Impacts: Towards a Conceptual Framework for Productive Science System Dynamics

Benneworth

Castro‐Martínez

Olmos‐Peñuela

et al. 2022

University-Industry Knowledge Interactions

View full text Add to dashboard Cite

In this paper we seek to realise the potential that Spaapen and van Drooge’s productive interactions concept offers, but which we argue has been lost through its operationalisation as a process of ‘counting interactions’. Productive interactions arise through moments of contact between two very different systems (the societal and the scientific), and each system values societal impact in very different ways. Finding mutual value in that interaction is important, and we argue that value in both arises when network arrangements shift, as academic disciplines solve urgent scientific problems and as societies improve living conditions. Productive interactions approach assumes the value-frameworks of the wider networks within which particular knowledge sets become actionable. However, our constructive critique highlights the omission of the wider elements of science and social systems within which productive interactions takes place (and whose dynamics ultimately determine the final scientific and societal impact of that research). Indeed, research evaluation to date has not considered the consequences of the productive interactions in terms of these changing relationships. To contribute to this lacuna, we propose a model that conceptualises a meso-level system comprising interactions between actors within two subsystems, highlighting the importance of coupling between researchers and users, valuation signals given to particular productive interactions from researcher and societal communities and the way these signals in turn embed useful knowledge practices. We apply it to a set of examples of productive interactions in the field of social sciences and humanities (SSH) gathered in the framework of a European project.

show abstract

Evaluating ‘impact’ in the UK’s Research Excellence Framework (REF): liminality, looseness and new modalities of scholarly distinction

Cited by 62 publications

References 26 publications

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

Visegrád countries' scientific productivity in the European context: A 10‐year perspective using Web of Science and Scopus

Rethinking the Role of Productive Interactions in Explaining SSH Research Societal Impacts: Towards a Conceptual Framework for Productive Science System Dynamics

Contact Info

Product

Resources

About