Co-existing Notions of Research Quality: A Framework to Study Context-specific Understandings of Good Research

Abstract

Notions of research quality are contextual in many respects: they vary between fields of research, between review contexts and between policy contexts. Yet, the role of these co-existing notions in research, and in research policy, is poorly understood. In this paper we offer a novel framework to study and understand research quality across three key dimensions. First, we distinguish between quality notions that originate in research fields (Field-type) and in research policy spaces (Space-type). Second, drawing on existing studies, we identify three attributes (often) considered important for 'good research': its originality/novelty, plausibility/reliability, and value or usefulness. Third, we identify five different sites where notions of research quality emerge, are contested and institutionalised: researchers themselves, knowledge communities, research organisations, funding agencies and national policy arenas. We argue that the framework helps us understand processes and mechanisms through which 'good research' is recognised as well as tensions arising from the co-existence of (potentially) conflicting quality notions. 3Co-existing Notions of Research Quality dimensions of our framework, encompassing: (i) types of quality notions; (ii) attributes of quality; and (iii) sites where notions are established and institutionalised. Third, we combine these three dimensions into an overall framework to study and understand research quality notions. We then discuss ways in which this proposed approach stands to change the research agenda.

Are evaluative cultures national or global? A cross-national study on evaluative cultures in academic recruitment processes in Europe

“…For instance, economists valued applicants' publication numbers higher than physicists, who emphasized candidates' research contributions. These findings align with prior studies addressing the fields' different evaluative cultures (Lamont 2009) and support the enduring importance of norms and values stemming from the research field's definition of peer recognition and prestige (Clark 1983;Driori et al 2003;Hessels et al 2019;Langfeldt et al 2019). They further underline the differences in research practices and collaboration patterns in different research fields, and that these have to be taken into account in studies of recruitment processes.…”

“…However, research quality and academic qualifications are not fixed entities but socially constructed and negotiated among academics in peer-review processes (Langfeldt et al 2019). Each field has its own evaluative culture with its own understanding of academic qualifications and research quality that is tightly linked to its identity, epistemology, and academic work (Becher and Trowler 1989;Lamont 2009).…”

Criteria for assessing grant applications: a systematic review

“…The overlap of criteria of funding agencies with the criteria of peers outlined above is a sign of the stability of the boundary while the differences indicate that negotiations are continuing. Since our comparison of the criteria of peers and funding agencies is very general, future research should address this in more detail, for example, by using the framework of Langfeldt et al (2019), which is designed to study context-specific understandings of research quality.…”

“…In addition to reasons directly related to improving grant peer review, there are also less applied reasons for studying the criteria of peers. For example, criteria can give insight into the recognition and reward system of science (Chase, 1970) and the interplay between research and research policy (Langfeldt et al, 2019). Research on criteria can also examine presumptions such as grant peer review is 'probably anti-innovation' (Guthrie et al, 2018, p. 4), interdisciplinary research may be disadvantaged because 'interdisciplinary proposal reviews may have to combine multiple distinct understandings of quality' (Guthrie et al, 2018, p. 6), and the high degree of concentration of grant funding on certain topics and researchers may be driven by uniform assessment criteria (Aagaard, 2019).…”

Bibliometrics in Academic Recruitment: A Screening Tool Rather than a Game Changer

“…Of these, a candidate's research output has been identified as the most salient criterion (Van den Brink and Benschop 2011). However, research quality is not a straightforward concept, but rather a fluid, negotiated, socially constructed and multifaceted concept covering both originality, plausibility, solidity, and academic and societal relevance (Polanyi 2000;Lamont 2009;Langfeldt et al 2020). Montgomery and Hemlin (1991) moreover observed how different aspects of research quality were emphasized in different phases of a recruitment process, where candidates were acknowledged for their stringency and productivity in the early stages of the recruitment process, whereas the interview committee regarded originality and breadth of their work as more important.…”