Accounting for Impact? The Journal Impact Factor and the Making of Biomedical Research in the Netherlands

Abstract

The range and types of performance metrics has recently proliferated in academic settings, with bibliometric indicators being particularly visible examples. One field that has traditionally been hospitable towards such indicators is biomedicine. Here the relative merits of bibliometrics are widely discussed, with debates often portraying them as heroes or villains. Despite a plethora of controversies, one of the most widely used indicators in this field is said to be the Journal Impact Factor (JIF). In this article we argue that much of the current debates around researchers’ uses of the JIF in biomedicine can be classed as ‘folk theories’: explanatory accounts told among a community that seldom (if ever) get systematically checked. Such accounts rarely disclose how knowledge production itself becomes more-or-less consolidated around the JIF. Using ethnographic materials from different research sites in Dutch University Medical Centers, this article sheds new empirical and theoretical light on how performance metrics variously shape biomedical research on the ‘shop floor.’ Our detailed analysis underscores a need for further research into the constitutive effects of evaluative metrics.

Bridging the Evaluation Gap

“…Most bibliometricians saw this as an orderly process in which professionally produced high quality bibliometric data "inform" the peer reviewers who subsequently exercise their professional judgment.! However, in the practice of knowledge creation and evaluation, we observe a much more diversified interaction and mixing of peer based evaluations and metrics informed assessments (Wouters 2016b;Rushforth and de Rijcke 2015;Wouters et al 2015;Sauder and Espeland 2009;Hicks et al 2015;Wouters 2016a). As a result, "metrics" and "peer review" are mutually interpenetrating each other.…”

Portfolios of Worth: Capitalizing on Basic and Clinical Problems in Biomedical Research Groups

“…Informants were able to align certain projects with the two regimes of worth, for instance, by citing examples of results that were published in high-impact journals and that could be mobilized to persuade funders, managers, and clinicians of the patient relevance of their work. What is interesting in the above examples of "high-impact" outputs associated with JIFs is that they do not occlude a priori other pertinent values in biomedical research, such as patient relevance or clinical translation, which is often implied in more outspoken "folk theories" on the influence of citations and the JIF in biomedicine (Rushforth and de Rijcke 2015). While on occasion there could be alignments between these regimes, tensions still tended to occur, more often than not.…”

“…In this scheme, while various kinds of worth may be cited by researchers as important and motivating, not all forms can be accumulated and reinvested in subsequent cycles of credit. For instance, producing research that goes "against the grain" or is ambitious may embody forms of epistemic worth with which researchers readily identify, yet in some institutional and disciplinary settings this work may not be capitalized upon by the researchers unless it has also appeared in a journal with, say, a high journal impact factor (JIF) score or goes onto attract a considerable number of citations (Rushforth and de Rijcke 2015). Focusing on capitalistic dynamics in the research process thus entails drawing attention to how multiplicitous forms of worth and the possibilities for accumulating capital from them shape the organization of academic research.…”

<b>Implicated in the Indicator Game? An Experimental Debate</b>

“…The results of some recent studies suggest that this could indeed be the case. Based on a study of biomedical research in the Netherlands, Rushforth and de Rijcke (2015) argue that indicators, among other functions, also serve as judgment devices in reflexively assessing the worth of a particular line of experimentation, both in relation to how well it can be published and the career value of the resulting publication. In this, indicator scores of journals are linked to epistemic properties of experiments, such as whether a study is purely descriptive or may be argued to also analyze biological mechanisms.…”