Professional and citizen bibliometrics: complementarities and ambivalences in the development and use of indicators—a state-of-the-art report

Abstract

Bibliometric indicators such as journal impact factors, h-indices, and total citation counts are algorithmic artifacts that can be used in research evaluation and management. These artifacts have no meaning by themselves, but receive their meaning from attributions in institutional practices. We distinguish four main stakeholders in these practices: (1) producers of bibliometric data and indicators; (2) bibliometricians who develop and test indicators; (3) research managers who apply the indicators; and (4) the scientists being evaluated with potentially competing career interests. These different positions may lead to different and sometimes conflicting perspectives on the meaning and value of the indicators. The indicators can thus be considered as boundary objects which are socially constructed in translations among these perspectives. This paper proposes an analytical clarification by listing an informed set of (sometimes unsolved) problems in bibliometrics which can also shed light on the tension between simple but invalid indicators that are widely used (e.g., the h-index) and more sophisticated indicators that are not used or cannot be used in evaluation practices because they are not transparent for users, cannot be calculated, or are difficult to interpret.

Same data—different results? Towards a comparative approach to the identification of thematic structures in science

“…Field delineation has become an important tool for normalising citation counts in bibliometric evaluations (e.g. Leydesdorff et al 2016b) as well as mapping emerging, inter-and transdisciplinary fields (e.g. Leydesdorff and Bornmann 2011;Small et al 2014).…”

In search of educational efficiency: 30 years ofMedical Education's top-cited articles

“…In this paper, we build from that analysis by looking at the content of the journal as it has been taken up by the journal's audience in the form of citations. Highly cited papers offer a means of establishing the perceived relevance and legitimacy of published scholarship by various stakeholders in the field such as researchers, educators and academic leaders . We recognise that we are using high citation rates to capture pieces that have generated high levels of engagement on the part of the research audiences of the journal, rather than as a way to establish legitimacy of the pieces under analysis.…”

“…Highly cited papers offer a means of establishing the perceived relevance and legitimacy of published scholarship by various stakeholders in the field such as researchers, educators and academic leaders. 5,[17][18][19][20] We recognise that we are using high citation rates to capture pieces that have generated high levels of engagement on the part of the research audiences of the journal, rather than as a way to establish legitimacy of the pieces under analysis. This approach has been used by others both within [21][22][23] and outside our field.…”

Bridging the Evaluation Gap

“…Hence, citation scores will be influenced by perception of reputation, whereas peer judgment may be informed by the value of performance metrics. In other words, most peer review may already be "informed" by metrics, albeit perhaps not in the systematic and expert led way the proponents of informed peer review would have wished for (Moed 2005;Leydesdorff, Wouters, and Bornmann 2016;Wouters et al 2013;Moed, Glänzel, and Schmoch 2005) Bringing these two themes together, we conceptualized the research assessment of our center as a form of "situated intervention" (Zuiderent 2015), which would help us to bridge the evaluation gap (rather than mind the gap). This should not be seen as an attempt to get a "real representation" of our research in the evaluation.…”