Citation counts for research evaluation: standards of good practice for analyzing bibliometric data and presenting and interpreting results

Abstract

With the ready accessibility of bibliometric data and the availability of ready-to-use tools for generating bibliometric indicators for evaluation purposes, there is the danger of inappropriate use. Here we present standards of good practice for analyzing bibliometric data and presenting and interpreting the results. Comparisons drawn between research groups as to research performance are valid only if (1) the scientific impact of the research groups or their publications are looked at by using box plots, Lorenz curves, and Gini coefficients to represent distribution characteristics of data (in other words, going beyond the usual arithmetic mean value), (2) different reference standards are used to assess the impact of research groups, and the appropriateness of the reference standards undergoes critical examination, and (3) statistical analyses comparing citation counts take into consideration that citations are a function of many influencing factors besides scientific quality.

Organizational factors influencing scholarly performance: a multivariate study of biomedical research groups

“…Baird 1986;Crane 1972;Martin and Irvine 1983;Neuhaus and Daniel 2009;Schubert and Braun 1996;Van Raan 2004;Whitley 2000). As a result, performance can only be evaluated in an appropriate context (Bornmann et al 2008;De Jong et al 2011).…”

Does “birds of a feather flock together” matter—Evidence from a longitudinal study on US–China scientific collaboration

“…Among those research adopting micro-level analysis, the omission of variables in model specification is problematic. As noted by Garfield, the founding director of the Institute for Scientific Information Philadelphia, a citation itself is a function of many other variables in addition to scientific quality (Bornmann, Mutz, Neuhaus, & Daniel, 2008;Egghe & Rousseau, 1990;Garfield, 1972;Moed & Van Leeuwen, 1996). It is for this very reason that more recent studies have begun to adopt statistical modeling to exclude competing explanations (Beirlant, Glänzel, Carbonez, & Leemans, 2007).…”

Sociological and Communication-Theoretical Perspectives on the Commercialization of the Sciences

“…However, I agree that the Triple Helix thesis (Etzkowitz and Leydesdorff 2000) relates to philosophical and ethical issues such as the differences between commercialization and entrepreneurship in university-industry relations (Etzkowitz 2002;Mirowski and Sent 2007;Slaughter and Rhoades 2004). Furthermore, the commercialization of quantitative science studies and the development of bibliometric indicators has driven this debate home because of the use of indicators for legitimatory purposes (e.g., Bornmann et al 2008;Husband Fealing et al 2011;Mirowski 2010). The interest of journals and publishing houses in improving their ''impact factors'' or institutions in order to improve their rankings function increasingly as a research incentive in quantitative STS.…”