Relative Citation Ratio (RCR): A new metric that uses citation rates to measure influence at the article level

Abstract

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov. Author SummaryAcademic researchers convey their discoveries to the scientific community by publishing papers in scholarly journals. In the biomedical sciences alone, this process now generates more than one million new reports each year. The sheer volume of available information, together with the increasing specialization of many scientists, has contributed to the adoption of metrics, including journal impact factor and h-index, as signifiers of a researcher's productivity or the significance of his or her work. Scientists and administrators agree that the use of these metrics is problematic, but in spite of this strong consensus, such judgments remain common practice, suggesting the need for a valid alternative. We describe here an improved method to quantify the influence of a research article by making novel PLOS Biology |

Citation Metrics: A Primer on How (Not) to Normalize

“…However, if that paper is cited by another field that has high citation density, its normalized citation score may decrease rather than increase, depending on the precise construction of the normalization procedure [12]. Overall, there is some evidence that definition of fields based on cocitation, such as the approach recently proposed by the relative citation ratio (RCR) method [13], is better than using taxonomies using journal categories. Alternative approaches to define fields that use direct citation and bibliographic coupling have been proposed to be even better [14], but the verdict is not final.…”

“…The extent to which limitations of different approaches are frequent or not requires further empirical evaluation [8,15]. For example, in an assessment of 200,000 articles published between 2003 and 2010, only 0.2% experienced a drop in RCR of 0.1 [13]. …”

“…Some controversy exists also on how to handle self-citations. Table 1 summarizes factors that may be considered or debated during normalization and shows for comparison two systems, the RCR method [13] and the normalization used in the Leiden Ranking of universities (http://www.leidenranking.com), both of which use citation data from the Web of Science.…”

Neophilia ranking of scientific journals

“…Second, while a move toward scientist-level measures has its benefits (see e.g. Hutchins et al 2015), it also changes the scientists' incentives and ability to game those measures. It is not evident that one level of ranking (journal vs. scientist) is necessarily superior to the other for all purposes, irrespective of whether one seeks to capture influence or novelty.…”

Necessary but not sufficient: unique author identifiers

“…The NIH recently announced a novel Relative Citation Ratio to better measure the true impact of scientific articles 12. However, the NIH/NLM National Center for Biotechnology Information (NCBI) SciENcv system, which allows biomedical researchers to link unique ‘My NCBI Bibliographies’ with NIH Biosketches, as well as automatically pull US federal grant information from the NIH Electronic Research Administration system (‘eRA Commons’), is still not fully linked with the PubMed Advanced Search Builder.…”