Relative Citation Ratio (RCR): A New Metric That Uses Citation Rates to Measure Influence at the Article Level

Abstract

Despite their recognized limitations, bibliometric assessments of scientific productivity have been widely adopted. We describe here an improved method to quantify the influence of a research article by making novel use of its co-citation network to field-normalize the number of citations it has received. Article citation rates are divided by an expected citation rate that is derived from performance of articles in the same field and benchmarked to a peer comparison group. The resulting Relative Citation Ratio is article level and field independent and provides an alternative to the invalid practice of using journal impact factors to identify influential papers. To illustrate one application of our method, we analyzed 88,835 articles published between 2003 and 2010 and found that the National Institutes of Health awardees who authored those papers occupy relatively stable positions of influence across all disciplines. We demonstrate that the values generated by this method strongly correlate with the opinions of subject matter experts in biomedical research and suggest that the same approach should be generally applicable to articles published in all areas of science. A beta version of iCite, our web tool for calculating Relative Citation Ratios of articles listed in PubMed, is available at https://icite.od.nih.gov.

External Tests of Peer Review Validity Via Impact Measures

“…Once again, many of these studies show significant type II errors (where unfunded applicants perform well) and sometimes only limited or no differences are found between funded and unfunded applicants with similar review scores or performance (Bornmann et al, 2008b(Bornmann et al, , 2010Pion and Cordray, 2008;Jacob and Lefgren, 2011a;Van den Besselaar and Sandstrom, 2015;Gush et al, 2017) although some similar comparisons do find differences (Robitaille et al, 2015). These ex post studies are related to the above ex ante results in that some literature has indicated that one of the strongest predictors of future citation performance is prior citation performance (Kaltman et al, 2014;Hutchins et al, 2016). Thus again, if peer review selects for applicants with higher previous productivity, it stands to reason that their post-funding productivity will be higher than unfunded applicants as well.…”

How to Choose Where to Publish Your Work

“…The latest of these is the iCite Relative Citation Ratio (RCR) metric. 12 The RCR was recently created by the National Institutes of Health (NIH), and represents a citation-based measure of the scientific influence of 1 or more articles. It is calculated as the citations per year of each paper, normalized to the citations per year received by NIH-funded papers in the same field and year.…”

“…This analysis can also be extended to the journal level. 12 Predatory publishers will claim "impact factors" for their journals. In most cases, these are bogus.…”

Applying Data Analytics and Visualization to Assessing the Research Impact of the Cancer Cell Biology (CCB) Program at the University of North Carolina at Chapel Hill

“…For example, The FWCI and CB from Scopus take into account the following factors to ensure "apples are compared to apples:" the year of publication, document type, and the disciplines associated with its source (Colledge and Verlinde 2014). The RCRs and NIH percentile generated by iCite quantify the influence of an article by comparing it with the average NIH-funded publications in the same field and year (Hutchins et al 2016). …”