Methods to Account for Citation Inflation in Research Evaluation

Abstract

Quantitative research evaluation requires measures that are transparent, relatively simple, and free of disciplinary and temporal bias. We document and provide a solution to a hitherto unaddressed temporal biascitation inflationwhich arises from the basic fact that scientific publication is steadily growing at roughly 4% per year. Moreover, because the total production of citations grows by a factor of 2 every 12 years, this means that the real value of a citation depends on when it was produced. Consequently, failing to convert nominal citation values into real citation values produces significant mis-measurement of scientific impact. To address this problem, we develop a citation deflator method, outline the steps to generalize and implement it using the Web of Science portal, and analyze a large set of researchers from biology and physics to demonstrate how two common evaluation metricstotal citations and h-indexcan differ by a remarkable amount depending on whether the underlying citation counts are deflated or not. In particular, our results show that the scientific impact of prior generations is likely to be significantly underestimated when citations are not deflated, often by 100% or more of the nominal value. Thus, our study points to the need for a systemic overhaul of the counting methods used evaluating citation impactespecially in the case of researchers, journals, and institutionswhich can span several decades and thus several doubling periods.

Global citation inequality is on the rise

“…Given the increase in the production of papers and references over time, the general value of a citation will depend on when it was produced. In the scientometric literature, this phenomenon is known as citation inflation (49). While field normalization and especially normalization by publication year should remove a large part of the temporal effect on a citation's value (e.g., papers published in 2000 are weighted according to other papers published only in the same year), it is unclear if an additional element of inflation may be at play for certain types of papers.…”

When silence goes viral, Africa sneezes! A perspective on Africa's subdued research response to COVID-19 and a call for local scientific evidence

“…The relative significance of the various potential reasons highlighted here may vary considerably among countries and even individual researchers. Moreover, research productivity and quality, and their evaluation are quite complex subjects that depend on several factors and their interactions ( Korytkowski and Kulczycki, 2019 ; Petersen et al, 2019 ). Hence, no single reason can fully explain the limited research on COVID-19 in Africa.…”

Multiscale impact of researcher mobility

“…before (2) or after (þ) t * i,T . We also use a measure of total citation impact, S þ,2 i , useful for assessing the magnitude of the mobility effect in real terms by applying a 'citation deflator' that accounts for 'citation inflation' [50].…”

“…For comparison, we also considered an 'extensive' citation impact measure, as opposed to z p , which is an 'intensive' impact measure. The deflated citation countn i,p,t accounts for the fact that the total number of references produced by science is steadily growing with time (electronic supplementary material, figure S1(A)), the result of which is a 'citation inflation' measurement bias associated with the nominal citation count n i,p,t [50]. However, the 'deflated' variablen i,p,t is well-suited for comparisons of citation counts for articles published in different years, and so we instead tallyn i,p,t for each interval, defining the total as S þ,À…”