Integrated impact indicators compared with impact factors: An alternative research design with policy implications

Abstract

In bibliometrics, the association of "impact" with centraltendency statistics is mistaken. Impacts add up, and citation curves therefore should be integrated instead of averaged. For example, the journals MIS Quarterly and Journal of the American Society for Information Science and Technology differ by a factor of 2 in terms of their respective impact factors (IF), but the journal with the lower IF has the higher impact. Using percentile ranks (e.g., top-1%, top-10%, etc.), an Integrated Impact Indicator (I3) can be based on integration of the citation curves, but after normalization of the citation curves to the same scale. The results across document sets can be compared as percentages of the total impact of a reference set. Total number of citations, however, should not be used instead because the shape of the citation curves is then not appreciated. I3 can be applied to any document set and any citation window.The results of the integration (summation) are fully decomposable in terms of journals or institutional units such as nations, universities, and so on because percentile ranks are determined at the paper level. In this study, we first compare I3 with IFs for the journals in two Institute for Scientific Information subject categories ("Information Science & Library Science" and "Multidisciplinary Sciences"). The library and information science set is additionally decomposed in terms of nations. Policy implications of this possible paradigm shift in citation impact analysis are specified.

A review of theory and practice in scientometrics

“…A specific metric, based on percentiles, called I3 has been proposed by Leydesdorff (2011b) which combines relative citation impact with productivity in terms of the numbers of papers but is normalised through the use of percentiles (see Section 4.3 for more explanation).…”

Comparative End-To-End Evaluation of Research Organizations

“…Thus, C combines impact i and output P by weighing each publication with its citation impact. The I3 indicator 6 combines normalized impact and output and is therefore a first-order indicator of performance. The exergy indicator of Prathap 7 is a second-order indicator of performance.…”

Comparative Study of Trace Metrics between Bibliometrics and Patentometrics

“…Although the aforementioned indicators have been widely applied in literature and bibliographic databases, they have some limitations. For example, the skewness of citation distributions is ignored in the citation counts and IF (Leydesdorff & Bornmann, 2011) and the h-index are somewhat inconsistent (Waltman & van Eck, 2012) and insensitive (Bornheim et al, 2008;Egghe, 2006;Kuan, Huang, & Chen, 2011).…”

“…Bornmann et al (2010) proposed three areas under the rankcitation curve: an area that has citations lower than the h-index (h2 lower; t-area in Figure 1), a square area captured by the h-index (h2 center; h-area in Figure 1), and an area in which citations exceed the h-index (h2 upper, e-area in Figure 1). Leydesdorff and Bornmann (2011) proposed using integrated impact indicators (I3s) instead of the IF for evaluating academic performance (Leydesdorff & Bornmann, 2011).…”

“…According to the definition of I3 (Leydesdorff & Bornmann (2011), an I3-type indicator can be formalized as where X i indicates the percentile ranks and f (X i ) indicates the frequencies of the ranks, i in [1,C] indicates the percentile rank classes. The number C is the total classes that the measures X i are divided into, each with a scoring function f (X i ) or weight (w i ).…”