Quantifying and suppressing ranking bias in a large citation network

Abstract

It is widely recognized that citation counts for papers from different fields cannot be directly compared because different scientific fields adopt different citation practices. Citation counts are also strongly biased by paper age since older papers had more time to attract citations. Various procedures aim at suppressing these biases and give rise to new normalized indicators, such as the relative citation count. We use a large citation dataset from Microsoft Academic Graph and a new statistical framework based on the Mahalanobis distance to show that the rankings by well known indicators, including the relative citation count and Google's PageRank score, are significantly biased by paper field and age. We propose a general normalization procedure motivated by the z-score which produces much less biased rankings when applied to citation count and PageRank score.

Unbiased evaluation of ranking metrics reveals consistent performance in science and technology citation data

“…For the APS, HEP, and PAT data, we use W = 1, 000, W = 2, 000, and W = 15, 000, respectively, which is roughly proportional to the number of nodes in each dataset. As shown in Mariani et al (2016; Ren (2019), rescaling significantly reduces the magnitude of the age bias-and, in the case of Vaccario et al (2017), of the age and field bias-of citation count and PageRank. We use this technique here to rescale all ranking metrics introduced above, and in turn compare their performance with original non-rescaled metrics.…”

“…The choice of the number of age groups G used in the computation of NIR is a compromise between improving the temporal resolution (lowering the time duration of each group) by increasing G and limiting the natural statistical variability of N z (g) by keeping G low. We use G = 40 adopted in previous literature (Mariani et al, 2016;Vaccario et al, 2017); other choices lead to qualitatively similar results. Note that due to the introduction of a penalizing factor, NIR cannot be higher than IR for a given ranking.…”

“…There are various questions that remain open for further research. First, to extend our analysis to more than one kind of bias (such as the age and field bias common in scholarly citation data (Vaccario et al, 2017)) and to generalize it to cases where the kind of bias is not explicitly known. Second, identification rate (referred to as recall in information filtering and statistical learning) is just one of various performance metrics (see (Dunaiski et al, 2018) for an overview of possibilities); we thus need to study how to account for bias in these other metrics.…”

“…Task 2 The need for unbiased evaluation, which is common in scientometrics (Mutz and Daniel, 2012;Mingers and Leydesdorff, 2015;Vaccario et al, 2017), motivates the second task: To rank the expert-selected nodes as high as possible whilst requiring that the ranking metric is unbiased. Since most structural ranking metrics are biased (see Appendix A for a demonstration), their evaluation must include a penalty for the performance gained thanks to the methods' bias.…”

Comparison of bibliographic data sources: Implications for the robustness of university rankings

“…MSA obtains bibliographic data through web pages crawled by Bing. MSA's emergence and fast growth (at a rate of 1.3 million records per month, according to has spurred its use in several bibliometrics studies (De Domenico et al, 2016;Portenoy et al, 2016;Sandulescu & Chiru, 2016;Wesley-Smith et al, 2016;Vaccario et al, 2017;Portenoy & West, 2017;Effendy & Yap, 2017). At the same time, various papers have tracked changes in the MSA database and compared it to other bibliographic sources (Paszcza, 2016;Harzing, 2016;Harzing & Alakangas, 2017a;Harzing & Alakangas, 2017b;.…”