Robustness of journal rankings by network flows with different amounts of memory

Abstract

As the number of scientific journals has multiplied, journal rankings have become increasingly important for scientific decisions. From submissions and subscriptions to grants and hirings, researchers, policy makers, and funding agencies make important decisions with influence from journal rankings such as the ISI journal impact factor. Typically, the rankings are derived from the citation network between a selection of journals and unavoidably depend on this selection. However, little is known about how robust rankings are to the selection of included journals. Here we compare the robustness of three journal rankings based on network flows induced on citation networks. They model pathways of researchers navigating scholarly literature, stepping between journals and remembering their previous steps to different degree: zero-step memory as impact factor, one-step memory as Eigenfactor, and two-step memory, corresponding to zero-, first-, and second-order Markov models of citation flow between journals. We conclude that higher-order Markov models perform better and are more robust to the selection of journals. Whereas our analysis indicates that higher-order models perform better, the performance gain for the secondorder Markov model comes at the cost of requiring more citation data over a longer time period.Science builds on previous science in a recursive quest for new knowledge (1-3). Researchers put great effort into finding the best work by other researchers and into achieving maximum visibility of their own work. Therefore, they both search for good work and seek to publish in prominent journals. Inevitably, where researchers publish becomes a proxy for how good their work is, which in turn influences decisions regarding hiring, promotion, and tenure, as well as university rankings and academic funding (4, 5). As a consequence, researchers depend on the perceived importance of the journals they publish in. While actually reading the work published in a journal is the only way to qualitatively evaluate the scientific content, different metrics are nevertheless used to quantitatively assess the importance of scientific journals (6-13). In different ways, the metrics extract information from the network of citations between articles published in the journals.In this paper, we analyze three flow-based journal rankings (12-14) that at different order of approximations seek to capture the pathways of researchers navigating scholarly literature. Specifically, the metrics measure the journal visit frequency of random walk processes that correspond to zero-, first-, and second-order Markov models. That is, given a citation network between journals and a random walker following the citations, movements in a zero-order model are independent of the cur- * Electronic address: ludvig.bohlin@physics.umu.se; Corresponding author † Electronic address: a.viamontes.esquivel@physics.umu.se ‡ Electronic address: andrea.lancichinetti@physics.umu.se § Electronic address: martin.rosvall@physics.umu.se rently visited journal, moveme...

Memory in network flows and its effects on spreading dynamics and community detection

“…When going from rankings based on counting links to measuring the average visit frequency of a random walker on a standard network,that is, calculating the PageRank 6 , the importance of neighbours becomes evident. Similarly, when going to PageRank on a network with secondorder memory, the amount of flow received from neighbours also depends on the flow's origin 15,48 . We define a generalized secondorder PageRank as the stationary solution of equation (6) …”

Quantifying Success in Science: An Overview

“…Zhang proposed the HR-PageRank algorithm to evaluate journal impact via weighted PageRank according to the author's H-index, and relevance between citing and cited papers [108]. Bohlin et al [109] studied the different performances of zero-(the classical Markov model), firstand second-order Markov model while ranking journals and found that higher-order Markov models performed better and were more robust. Some evaluation methods consider the structural position of journals in the journal citation network.…”

Does “Evaluating Journal Quality and the Association for Information Systems Senior Scholars Journal Basket…” Support the Basket with Bibliometric Measures?

“…In that case, their connections would be too tenuous to regard them as providing meaningful scholarly cross-fertilization. We wanted to ensure that the paths needed to reach the Lowry et al sample were no longer than two steps in each direction (for the rationale: Bohlin, Viamontes Esquivel, Lancichinetti, & Rosvall, 2016). That is, each journal should be able to reach every Lowry et al journal with no more than one indirect cross-citation.…”