How robust are journal rankings based on the impact factor? Evidence from the economic sciences

Seiler, Christian; Wohlrabe, Klaus

doi:10.1016/j.joi.2014.09.001

Cited by 22 publications

(13 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These differences may be due to: (1) a delayed update of the database (the IPPs were published by Scopus in June 2015), and (2) a larger set of citing sources and documents (with Scopus, it is not possible to limit the citation report to particular citing sources or documents). Similar differences between official and observed values have been found and discussed, for instance, by Leydesdorff and Opthof ( 2010 ), Stern ( 2013 ) and Seiler and Wohlrabe ( 2014 ). Nonetheless, in this case the ACPP should, theoretically, represent a 3-year synchronous impact factor for the year 2014 (Ingwersen et al 2001 ; Ingwersen 2012 ) in that we considered only citations received during 2014 of papers published within the previous 3 years.…”

Section: Two Empirical Studiessupporting

confidence: 87%

A theoretical model of the relationship between the h-index and other simple citation indicators

Barsotti

Lando

2017

Scientometrics

View full text Add to dashboard Cite

Of the existing theoretical formulas for the h-index, those recently suggested by Burrell (J Informetr 7:774–783, 2013b) and by Bertoli-Barsotti and Lando (J Informetr 9(4):762–776, 2015) have proved very effective in estimating the actual value of the h-index Hirsch (Proc Natl Acad Sci USA 102:16569–16572, 2005), at least at the level of the individual scientist. These approaches lead (or may lead) to two slightly different formulas, being based, respectively, on a “standard” and a “shifted” version of the geometric distribution. In this paper, we review the genesis of these two formulas—which we shall call the “basic” and “improved” Lambert-W formula for the h-index—and compare their effectiveness with that of a number of instances taken from the well-known Glänzel–Schubert class of models for the h-index (based, instead, on a Paretian model) by means of an empirical study. All the formulas considered in the comparison are “ready-to-use”, i.e., functions of simple citation indicators such as: the total number of publications; the total number of citations; the total number of cited paper; the number of citations of the most cited paper. The empirical study is based on citation data obtained from two different sets of journals belonging to two different scientific fields: more specifically, 231 journals from the area of “Statistics and Mathematical Methods” and 100 journals from the area of “Economics, Econometrics and Finance”, totaling almost 100,000 and 20,000 publications, respectively. The citation data refer to different publication/citation time windows, different types of “citable” documents, and alternative approaches to the analysis of the citation process (“prospective” and “retrospective”). We conclude that, especially in its improved version, the Lambert-W formula for the h-index provides a quite robust and effective ready-to-use rule that should be preferred to other known formulas if one’s goal is (simply) to derive a reliable estimate of the h-index.

show abstract

Section: Two Empirical Studiessupporting

confidence: 87%

A theoretical model of the relationship between the h-index and other simple citation indicators

Barsotti

Lando

2017

Scientometrics

View full text Add to dashboard Cite

show abstract

“…Hudson (2013) predicted the Keele journal classification by regression methods and measured the probability of correct classification. The robustness of rankings of economics journals was studied by Seiler and Wohlrabe (2014), where the rankings were based on the JIF, by considering the skewness of the distribution of citations to the articles in a journal. Horrace and Parmeter (2017) revisited the Stern study (2013) referred to above, but performed multivariate inference to control the overall error rate of the testing procedure.…”

Section: Introductionmentioning

confidence: 99%

Uncertainty and the ranking of economics journals

Lyhagen

Ahlgren

2020

Scientometrics

View full text Add to dashboard Cite

Journal rankings often show significant changes compared to previous rankings. This gives rise to the question of how well estimated the rank of a journal is. In this contribution, we consider uncertainty in a ranking of economics journals. We use the invariant method of Pinski and Narin to rank the journals. We propose an uncertainty measure, which is based on a bootstrap approach. The measure is the average absolute change in rank, which we see as a reasonable uncertainty measure regarding rankings. We further calculate, based on the bootstrap method, 95% confidence interval for the observed values of the invariant method. We show that ranks of the highest, as well as the lowest, ranked journals are well estimated, while there is a high degree of uncertainty regarding the rank of many mid-ranked journals. The distribution of the underlying measure is useful for identifying groups of journals that are more or less of the same quality (from the point of view of the invariant measure). The journal with the highest observed value of the invariant measure, Journal of Political Economy, has the best performance and constitutes a singleton, whereas Quarterly Journal of Economics and Econometrica form the next group (there is a slight overlap between the two with respect to confidence intervals). The journals ranked between about 190–230 form another group in which there are no major quality differences between the journals, as the confidence intervals are overlapping.

show abstract

“…This is in line with the results reported in Oswald (2007) who documents that top-cited articles are also published in lower ranked journals. Seiler and Wohlrabe (2014) show that for almost all journals in economics the citation distribution is skewed, i.e. dominated by the respective top-cited article.…”

Section: Datamentioning

confidence: 99%

“…In RePEc, all citations are related to total number of registered articles in a journal. For further details on RePEc see Zimmermann (2013) and Seiler and Wohlrabe (2012).…”

Section: Datamentioning

confidence: 99%

Citations, journal ranking and multiple authorships reconsidered: evidence from almost one million articles

Sommer

Wohlrabe

2016

Applied Economics Letters

Self Cite

View full text Add to dashboard Cite

In this paper we reconsider the investigation by Moosa (2016) using a much larger data set of almost one million articles listed in RePEc. This article provides new insights into the effects of co-authorship on citation counts and the correlation between quality of papers and quality of the publishing journal. Our evidence is partially in contrast to the results reported in Moosa (2016). We find a positive correlation between the h-index of a journal and the quality of papers measured in terms of citations. This correlation becomes almost perfect using a non-linear model. Results from a regression of citation counts on the number of authors show evidence of a positive and significant effect of coauthorship on the quality of a paper when time effects and large sets of top-cited articles are taken into account. The inclusion of time effects and the large data set, that allows to differentiate between top-cited cohorts, add further insights to the existing literature.

show abstract

How robust are journal rankings based on the impact factor? Evidence from the economic sciences

Cited by 22 publications

References 13 publications

A theoretical model of the relationship between the h-index and other simple citation indicators

A theoretical model of the relationship between the h-index and other simple citation indicators

Uncertainty and the ranking of economics journals

Citations, journal ranking and multiple authorships reconsidered: evidence from almost one million articles

Contact Info

Product

Resources

About