How does the normalization of data affect the ARWU ranking?

Jovanović, Milica; Jeremić, Veljko; Savić, Gordana; Bulajić, Milica; Martić, Milan

doi:10.1007/s11192-012-0674-0

Cited by 42 publications

(17 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The differences in the results between the covariate‐adjusted university ranking and the LR in Waltman et al. (; or the unadjusted ranking based on multilevel analysis) support the conclusion that, in the future, university rankings should always be conducted also using covariates that explain the variance (in addition to a variant without the covariates; see also Jovanovic et al, ). We use a specific set of covariates in this study.…”

Section: Discussionmentioning

confidence: 85%

Multilevel‐statistical reformulation of citation‐based university rankings: The Leiden ranking 2011/2012

Bornmann

Mutz

Daniel

2013

J Am Soc Inf Sci Tec

View full text Add to dashboard Cite

Since the 1990s, with the heightened competition and the strong growth of the international higher education market, an increasing number of rankings have been created that measure the scientific performance of an institution based on data. The Leiden Ranking 2011/2012 (LR) was published early in 2012. Starting from Goldstein and Spiegelhalter's (1996) recommendations for conducting quantitative comparisons among institutions, in this study we undertook a reformulation of the LR by means of multilevel regression models. First, with our models we replicated the ranking results; second, the reanalysis of the LR data showed that only 5% of the PPtop10% total variation is attributable to differences between universities. Beyond that, about 80% of the variation between universities can be explained by differences among countries. If covariates are included in the model the differences among most of the universities become meaningless. Our findings have implications for conducting university rankings in general and for the LR in particular. For example, with Goldstein‐adjusted confidence intervals, it is possible to interpret the significance of differences among universities meaningfully: Rank differences among universities should be interpreted as meaningful only if their confidence intervals do not overlap.

show abstract

Section: Discussionmentioning

confidence: 85%

Multilevel‐statistical reformulation of citation‐based university rankings: The Leiden ranking 2011/2012

Bornmann

Mutz

Daniel

2013

J Am Soc Inf Sci Tec

View full text Add to dashboard Cite

show abstract

“…Bookstein et al (2010) found unacceptably high year-to-year variances in the score of lower ranked universities in the THE, Jovanovic et al (2012) and Docampo (2013) found a large number of fluctuations and inconsistencies in the ranks of the ARWU. As we again observed puzzling results in the THE 2014–15 and the ARWU 2014 that were both published recently, we accordingly analyzed the fluctuations in score and rank of the THE and the ARWU.…”

Section: Introductionmentioning

confidence: 95%

Inconsistent year-to-year fluctuations limit the conclusiveness of global higher education rankings for university management

Sorz

Wallner

Seidler

et al. 2015

PeerJ

View full text Add to dashboard Cite

Backround. University rankings are getting very high international media attention, this holds particularly true for the Times Higher Education Ranking (THE) and the Shanghai Jiao Tong University’s Academic Ranking of World Universities Ranking (ARWU). We therefore aimed to investigate how reliable the rankings are, especially for universities with lower ranking positions, that often show inconclusive year-to-year fluctuations in their rank, and if these rankings are thus a suitable basis for management purposes.Methods. We used the public available data from the web pages of the THE and the ARWU ranking to analyze the dynamics of change in score and ranking position from year to year, and we investigated possible causes for inconsistent fluctuations in the rankings by the means of regression analyses.Results. Regression analyses of results from the THE and ARWU from 2010–2014 show inconsistent fluctuations in the rank and score for universities with lower rank positions (below position 50) which lead to inconsistent “up and downs” in the total results, especially in the THE and to a lesser extent also in the ARWU. In both rankings, the mean year-to-year fluctuation of universities in groups of 50 universities aggregated by descending rank increases from less than 10% in the group of the 50 highest ranked universities to up to 60% in the group of the lowest ranked universities. Furthermore, year-to-year results do not correspond in THES- and ARWU-Rankings for universities below rank 50.Discussion. We conclude that the observed fluctuations in the THE do not correspond to actual university performance and ranking results are thus of limited conclusiveness for the university management of universities below a rank of 50. While the ARWU rankings seems more robust against inconsistent fluctuations, its year to year changes in the scores are very small, so essential changes from year to year could not be expected. Furthermore, year-to-year results do not correspond in THES- and ARWU-Rankings for universities below rank 50. Neither the THES nor the ARWU offer great value as a tool for university management in their current forms for universities ranked below 50, thus we suggest that both rankings alter their ranking procedure insofar as universities below position 50 should be ranked summarized only in groups of 25 or 50. Additionally, the THE should omit the peer reputation survey, which most likely contributes heavily to the inconsistent year-to-year fluctuations in ranks, and ARWU should be published less often to increase its validity.

show abstract

“…It is necessary to fix one entity as a reference in the observed set using the I-distance methodology. The ranking of entities in the set is based on the calculated distance from the referent entity (Jovanovic et al 2012).…”

Section: Methodsmentioning

confidence: 99%

Towards an impartial Responsible Competitiveness Index: a twofold multivariate I-distance approach

Maričić

Kostić-Stanković

2014

Qual Quant

View full text Add to dashboard Cite

Issues surrounding the measurement of corporate social responsibility (CSR) are complex and far-reaching. The indicator based measurement of CSR has started to play an important role in defining the national level of CSR, but the controversy about the choice of indicators and weights assigned to them still remains. This paper has two goals: first, to analyze the Responsible Competitiveness Index (RCI), a composite index which shows how countries are performing in their efforts to promote and apply responsible business practices and second, to review the current indicators used in calculating this index in an effort to form a new theoretical framework. A statistical I-distance method which is able to overcome the problems concerning the aggregation of a composite index that uses subjective weighting factors was used. In a case study we scrutinized two data sets: the official one of 108 countries for the year 2007 was analyzed using I-distance methodology and the one we recreated of 98 countries for year 2013 was analyzed using a twofold I-distance approach. A twofold approach gave an insight to not just domain dynamics, but also to total score dynamic. The results presented in this research paper clearly point out countries with high level of social responsibility and a framework for improvement of the RCI. The framework proposed here can assist in the evaluation of other CSR indexes. Moreover, our approach can be a foundation for further academic CSR research which will be independent from subjectively formed weighting factors.

show abstract

How does the normalization of data affect the ARWU ranking?

Cited by 42 publications

References 21 publications

Multilevel‐statistical reformulation of citation‐based university rankings: The Leiden ranking 2011/2012

Multilevel‐statistical reformulation of citation‐based university rankings: The Leiden ranking 2011/2012

Inconsistent year-to-year fluctuations limit the conclusiveness of global higher education rankings for university management

Towards an impartial Responsible Competitiveness Index: a twofold multivariate I-distance approach

Contact Info

Product

Resources

About