Inconsistent year-to-year fluctuations limit the conclusiveness of global higher education rankings for university management

Abstract

Backround. University rankings are getting very high international media attention, this holds particularly true for the Times Higher Education Ranking (THE) and the Shanghai Jiao Tong University’s Academic Ranking of World Universities Ranking (ARWU). We therefore aimed to investigate how reliable the rankings are, especially for universities with lower ranking positions, that often show inconclusive year-to-year fluctuations in their rank, and if these rankings are thus a suitable basis for management purposes.Methods. We used the public available data from the web pages of the THE and the ARWU ranking to analyze the dynamics of change in score and ranking position from year to year, and we investigated possible causes for inconsistent fluctuations in the rankings by the means of regression analyses.Results. Regression analyses of results from the THE and ARWU from 2010–2014 show inconsistent fluctuations in the rank and score for universities with lower rank positions (below position 50) which lead to inconsistent “up and downs” in the total results, especially in the THE and to a lesser extent also in the ARWU. In both rankings, the mean year-to-year fluctuation of universities in groups of 50 universities aggregated by descending rank increases from less than 10% in the group of the 50 highest ranked universities to up to 60% in the group of the lowest ranked universities. Furthermore, year-to-year results do not correspond in THES- and ARWU-Rankings for universities below rank 50.Discussion. We conclude that the observed fluctuations in the THE do not correspond to actual university performance and ranking results are thus of limited conclusiveness for the university management of universities below a rank of 50. While the ARWU rankings seems more robust against inconsistent fluctuations, its year to year changes in the scores are very small, so essential changes from year to year could not be expected. Furthermore, year-to-year results do not correspond in THES- and ARWU-Rankings for universities below rank 50. Neither the THES nor the ARWU offer great value as a tool for university management in their current forms for universities ranked below 50, thus we suggest that both rankings alter their ranking procedure insofar as universities below position 50 should be ranked summarized only in groups of 25 or 50. Additionally, the THE should omit the peer reputation survey, which most likely contributes heavily to the inconsistent year-to-year fluctuations in ranks, and ARWU should be published less often to increase its validity.

Eigenvector-Based Centrality Measures for Temporal Networks

“…For example, when ≥ 10 −1 , the conditional centrality of Georgia Tech at times t and t+1 are typically very dissimilar, which appears to be a consistent property of conditional node-layer centralities in the limit → ∞. It is our believe that the highly volatile rankings for large do not appropriately describe the dynamics of department prestige [115]; this observation has motivated us to focus on the small (i.e., strong coupling) regime in this paper. The limiting cases → 0 + and → ∞, respectively, do a good job of describing regimes with very small and very large , but the intermediate ("transitional") regime between these extremes is not straightforward to interpret.…”

“…As increases, we found a transition, which we observed in two ways: the joint centrality localizes to just a few layers, and the conditional centrality begins to exhibit large fluctuations from one time layer to the next (i.e., trajectories no longer vary slowly). We believe this weak-coupling regime to be inappropriate for the MGP Ph.D. exchange network, as mathematics department prestige should not fluctuate wildly from one year to the next [115]. Instead, it should change on a slower time scale.…”

“…Rankings can of course change over time (either slowly or rapidly). To give an example that is directly relevant to one of our case studies, it was demonstrated recently that uncoupled university rankings can fluctuate significantly from year to year [115], which we consider to be a suboptimal description for the temporal evolution of departmental prestige.…”

Importance of collaboration with employers towards the rankings of higher education institutions

“…Researchers have paid attention to inconsistent year-to-year fluctuations limit in global higher education rankings for respective university management and researchers have suggested their findings for deeper use for university management and decision making (Sorz J., Wallner B., Seidler H., Fieder M., 2015). Researchers have analysed often asked questions on global university rankings correspondence to reality and respective consequences (Luca M., Smith J., 2013).…”

Research strengths identified by esteem and bibliometric indicators: a case study at the University of Vienna

“…Instead it is necessary to identify unique characteristics (what the University of Vienna in particular stands for) and to decide which research areas need to be emphasised. A research profile based on these reflections is a valuable component for the university's public image, for the correct adjustment of the self-perception, for resource decisions and benchmarking (Laudel and Weyer 2014;van Vught et al 2010). The challenge for Universities is to find the right balance between broadness and focus (Müller 2017).…”