Do Rankings Reflect Research Quality?

Frey, Bruno S.; Rost, Katja

doi:10.2139/ssrn.1280763

Cited by 47 publications

(47 citation statements)

References 77 publications

(29 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In some cases, scholars have confirmed the validity of rankings: for example, some authors have found that ranking Australian universities by reputation, using an international survey of senior academics, and ranking by research performance measures produces positively correlated results (Williams and Van Dyke, 2008). In other cases, on the contrary, authors have argued that there is no correlation between different dimensions of performance: for example ranking individual scientists and universities in economics by publication and by membership of important journals gives substantially different results, raising doubts about the robustness of ranking universities only by publications (Frey and Rost, 2008).…”

Section: Diversity and Unidimensional Rankingmentioning

confidence: 99%

Characterizing the European university system: a preliminary classification using census microdata

Bonaccorsi

Daraio

2009

Sci. and Pub. Pol.

View full text Add to dashboard Cite

The paper addresses the issue of characterization and classification of universities in the European system, by using the recently developed Aquameth dataset. Preliminary cluster analysis based on structural variables identifies systematic differences in size. However, this structural distinction is associated with differences in strategic orientation of universities (towards research or towards teaching, respectively) only in a few countries. In most European countries there are no discernible differences across universities along these dimensions. The paper argues that countries in which universities are more differentiated according to research or teaching dimensions have implemented differentiation policies through a variety of policy instruments. In turn, these countries also are ranked high in international rankings of universities. This suggests a structural linkage between the poor performance of European universities in research-based rankings and the lack of differentiation. NIVERSITIES ARE BECOMING a crucial focus for analysis and policy-making in most advanced countries. In contrast to the rising attention to the subject, the state of the art of the analysis of universities, from an economic, institutional and policy-oriented point of view, is still quite underdeveloped. One of the reasons is that comparability of data across countries is made extremely difficult by institutional and definitional differences, so that it is problematic to test hypotheses on a large scale and develop broad conceptualizations. Most comparative research is qualitative and based on case studies, or careful examination of national policies. This situation is even worse for European universities, due to deep historical differences across countries. Quite remarkably, this state of the art is harmful for evidence-based policy-making.In recent years, however, some opportunities for large-scale comparative analysis based on quantitative information have been opened through a series of pioneering studies producing original datasets.This paper makes use of one of these datasets (developed within the Aquameth project 1 ) in order to address the issue of diversity in the European university system. Using clustering techniques we obtain several classification schemes. Rather than using these schemes as a 'true' classification, we use them in order to introduce the debated issue of research-intensive universities.The paper is organized as follows:• The first section discusses some theoretical arguments for diversity and differentiation in the higher education system. • The second section introduces methodological and data issues. • The third section presents the results on European universities. • The fourth section offers a perspective on two country cases of keen interest for opposite reasons, United Kingdom and Italy. • The final section discusses policy implications and concludes suggesting further developments for research.

show abstract

Section: Diversity and Unidimensional Rankingmentioning

confidence: 99%

Characterizing the European university system: a preliminary classification using census microdata

Bonaccorsi

Daraio

2009

Sci. and Pub. Pol.

View full text Add to dashboard Cite

show abstract

“…In the same vein, economist Bruno Frey found »that such rankings do not effectively measure research quality«, and »career decisions based on rankings are dominated by chance« (Frey/Rost 2008).…”

Section: Forummentioning

confidence: 99%

Forum: Assessing economic research and the future of heterodox economics. Failures and alternatives of journals, departments, and scholars rankings

Elsner¹,

Lee²

2010

European Journal of Economics and Economic Policies: Intervention

View full text Add to dashboard Cite

Journal and department rankings and the ruling game of mainstream economicsEvaluating economic research today is a most contested fi eld. Th is applies, most notably, since individual careers of a whole generation of critical young economists are aff ected. And it applies in economics, perhaps more than in any other discipline, since it is the most important academic discipline for the ideological legitimization of capitalism and one of the few, perhaps the only, fundamentally divided and contested discipline. What the ruling forces of the economy, of professional politics, of science administration, and particularly of economic science have made out of the complex issues and processes of evaluating research quality is reducing them down to a simplistic, allegedly exact, objective, and obvious, but fundamentally mistaken procedure of a one-dimensional ranking of quantitative domination, a cumulative dictatorship of mass. And this is done in surprisingly unprofessional ways, subject to many obvious misconceptions and failures. For example, the Also? Der alte fruchtlose Streit eines ›Entweder/Oder‹ ist, wie schon Schumpeter in Zusammenhang mit der Menger/Schmoller-Auseinandersetzung sagte, »im Wesentlichen eine Geschichte vergeudeter Energie« (Schumpeter 1965: 994). Die Fragen bezüglich Pluralismus, Mainstream-Ansprüchen und dergleichen bleiben aktuell und Gegenstand fortlaufender Diskussionen. Die FAZ/HB-Konfrontation hat allerdings wenig dazu beigetragen.

show abstract

“…These decisions work to check on the correctness of submitted papers, but they also let other scientists, administrators, and funding agencies know what is considered novel, important, and worthy of study (e.g. Brown 2014; Frey and Katja 2010; Katerattanakul et al 2005; Weingart 2005). Highly ranked journals thus exert considerable influence on the direction that scientific disciplines move.…”

Section: Introductionmentioning

confidence: 99%

“…Abbott et al (2010), Adam (2002), Chapron and Husté (2006), Egghe (2006), Engemann and Wall (2009), Frey and Katja (2010), Garfield (1972), Hirsch (2005), Moed (2008) Palacios-Huerta and Volij (2004, 2014), Tort et al (2012) and Hutchins et al (2015). …”

mentioning

confidence: 99%

Neophilia ranking of scientific journals

Packalén

Bhattacharya

2016

Scientometrics

View full text Add to dashboard Cite

The ranking of scientific journals is important because of the signal it sends to scientists about what is considered most vital for scientific progress. Existing ranking systems focus on measuring the influence of a scientific paper (citations)—these rankings do not reward journals for publishing innovative work that builds on new ideas. We propose an alternative ranking based on the proclivity of journals to publish papers that build on new ideas, and we implement this ranking via a text-based analysis of all published biomedical papers dating back to 1946. In addition, we compare our neophilia ranking to citation-based (impact factor) rankings; this comparison shows that the two ranking approaches are distinct. Prior theoretical work suggests an active role for our neophilia index in science policy. Absent an explicit incentive to pursue novel science, scientists underinvest in innovative work because of a coordination problem: for work on a new idea to flourish, many scientists must decide to adopt it in their work. Rankings that are based purely on influence thus do not provide sufficient incentives for publishing innovative work. By contrast, adoption of the neophilia index as part of journal-ranking procedures by funding agencies and university administrators would provide an explicit incentive for journals to publish innovative work and thus help solve the coordination problem by increasing scientists' incentives to pursue innovative work.

show abstract

Do Rankings Reflect Research Quality?

Cited by 47 publications

References 77 publications

Characterizing the European university system: a preliminary classification using census microdata

Characterizing the European university system: a preliminary classification using census microdata

Forum: Assessing economic research and the future of heterodox economics. Failures and alternatives of journals, departments, and scholars rankings

Neophilia ranking of scientific journals

Contact Info

Product

Resources

About