The heterogeneous landscape of bibliometric indicators: Evaluating models for allocating resources at Swedish universities

Abstract

The use of bibliometric indicators on individual and national levels has gathered considerable interest in recent years, but the application of bibliometric models for allocating resources at the institutional level has so far gathered less attention. This article studies the implementation of bibliometric measures for allocating resources at Swedish universities. Several models and indicators based on publications, citations, and research grants are identified. The design of performance-based resource allocation across major universities is then analysed using a framework from the field of evaluation studies. The practical implementation, the incentives as well as the 'ethics' of models and indicators, are scrutinized in order to provide a theoretically informed assessment of evaluation systems. It is evident that the requirements, goals, possible consequences, and the costs of evaluation are scarcely discussed before these systems are implemented. We find that allocation models are implemented in response to a general trend of assessment across all types of activities and organizations, but the actual design of evaluation systems is dependent on size, orientation, and the overall organization of the institution in question.

Balancing accountability and trust: university reforms in the Nordic countries

“…The consequences of the evaluations, in terms of 'carrots and sticks', also vary, from a primarily 'shaming' strategy for those who underperform to financial rewards for those who excel (Karlsson et al 2014). Furthermore, many HEIs have introduced performance-based funding systems that rely on publication and citation data (Hammarfelt et al 2016).…”

Taking Comfort in Points: The Appeal of the Norwegian Model in Sweden

“…A survey conducted among bibliometricans at Swedish higher education institutions (HEIs) in 2014 found that 11 out of 27 universities used the "Norwegian model" or parts of it (Hammarfelt, Nelhans, Eklund, & Åström, 2016). The indicator is used for allocating resources either on the level of the university as a whole or in selected faculties.…”

“…An example of this phenomenon, what we may call the "after-life of indicators", is the ERA-list in Australia which was formally used only for a short period, yet this ranked list of journals still plays an important role among researchers (Hammarfelt & Haddow, 2018). Previous studies has pointed to the problem of the "Norwegian publication indicator" being used in unintended ways (Aagaard, 2015;Hammarfelt et al, 2016), and actions, such as "inter-instutional learning areas" has been implemented in order to limit inappropriate use (MLE on Performance-based Funding of Public Research Organisations 2018.) Such efforts are commendable, and yet there is a risk that Norwegian publication…”

Perspectives from a water research institute on knowledge management for sustainable water management

“…The deluge of scientific and technical literature (including that related to SWM) has been linked with trends in metrics and incentives in academics and funding agencies [Benedictus et al, 2016;Hammarfelt et al, 2016]. In addition, bibliometric indicators have been criticized as having an inhibitory effect on inter-and transdisciplinary research [Zilahy et al, 2009], which is clearly an issue for SWM.…”