Qualifying the performance evaluation of Big Science beyond productivity, impact and costs

Abstract

The use of quantitative performance measures to evaluate the productivity, impact and quality of research has spread to almost all parts of public R&D systems, including Big Science where traditional measures of technical reliability of instruments and user oversubscription have been joined by publication counts to assess scientific productivity. But such performance assessment has been shown to lead to absurdities, as the calculated average cost of single journal publications easily may reach hundreds of millions of dollars. In this article, the issue of productivity and impact is therefore further qualified by the use of additional measures such as the immediacy index as well as network analysis to evaluate qualitative aspects of the impact of contemporary Big Science labs. Connecting to previous work within what has been called "facilitymetrics", the article continues the search for relevant bibliometric measures of the performance of Big Science labs with the use of a case study of a recently opened facility that is advertised as contributing to "breakthrough" research, by using several more measures and thus qualifying the topic of performance evaluation in contemporary Big Science beyond simple counts of publications, citations, and costs.

A critical analysis of large scale research infrastructures’ performances

“…The purpose of experimental platforms consists primarily in providing scientist-users access to highly specialised scientific instrumentation and experimental conditions beyond the reach of most university organisations. In areas such as neutron scattering, synchrotron radiation and free electron laser facilities, experimental platforms require intense collaboration between permanent scientists and external users, which is essential to produce science (Hackett et al , 2004; Hallonsten, 2016), and facilities need to be highly specialised while remaining attractive to broad scientific communities (Heidler and Hallonsten, 2015).…”

Measuring national capability over big science’s multidisciplinarity: A case study of nuclear fusion research

“…It attracts researchers of diverse disciplines and enhances scientific collaborations. Despite its scientific importance, little attention has been paid to examining how large facilities raise national research capacities because of difficulties in unraveling the multidisciplinarity of big science [9][10][11]. Moreover, national research capacity is difficult to quantify as it is built on the complex interactions between private and public domains [12,13].…”