On bibliometrics in academic promotions: a case study in computer science and engineering in Italy

Abstract

Due to its quantitative nature, bibliometrics is becoming increasingly popular among policy makers for academic hiring and career promotions. In this article, we quantitatively assess the impact that the granularity level in the classification of scientific areas would entail on research evaluation based on bibliometric indicators. We use as a case study the Italian national habilitation system (ASN), which classifies faculty members according to their academic discipline and relies on journal counts, citations, and h-indices as a basis for promoting tenure track researchers to associate professors and associate to full professors. The assessment checks whether the individual indicators of a researcher are above a certain threshold, e.g., the median over the population of researchers working in the same discipline. Our investigation focuses on two related, rather broad disciplines: computer science and computer engineering. We show that the ASN practice of using the same thresholds for all members of a scientific discipline can favor certain sub-communities that are characterized by higher bibliometric indicators, and disfavor others. We report evidence that up to 30% of Italian faculty members of certain sub-communities would see their indicators drop below the threshold, thus becoming not eligible for promotion, if the ASN were conducted on a more accurate, fine-grained classification. Conversely, in the same scenario, up to 11% of faculty members, in different sub-communities, would see their indicators rise above the threshold, granting them eligibility. Our data set includes 1,685 authors, 89,185 distinct publications, and 262,286 author-publication pairs.

Financial Cycles in the Economy and in Economic Research: A Case Study in China

“…Also, the connection between behavioral finance and decision making with cognitive biases was explored (Costa et al, 2017). Based on bibliometric indicators, the impact of the different granularity level in the classification of scientific fields on research evaluation was quantitatively estimated (Demetrescu et al, 2020).…”

Which Conference Is That? A Case Study in Computer Science

“…As an example, the accuracy of names in bibliographic data sources is investigated by Demetrescu et al [5]. Similarly, in order to study the use of bibliometrics in the CS ield in Italy, in [4] names of publication venues extracted from Scopus had to be matched against conferences and journals in a variety of lists. This required to face a matching problem akin to the one addressed in this work.…”

“…This required to face a matching problem akin to the one addressed in this work. The heuristic procedure used in [4] was based on a multi-pass greedy approach: entries that were matched incorrectly at each phase were reinserted into the pool of unmatched ones for further processing in later phases. This required, however, signiicant manual intervention.…”

The role of citation networks to explain academic promotions: an empirical analysis of the Italian national scientific qualification

“…Several studies have attempted to investigate what are the best predictors of the qualification attainment, and what is the role of bibliometrics in academic promotions and career outcomes (see, e.g., Bedogni et al, 2022;Demetrescu et al, 2020;Jensen et al, 2008;Marini, 2017;Poggi et al, 2019;Tregellas et al, 2018;Vieira et al, 2014), but none of these tried to insert predictors based on citation network indices, as discussed in details in "Related work". As pointed out by Ding (2011), in fact, citation networks contain relevant information to quantify and qualify citations and citation structure, providing a sort of endorsement pattern among the nodes.…”

“…Among the CDs we choose to focus on the Computer Science, a discipline that has been studied with different purposes both in the Italian context (see, e.g., Demetrescu et al, 2020;Di Iorio et al, 2019) and in the international one (Ding, 2011;Ding et al, 2009). Among NDs we concentrate on Statistics, a discipline that does not have a strong bibliometric tradition, but is going to adopt a citation-based approach in the next Italian assessment for institutions (i.e.…”