Normalization of zero-inflated data: An empirical analysis of a new indicator family and its use with altmetrics data

Abstract

Recently, two new indicators (Equalized Mean-based Normalized Proportion Cited, EMNPC;Mean-based Normalized Proportion Cited, MNPC) were proposed which are intended for sparse scientometrics data. The indicators compare the proportion of mentioned papers (e.g.on Facebook) of a unit (e.g., a researcher or institution) with the proportion of mentioned papers in the corresponding fields and publication years (the expected values). In this study, we propose a third indicator (Mantel-Haenszel quotient, MHq) belonging to the same indicator family. The MHq is based on the MH analysisan established method in statistics for the comparison of proportions. We test (using citations and assessments by peers, i.e. F1000Prime recommendations) if the three indicators can distinguish between different quality levels as defined on the basis of the assessments by peers. Thus, we test their convergent validity. We find that the indicator MHq is able to distinguish between the quality levels in most cases while MNPC and EMNPC are not. Since the MHq is shown in this study to be a valid indicator, we apply it to six types of zero-inflated altmetrics data and test whether different altmetrics sources are related to quality. The results for the various altmetrics demonstrate that the relationship between altmetrics (Wikipedia, Facebook, blogs, and news data) and assessments by peers is not as strong as the relationship between citations and assessments by peers. Actually, the relationship between citations and peer assessments is about two to three times stronger than the association between altmetrics and assessments by peers.

Do altmetrics correlate with the quality of papers? A large-scale empirical study based on F1000Prime data

“…The reported Spearman coefficient for the correlation of Twitter counts and REF quality scores amounted to r s = 0.07. The results of this study accords also to the results of Bornmann and Haunschild [ 38 ] who investigated the relationship between altmetrics and assessments by peers. They found using the new MHq performance indicator which is especially suited for zero-inflated altmetrics data that “the relationship between altmetrics and peers’ assessments (one aspect of scientific quality) is not as strong as the relationship between peers’ assessments and citations”.…”

Introducing the EMPIRE Index: A novel, value-based metric framework to measure the impact of medical publications

“…Pairwise correlations can give useful insights into relationships between different metrics, but for the purposes of reducing data into composite scores it is helpful to understand the shared variance between multiple metrics. The exploratory factor analysis in our study produced findings consistent with those reported in previous literature [10][11][12][13]. Separating articles into those that were older (1H) and younger (2H) showed that citations (including policy and guideline mentions) and Mendeley readers consistently grouped into one factor; news, blogs, Wikipedia, and F1000Prime mentions grouped into a second factor; and Twitter (and, usually, Facebook) mentions comprised a third factor.…”

Can altmetrics reflect societal impact considerations?: Exploring the potential of altmetrics in the context of a sustainability science research center

“…For zero-inflated data, it is not possible to use methods for field normalization that are usually applied in bibliometrics (methods based on mean citations or citation percentiles; Bornmann et al, 2013). Since Bornmann and Haunschild (2018) and 2 The explanation of the MHq indicator has been mainly adopted from Bornmann and Haunschild (2018) and Haunschild and Bornmann (2018). Note: WoS papers (unit 0; neither accepted, nor rejected for funding), papers published by rejected applicants (unit 1), and papers published by accepted applicants (unit 2).…”

“…The MH analysis results in a summary odds ratio for multiple 2 × 2 cross tables, which Bornmann and Haunschild (2018) and Haunschild and Bornmann (2018) name MHq. For the comparison of the papers published by the applicants with reference sets in view of impact, the 2 × 2 cross tables (which are pooled) consist of the number of papers mentioned and not mentioned in subject category and publication year combinations f. In the 2 × 2 subjectspecific cross table (see Table 2), the cells a f , b f , c f , and d f , are defined as follows:…”