Citation counts and journal impact factors do not capture some indicators research quality in the behavioral and brain sciences

Abstract

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed regarding whether they are valid proxies for key aspects of research quality. We use data from three large scale studies to assess whether citation counts and impact factors predict four indicators of aspects of research quality: (1) the number of statistical reporting errors in a paper, (2) the evidential value of the reported data, (3) the expected replicability of reported research findings in peer reviewed journals, and (4) the actual replicability of a given experimental result. Both citation counts and impact factors were weak and inconsistent predictors of research quality, so defined, and sometimes negatively related to quality. Our findings impugn the validity of citation data and impact factors as indices of research quality and call into question their usefulness in evaluating scientists and their research. In light of these results, we argue that research evaluation should instead focus on the process of how research is conducted and incentivize behaviors that support open, transparent, and reproducible research.

A Memory-Theoretic Account of Citation Propagation

“…The exposure model clearly challenges the common belief that citation counts reflect the underlying scientific merit of the cited work. Unfortunately, the emerging literature examining the relationship between scientific quality and citation counts has largely failed to find consistent relationships between measures of research quality and citation counts (Dougherty & Horne, 2019;Nieminen et al, 2006;Paiva et al, 2012;Ruano et al, 2018;West & McIlwaine, 2002). (Petersen et al, 2014).…”

“…The simulations presented in the left column of Figure 1 and (Aksnes et al, 2019;Dougherty & Horne, 2019). The question we address here is the degree to which the modeled distributions shift when the playing field is uneven.…”

“…Although it is tempting to maintain the belief that citation counts reflect some dimension of quality, the scientific literature examining this issue is equivocal. For instance, Dougherty and Horne (2019) examined the relationship between citation counts and objective measures of article quality such as the presence of inconsistencies in the reporting of statistics, the degree of evidential support provided by the data, and replicability of scientific findings. The revealed relationships were weak at best and definitely not meaningful, and in some cases inversely related to quality.…”

Ten steps toward a better personality science - how quality may be rewarded more in research evaluation

“…Accordingly, some research suggests that the impact factor correlates, if anything, with the unreliability of the scientific findings in a journal (Brembs, Button, & Munafò, 2013;Dougherty & Horne, 2019). We therefore argue in favor of assessing the quality of individual papers directly, by explicitly weighting them with their good science merits, and pay much less attention to the journals in which they were published.…”

An immunization program against the COVID-19 infodemic

“…As the biomedical community was developing its critical thinking skills during these past decades, the pandemic got us by surprise at a time we were not there yet. We were not sufficiently immunized against a dysfunctional ecosystem of sharing scientific information that values prestige, brand and citation over quality cf [15][16][17][18] . The plethora of fast tracked publication easily becomes viral through social media and instant messaging apps.…”