Over-optimization of academic publishing metrics: observing Goodhart’s Law in action

Abstract

Background The academic publishing world is changing significantly, with ever-growing numbers of publications each year and shifting publishing patterns. However, the metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades. Moreover, recent studies indicate that these metrics have become targets and follow Goodhart’s Law, according to which, “when a measure becomes a target, it ceases to be a good measure.” Results In this study, we analyzed >120 million papers to examine how the academic publishing world has evolved over the last century, with a deeper look into the specific field of biology. Our study shows that the validity of citation-based measures is being compromised and their usefulness is lessening. In particular, the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers. Citation-based metrics, such citation number and h-index, are likewise affected by the flood of papers, self-citations, and lengthy reference lists. Measures such as a journal’s impact factor have also ceased to be good metrics due to the soaring numbers of papers that are published in top journals, particularly from the same pool of authors. Moreover, by analyzing properties of >2,600 research fields, we observed that citation-based metrics are not beneficial for comparing researchers in different fields, or even in the same department. Conclusions Academic publishing has changed considerably; now we need to reconsider how we measure success.

Citation counts and journal impact factors do not capture some indicators research quality in the behavioral and brain sciences

“…Goodhart's Law states that a measure ceases to be a good measure when it becomes a target (Goodhart, 1975). Even if citation counts and journal impact factors were once reasonable proxies for quality (though there is no evidence to suggest this), it is now clear that they have become targets and as a result lack validity as measures of faculty performance (Fire & Guestrin, 2019). Our findings are consistent with this conclusion: Neither citation counts nor impact factors were meaningfully related to research quality, as defined by errors in statistical reporting or strength of evidence (Study 1), estimated replicability as captured by the replicability index (Study 2), or actual replicability (Study 3).…”

Preprints in Medicine: Useful or Harmful?

“…In previous decades, the world of scientific publishing has changed enormously with an explosion in the number of publishers and journals. Therefore, there has been an exponential increase in the number of scientific papers published (4). The way that information is distributed has also changed, with an increasing number of journals being available in open access format.…”

A simple model suggesting economically rational sample-size choice drives irreproducibility

“…Proxyeconomics refers to any competitive societal system in which an abstract goal (here: scientific progress) is promoted using competition based on proxy measures (here: publications). In such cases, the measures or system may become corrupted due to an overoptimization toward the proxy measure [44][45][46][47][48]. As discussed above, such systems have the general potential to create a situation of limited individual agency and system-level lock-in [23].…”