Heuristics as conceptual lens for understanding and studying the usage of bibliometrics in research evaluation

Abstract

While bibliometrics are widely used for research evaluation purposes, a common theoretical framework for conceptually understanding, empirically studying, and effectively teaching its usage is lacking. In this paper, we outline such a framework: the fast-and-frugal heuristics research program, proposed originally in the context of the cognitive and decision sciences, lends itself particularly well for understanding and investigating the usage of bibliometrics in research evaluations. Such evaluations represent judgments under uncertainty in which typically not all possible options, their consequences, and those consequences' probabilities of occurring may be known. In these situations of incomplete information, candidate descriptive and prescriptive models of human behavior are heuristics. Heuristics are simple strategies that, by exploiting the structure of environments, can aid people to make smart decisions. Relying on heuristics does not mean trading off accuracy against effort: while reducing complexity, heuristics can yield better decisions than more information-greedy procedures in many decision environments. The prescriptive power of heuristics is documented in a cross-disciplinary literature, cutting across medicine, crime, business, sports, and other domains. We outline the fast-and-frugal heuristics research program, provide examples of past empirical work on heuristics outside the field of bibliometrics, explain why heuristics may be especially suitable for studying the usage of bibliometrics, and propose a corresponding conceptual framework.

Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

“…Indicators are measurable quantities which are "substitutes for something less readily measurable and is presumed to associate with it without directly measuring it" (Wilsdon et al, 2015, p. 5). Bornmann and Marewski (2019) coined the term bibliometrics-based heuristics for scientific performance assessments if the decisions are based on only two pieces of information from the available indicators space: citations and publications. Most of the indicators which have been developed in bibliometrics hitherto focus on four major areas:…”

Studying Bibliometrics-Based Heuristics (BBHs): A New Research Program on the use of Bibliometrics in Research Evaluation

“…Simple decision making strategies based on bibliometrics can be similarly accurate to complex strategies, such as peer review processes. Bornmann and Marewski (2019) recommend that assessment processes in science be empirically studied to reveal the situations in which this is probably the case and those in which it is not.…”

“…Do they prefer to rely on the h index for their decisions? Bornmann and Marewski (2019) suggest that these and similar questions can be empirically answered by studying the use of bibliometrics within the heuristics research program.…”

“…Since scientific quality is seen as multi-dimensional phenomenon, complex ways are regarded as a gold standard for research evaluation processes. In a recent paper, Bornmann and Marewski (2019) argue that in certain evaluation situations bibliometrics-based heuristics (BBHs) might be good alternatives to complex assessment processes. BBHs are reasonable decision strategies which reduce the considered information in assessments to indicators based on publication and citation data.…”

“…BBHs are reasonable decision strategies which reduce the considered information in assessments to indicators based on publication and citation data. In this short communication, main lines proposed by Bornmann and Marewski (2019) are summarized in a brief overview.…”