Metrics and methods in the evaluation of prestige bias in peer review: A case study in computer systems conferences

Abstract

The integrity of peer review is essential for modern science. Numerous studies have therefore focused on identifying, quantifying, and mitigating biases in peer review. One of these better-known biases is prestige bias, where the recognition of a famous author or affiliation leads reviewers to subconsciously treat their submissions preferentially. A common mitigation approach for prestige bias is double-blind reviewing, where the identify of authors is hidden from reviewers. However, studies on the effectivness of this mitigation are mixed and are rarely directly comparable to each other, leading to difficulty in generalization of their results. In this paper, we explore the design space for such studies in an attempt to reach common ground. Using an observational approach with a large dataset of peer-reviewed papers in computer systems, we systematically evaluate the effects of different prestige metrics, aggregation methods, control variables, and outlier treatments. We show that depending on these choices, the data can lead to contradictory conclusions with high statistical significance. For example, authors with higher h-index often preferred to publish in competitive conferences which are also typically double-blind, whereas authors with higher paper counts often preferred the single-blind conferences. The main practical implication of our analyses is that a narrow evaluation may lead to unreliable results. A thorough evaluation of prestige bias requires a careful inventory of assumptions, metrics, and methodology, often requiring a more detailed sensitivity analysis than is normally undertaken. Importantly, two of the most commonly used metrics for prestige evaluation, past publication count and h-index, are not independent from the choice of publishing venue, which must be accounted for when comparing authors prestige across conferences.

How many submissions are needed to discover friendly suggested reviewers?

“…Although we can find publicly available data on peer-review results (e.g. [25,26])allowing to quantify potential biases based on factors such as prestige [27], gender [28][29][30], ethnicity [31,32], and place of origin [33,34]-this data is anonymized to protect the privacy of authors and reviewers. On account of this, it is not possible to track the submission history of a single author across multiple publication venues to determine whether they are biasing editorial decisons through their list of suggested reviewers.…”

The Peer Review Process: Past, Present, and Future

“…Another important aspect of bias in peer review is author affiliation. Prestige bias relates to an author’s reputation, either with respect to individual notoriety in the field, or the author’s institution [ 42 ]. There is evidence that authors from higher income countries or more prestigious institutions receive more favourable review reports, relative to those affiliated with poorer countries or less renowned research institutes [ 43 ].…”