PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review

Abstract

We consider the problem of automated assignment of papers to reviewers in conference peer review, with a focus on fairness and statistical accuracy. Our fairness objective is to maximize the review quality of the most disadvantaged paper, in contrast to the commonly used objective of maximizing the total quality over all papers. We design an assignment algorithm based on an incremental max-flow procedure that we prove is near-optimally fair. Our statistical accuracy objective is to ensure correct recovery of the papers that should be accepted. We provide a sharp minimax analysis of the accuracy of the peer-review process for a popular objective-score model as well as for a novel subjective-score model that we propose in the paper. Our analysis proves that our proposed assignment algorithm also leads to a near-optimal statistical accuracy. Finally, we design a novel experiment that allows for an objective comparison of various assignment algorithms, and overcomes the inherent difficulty posed by the absence of a ground truth in experiments on peer-review. The results of this experiment corroborate the theoretical guarantees of our algorithm.

Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments

“…Finally, this work does not address the problem of reviewers colluding with each other to give dishonest favorable reviews after being assigned to each others' papers; we leave this issue for future work. Due to the minimum in the objective, this problem is NP-hard [50]; the paper [21] presents an algorithm to find an approximate solution.…”

“…An alternate objective to the sum-similarity objective has been studied in past work [20,21], aiming to improve the fairness of the assignment with respect to the papers. Rather than maximizing the sum-similarity across all papers, this objective maximizes the minimum total similarity assigned to any paper: thereby violating the constraints of T .…”

“…We treat the bidding process and computation of similarities as given, and focus primarily on adjusting the optimization problem to address the three aforementioned challenges. Some work has considered other optimization objectives such as fairness [20,21]. We also consider a similar fairness objective in a variant of our algorithm.…”

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

“…Assignment procedure Each paper submitted to the ICML conference was first automatically assigned to three reviewers from the main pool, two of whom were curated reviewers and one was a selfnominated reviewer. In that, we tried to satisfy reviewer bids and optimize for the notion of textual similarity (Charlin and Zemel, 2013) between submissions and assigned reviewers, subject to a requirement that each reviewer is assigned at most six papers (a small set of reviewers requested a lower quota) and under a fairness constraint that aims at balancing assignment quality across submissions (Stelmakh et al, 2018).…”

Fairness Perception from a Network-Centric Perspective

“…Peer evaluation of scientific work has significant effect on scientific advancement. However, similar to other systems designed by humans, it is potentially biased, favoring certain groups of individuals (e.g., famous authors or researchers from well-known institutions) [11], [12]. The unintended consequences of the bias in peer review go beyond the decision about a single paper, it can significantly effect the career path of the authors.…”