Combating Collusion Rings is Hard but Possible

Abstract

A recent report of Littmann [Commun. ACM '21] outlines the existence and the fatal impact of collusion rings in academic peer reviewing. We introduce and analyze the problem CYCLE-FREE REVIEWING that aims at finding a review assignment without the following kind of collusion ring: A sequence of reviewers each reviewing a paper authored by the next reviewer in the sequence (with the last reviewer reviewing a paper of the first), thus creating a review cycle where each reviewer gives favorable reviews. As a result, all papers in that cycle have a high chance of acceptance independent of their respective scientific merit.We observe that review assignments computed using a standard Linear Programming approach typically admit many short review cycles. On the negative side, we show that CYCLE-FREE RE-VIEWING is NP-hard in various restricted cases (i.e., when every author is qualified to review all papers and one wants to prevent that authors review each other's or their own papers or when every author has only one paper and is only qualified to review few papers). On the positive side, among others, we show that, in some realistic settings, an assignment without any review cycles of small length always exists. This result also gives rise to an efficient heuristic for computing (weighted) cycle-free review assignments, which we show to be of excellent quality in practice.2 was a soft constraint [Leyton-Brown and Mausam, 2021]. Yet, there is a lack of systematic studies concerning the computation of such assignments. Motivated by this, we propose and analyze CYCLE-FREE REVIEWING, the problem of computing an assignment of papers to agents that is free of review cycles of length at most z, both from a theoretical and practical perspective.

Reviewer Assignment Problem: A Systematic Review of the Literature

“…Due to the dual difficulties of a large volume of proposals and reviewers and limited time, the problem cannot be addressed efficiently by exact algorithms in an acceptable amount of time and memory. Therefore, numerous researchers have investigated the use of approximation or heuristic algorithms for the problem of allocating reviewers, such as the greedy algorithm (Cook et al, 2005;Karimzadehgan & Zhai, 2009Long et al, 2013;Guo et al, 2018;Boehmer et al, 2021), two-phase stochastic-biased greedy algorithm (Wang et al, 2013), stage deepening greedy algorithm (Kou et al, 2015a(Kou et al, , 2015bMirzaei et al, 2019), weighted-matrix factorization based greedy algorithm (Pradhan et al, 2020), greedy reviewer round-robin algorithm (Payan & Zick, 2022), ant colony algorithm (Li et al, 2008), random split (Jecmen et al, 2021), row/column generation (Leyton-Brown et al, 2022), genetic algorithm (Li et al, 2007;Xu et al, 2010;Chen et al, 2012;, particle swarm optimization algorithm (Yang et al, 2020), hill-climbing algorithm (Hettich & Pazzani, 2006), OBH algorithm (Yeşilçimen & Yıldırım, 2019), PeerReview4All (Stelmakh et al, 2019) and FairFlow (Kobren et al, 2019). Some researchers have proposed hybrid algorithms, which are a combination of more than one algorithm and are used to eliminate the drawbacks of single algorithms (Yang et al, 2020).…”

“…Such reviewer-proposal loops violate anonymity and threaten the fairness and legitimacy of the peer review process and should therefore be avoided in automatic reviewer assignments. * (1) Hartvigsen et al, 1999(2) Guerv´os & Valdivieso, 2004(3) Cook et al, 2005 Janak et al, 2006;(5) Sun et al, 2007 (6) Li et al, 2007 (7) Li et al, 2008 (8) Karimzadehgan & Zhai, 2009 (9) Conry et al, 2009 (10) Xu et al, 2010 (11) Garg et al, 2010 (12) Liu and Hong, 2010 (13) Tang et al, 2010(14) Tang et al, 2012 Chen et al, 2012 (16) Karimzadehgan & Zhai, 2012 (17) Xue et al, 2012 (18) Charlin et al, 2012 (19) Long et al, 2013 (20) Tayal et al, 2014 (21) Wang et al, 2013 (22) Li et al, 2013 (23) Charlin & Zemel, 2013(24) Cechlarova et al, 2014 Daş & Gökçen, 2014 (26) Silva et al, 2014 (27) Kale et al, 2015 (28) Kou et al, 2015aKou et al, , 2015b Liu et al, 2016 (30) Yue et al, 2017(31) Ogunleyu et al, 2017 Yeşilçimen & Yıldırım, 2019 (33) Mirzaei et al, 2019 (34) Yang et al, 2020 (35) Jecmen et al, 2020 (36) Jin et al, 2018b (37) Pradhan et al, 2020 (38) Kat, 2021 (39) Boehmer et al, 2021(40) Kobren et al, 2019 Dhull et al, 2022 (42) Guo et al, 2018 (43) Xu et al, 2019 (44)…”

“…Closer to Guo et al (2018) setting, Boehmer et al (2021) also consider the computation of cycle-free review assignments. They proposed a weighted cycle-free reviewer assignment problem and developed greedy algorithms to solve this problem.…”

A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences

“…Second, automated estimates are corrected by reviewers who can read abstracts of submissions and report bids-preferences in reviewing the submissions. A number of works focus on the bidding stage and (i) explore the optimal strategy to assist reviewers in navigating the pool of thousands of submissions (Fiez et al, 2020;Meir et al, 2020) or (ii) protect the system from strategic bids made by colluding reviewers willing to get assigned to each other's paper (Jecmen et al, 2020;Wu et al, 2021;Boehmer et al, 2021;Jecmen et al, 2022).…”