Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise

Dhull, Komal; Steven, Jecmen,; Kothari, Pravesh K.; Shah, Nihar B.

doi:10.48550/arxiv.2201.10631

Cited by 2 publications

(3 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, there has been a flurry of works that address various other problems in peer review, such as bias [32][33][34], miscalibration [35][36][37][38], subjectivity [39,40], dishonest behavior [41][42][43][44][45][46], and others. See [47] for a survey.…”

Section: Related Workmentioning

confidence: 99%

Integrating Rankings into Quantized Scores in Peer Review

Liu¹,

Yi‐chong²,

Shah³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

In peer review, reviewers are usually asked to provide scores for the papers. The scores are then used by Area Chairs or Program Chairs in various ways in the decision-making process. The scores are usually elicited in a quantized form to accommodate the limited cognitive ability of humans to describe their opinions in numerical values. It has been found that the quantized scores suffer from a large number of ties, thereby leading to a significant loss of information. To mitigate this issue, conferences have started to ask reviewers to additionally provide a ranking of the papers they have reviewed. There are however two key challenges. First, there is no standard procedure for using this ranking information and Area Chairs may use it in different ways (including simply ignoring them), thereby leading to arbitrariness in the peer-review process. Second, there are no suitable interfaces for judicious use of this data nor methods to incorporate it in existing workflows, thereby leading to inefficiencies.We take a principled approach to integrate the ranking information into the scores. The output of our method is an updated score pertaining to each review that also incorporates the rankings. Our approach addresses the two aforementioned challenges by: (i) ensuring that rankings are incorporated into the updates scores in the same manner for all papers, thereby mitigating arbitrariness, and (ii) allowing to seamlessly use existing interfaces and workflows designed for scores. We empirically evaluate our method on synthetic datasets as well as on peer reviews from the ICLR 2017 conference, and find that it reduces the error by approximately 30% as compared to the best performing baseline on the ICLR 2017 data.

show abstract

Section: Related Workmentioning

confidence: 99%

Integrating Rankings into Quantized Scores in Peer Review

Liu¹,

Yi‐chong²,

Shah³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In order to avoid or mitigate this type of manipulation, Dhull et al (2022) proposed a maximum similarity strategyproof reviewer assignment, in which no reviewer can alter their evaluation to better the outcome of their own proposal. They used the partitioning method, which was introduced by Alon et al (2011), to ensure strategyproofness.…”

Section: Torpedo Reviewing Avoidancementioning

confidence: 99%

“…Xu et al (2020) also dealt with the assignment of reviewers to proposals that maximize the similarity score under the constraints of strategyproofness. Dhull et al (2022) assumed that each reviewer submits a proposal and that each submission is written by a reviewer. In contrast, Xu et al (2020) considered arbitrary authorships in which each proposal may have many authors, and each reviewer may have submitted multiple proposals.…”

Section: Torpedo Reviewing Avoidancementioning

confidence: 99%

Reviewer Assignment Problem: A Systematic Review of the Literature

Meltem¹,

Yanık

Amasyalı

2023

jair

View full text Add to dashboard Cite

Appropriate reviewer assignment significantly impacts the quality of proposal evaluation, as accurate and fair reviews are contingent on their assignment to relevant reviewers. The crucial task of assigning reviewers to submitted proposals is the starting point of the review process and is also known as the reviewer assignment problem (RAP). Due to the obvious restrictions of manual assignment, journal editors, conference organizers, and grant managers demand automatic reviewer assignment approaches. Many studies have proposed assignment solutions in response to the demand for automated procedures since 1992. The primary objective of this survey paper is to provide scholars and practitioners with a comprehensive overview of available research on the RAP. To achieve this goal, this article presents an in-depth systematic review of 103 publications in the field of reviewer assignment published in the past three decades and available in the Web of Science, Scopus, ScienceDirect, Google Scholar, and Semantic Scholar databases. This review paper classified and discussed the RAP approaches into two broad categories and numerous subcategories based on their underlying techniques. Furthermore, potential future research directions for each category are presented. This survey shows that the research on the RAP is becoming more significant and that more effort is required to develop new approaches and a framework.

show abstract

Strategyproofing Peer Assessment via Partitioning: The Price in Terms of Evaluators' Expertise

Cited by 2 publications

References 0 publications

Integrating Rankings into Quantized Scores in Peer Review

Integrating Rankings into Quantized Scores in Peer Review

Reviewer Assignment Problem: A Systematic Review of the Literature

Contact Info

Product

Resources

About