Optimal Allocation of Proposals to Reviewers to Facilitate Effective Ranking

Abstract

Peer review of research proposals and articles is an essential element in research and development processes worldwide. Here we consider a problem that, to the best of our knowledge, has not been addressed until now: how to assign subsets of proposals to reviewers in scenarios where the reviewers supply their evaluations through ordinal ranking. The solution approach we propose for this assignment problem maximizes the number of proposal pairs that will be evaluated by one or more reviewers. This new approach should facilitate meaningful aggregation of partial rankings of subsets of proposals by multiple reviewers into a consensus ranking. We offer two ways to implement the approach: an integer-programming set-covering model and a heuristic procedure. The effectiveness and efficiency of the two models are tested through an extensive simulation experiment.

Assigning Papers to Referees

“…Previous work considered mostly optimizing global properties of the assignment, and the proposed algorithms are based on min-cost matching/flow [11,13,23], integer programming [15], or heuristics without provable guarantees [9,21]. Fairness of the assignment was not considered as an objective.…”

A code reviewer assignment model incorporating the competence differences and participant preferences

“…Integer linear programming is a very successful and popular method being used to solve the RAP [2,14,6]. Based on these work, we recommend a new approach to study the code reviewer assignment problem with integer linear programming.…”

“…Reviewer assignment problem (RAP) is a common task to people such as software developers, conference organizers, journal editors, grant administrators and educators [2,24] because qualified reviewers, especially highly competent ones, are usually scarce intellectual resources and play important roles in their fields. Sun, Ma, Fan, and Wang proposed that a good match between a reviewer's knowledge and a project would make useful and professional judgments on the research project to be funded [14].…”

Tweeting About Mental Health: Big Data Text Analysis of Twitter for Public Policy

“…Additionally, the fact that different papers rank different conditions means that most voting systems (such as a single transferable vote) fail to produce consistent results. I created an approach to identify the most common ranking inspired by the work of Cook et al (2005). The approach seeks to identify an overall ranking that would disagree with the fewest of the ten articles.…”