Creating a consensus ranking of proposals from reviewers’ partial ordinal rankings

Abstract

Peer review of research proposals and articles is an essential element in R&D processes world-wide. In most cases, each reviewer evaluates a small subset of the candidate proposals.The review board is then faced with the challenge of creating an overall "consensus" ranking on the basis of many partial rankings. In this paper we propose a branch and bound model to support the construction of an aggregate ranking from the partial rankings provided by the reviewers. In a recent paper we proposed ways to allocate proposals to reviewers so as to achieve the maximum possible overlap among the subsets of proposals allocated to different reviewers. Here, we develop a special branch and bound algorithm that utilizes the overlap generated through our earlier methods to enable discrimination in ranking the competing proposals. The effectiveness and efficiency of the algorithm is demonstrated with small numerical examples and tested through an extensive simulation experiment.

Discovering a taste for the unusual: exceptional models for preference mining

“…A consensus ranking can be seen as an overall ranking that has the highest agreement with a given set of rankings (Cook et al 2007). Different methods to derive the consensus ranking can be found in the literature (Sculley 2007;Svendová and Schimek 2017).…”

Ordinal evaluation and assignment problems

“…Our interest in these problems is motivated by the fa ct that experts are generally selected to meet the evaluation needs of a set of documents rather than randomly assembled together. Thus, unlike in the assignment problems considered in [5,7], minimizing the number of experts is the main objective in the assignments of documents to experts in our work. We consider the assignments of documents to experts without specialties.…”

“…The ranking and selection of documents typically relies on cardinal (quantitative) or ordinal (preference) -based comparisons as described in the context of research documents in [6,13]. Cook et al demonstrated that cardinal comparisons such as using average scores of documents could be unreliable especially when experts' scores are not normalized [5][6][7]. They suggested that quantifying the intrinsic values of documents may be difficult, and therefore it is more practical to rely on ordinal rankings.…”

Ordinal covering using block designs

“…Ordinal or relative evaluations of documents have been suggested as a more reliable alternative in identifYing high quality documents over cardinal evaluations such as using average scores assigned to the documents by a set of experts [1][2][3][4][5]. Ordinal and cardinal strengths of preferences have also been advocated in [6,8] as natural extensions of ordinal comparison models.…”

“…Let A, t, and r be positive integers, where 2 S t < u. A block design (G,B) is called a (u,v,r,t,A)-balanced and incomplete block design (BIBD) if (1) (2) each element in G appears in exactly r blocks, (3) all blocks in B have t elements, and (4) each pair of elements in G appears in exactly A blocks. We have ur = vt since each of the u elements in G appears r times in all the blocks and the union of the blocks as a multiset contains exactly vt elements.…”