An algorithm for automatic assignment of reviewers to papers

Kalmukov, Yordan

doi:10.1007/s11192-020-03519-0

Cited by 17 publications

(13 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is no wonder that every day it is more difficult to recruit those (Fox, Albert and Vines 2017 ), because later they are blinkered by bibliometrics (Stephan Veugelers and Wang 2017 ). Their availability and sophisticated assignment (i. e. Kalmukov 2020 ) are more reminiscent of a skilful fishing exercise than the fundamental ready participation that illuminated Oldenburg ( 1667 ). Most of these reviewers are bright young people but precarious researchers working altruistically for publishers and other “big guys”; the latter, in contrast to the former, are a band of mediocre people who swarm and vegetate in science but never lose their chairs.…”

Section: The Reviewer As a Useful Slavementioning

confidence: 99%

Letter to the Editor: publish, publish … cursed!

Cano

2021

Scientometrics

View full text Add to dashboard Cite

This paper comments on the phenomenon of “publish or perish” associated with the current information explosion and its awful consequence: the curse that hangs over academia which dooms it to publish incessantly irrelevant and pointless documents. The overabundance of publications is not justified and is not even necessary in many contexts for personal promotion, and even less for the advancement of science. Therefore, the current role of scientific journals is highly questionable that its aim could be misleading. Huge numbers of articles are published, but they are not read because the aim is principally “publish for publish,” or publication for its own sake. The standard corrective tool for improving scientific communication—peer review—cannot function adequately, and biases and perversions are introduced which undermine society’s confidence in the scientific enterprise. A dark landscape unfurls itself across the world of scientific information, forcing us to question and improve its current state. Methodologically this paper goes halfway between the essay and the review trying to provoke engaged and useful controversy.

show abstract

Section: The Reviewer As a Useful Slavementioning

confidence: 99%

Letter to the Editor: publish, publish … cursed!

Cano

2021

Scientometrics

View full text Add to dashboard Cite

show abstract

“…It builds reviewers' profiles based on their previous publications obtained from Google Scholar or uploaded by the reviewers themselves. By using Latent Dirichlet Allocation (LDA) [1], TPMS extracts reviewers' research topics from their publications. [8] used latent semantic indexing to automate the assignment of papers to reviewers in Hypertext'91 conference.…”

Section: Related Workmentioning

confidence: 99%

“…Its accuracy depends on both the assignment algorithm and the method of describing papers and reviewers' competencies. Assignment algorithms are studied in details in [1] and will not be discussed here. Instead, this article focuses on the methods of describing papers and identifying reviewers' competencies.…”

Section: Introductionmentioning

confidence: 99%

Comparison of Latent Semantic Analysis and Vector Space Model for Automatic Identification of Competent Reviewers to Evaluate Papers

Kalmukov¹

2022

IJACSA

Self Cite

View full text Add to dashboard Cite

The assignment of reviewers to papers is one of the most important and challenging tasks in organizing scientific events. A major part of it is the correct identification of proper reviewers. This article presents a series of experiments aiming to test whether the latent semantic analysis (LSA) could be reliably used to identify competent reviewers to evaluate submitted papers. It also compares the performance of the LSA, the vector space model (VSM) and the method of explicit document description by a taxonomy of keywords, in computing accurate similarity factors between papers and reviewers. All the three methods share the same input datasets, taken from real-life conferences and the produced paper-reviewer similarities are evaluated with the same evaluation methods, allowing a fair and objective comparison between them. Experimental results show that in most cases LSA outperforms VSM and could even slightly outperform the explicit document description by a taxonomy of keywords, if the term-document matrix is composed of TF-IDF values, rather than the raw number of term occurrences.

show abstract

“…For example, Jecmen et al [11] consider assignments as probabilistic and define the optimization objective as the expected sum of similarity scores across all submission-CR pairs, subject to the constraint on the probability of each assignment, as well as the typical load constraints. On the other hand, Pradhan et al [36] define and solve RAP as an equilibrium multi-job problem; Liu et al [23] propose a hybrid method that combines knowledge rules and an optimisation method, whereas Kalmukov [15] treats RAP as the task of maximising the sum of edge weights (which reflect the level of matching) of an undirected bipartite graph of CRs and submissions.…”

Section: Reviewer (Panel) Assignment Through Optimization Of An Objec...mentioning

confidence: 99%

“…Evaluation measures. A summary of the evaluation measures used across the selected studies is given in Table EvalMeasures in the supplementary file 15 . Only a small subset of studies (7 out of 26) used traditional Information retrieval (IR) evaluation measures (e.g., precision@k, recall@k, F1, mean averaged precision) or measures typical for evaluation of recommender systems (e.g., normalized discounted cumulative gain -nDCG).…”

mentioning

confidence: 99%

Reviewer Paper Assignment Problem – A Brief Review

Patil¹,

Mahalle²

2022

Handbook on ICT in Developing Countries

View full text Add to dashboard Cite

The quality of peer review, and consequently the published research, depends to a large extent on the ability to recruit adequate reviewers for submitted papers. However, finding such reviewers is an increasingly difficult task. To alleviate this challenge, numerous solutions have been suggested for automated association of papers with "well matching" reviewers -the task often referred to as reviewer assignment problem (RAP). Yet, to our knowledge, a recent systematic synthesis of the RAP-related literature is missing. To fill this gap and support further RAP-related research, we present a scoping review on computational approaches to RAP. Following the latest methodological guidance for scoping reviews, we have collected and synthesised the literature on RAP published over the last five years (Jan 2016 -Sept 2021). Specifically, the review is focused on the following aspects of RAP research: i) the overall approach to RAP; ii) the criteria used for reviewer selection; iii) the modelling of candidate reviewers and submissions; iv) the computational methods for matching reviewers and submissions; and v) the evaluation methods used for assessing the performance of the proposed solutions. The paper summarises and discusses the findings for each of the aforementioned aspects of RAP research and suggests future research directions.

show abstract

An algorithm for automatic assignment of reviewers to papers

Cited by 17 publications

References 20 publications

Letter to the Editor: publish, publish … cursed!

Letter to the Editor: publish, publish … cursed!

Comparison of Latent Semantic Analysis and Vector Space Model for Automatic Identification of Competent Reviewers to Evaluate Papers

Reviewer Paper Assignment Problem – A Brief Review

Contact Info

Product

Resources

About