Can We Automate Scientific Reviewing?

Abstract

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question “can we automate scientific reviewing? ”, discussing the possibility of using natural language processing (NLP) models to generate peer reviews for scientific papers. Because it is non-trivial to define what a “good” review is in the first place, we first discuss possible evaluation metrics that could be used to judge success in this task. We then focus on the machine learning domain and collect a dataset of papers in the domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers as input and generate reviews as output. Comprehensive experimental results on the test set show that while system-generated reviews are comprehensive, touching upon more aspects of the paper than human-written reviews, the generated texts are less constructive and less factual than human-written reviews for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. Given these results, we pose eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research in this direction. We make relevant resource publicly available for use by future research: https://github. com/neulab/ReviewAdvisor. In addition, while our conclusion is that the technology is not yet ready for use in high-stakes review settings we provide a system demo, ReviewAdvisor (http://review.nlpedia.ai/), showing the current capabilities and failings of state-of-the-art NLP models at this task (see demo screenshot in A.2). A review of this paper written by the system proposed in this paper can be found in A.1.

NeurReview: A Neural Architecture Based Conformity Prediction of Peer Reviews

“…Ines Arous et al proposed a Bayesian framework that integrates a machine learning model with peer grading to assess the conformity of scholarly reviews [5]. To evaluate the generated review, Yuan et al proposed a variety of diagnostic criteria for review quality, including review aspect coverage and informativeness [4]. Falkenberg et al analyzed the characteristics of review text that distinguish high-quality reviews from lower-quality reviews for editors, but only a small sample of reviews was investigated [21].…”

“…The reviewers largely follow a well-defined structure while writing reviews identifying the pros and cons of the paper. A good review should be well-organized, typically starting with a brief summary of the paper's contributions, then following with opinions gauging the quality of a paper from different aspects, together with evidence [4] . Moreover, a peer review should be sentiment rich and the reviewers would tend to express varying sentiments across various aspects.…”

“…The annotation process can be formulated as a sequence labeling problem where the input is a sentence consisting of n words S = {w 1 , w 2 , ..., w n }. Yuan et al have conducted manual annotation for sample reviews of ICLR, ACL [4]. Taking their annotations as labels, we can annotate the rest of the data using a pre-trained model.…”

“…The wellknown NIPS experiment [3] observed observation that 43% of papers accepted to the conference would be accepted again if the conference review process were repeated. The bias of the reviewers [4] leading to inconsistencies between their review reports further aggravates the problem. Apart from that, peer review has also been challenged by the rapid increase of paper submissions.…”

“…As far as we consider, evaluating peer review quality is rather complicated. It has received attention from multiple perspectives such as justification [5], bias [6], comprehen-siveness [4]. In our present work, we would rather focus on the conformity of reviews with meta-reviews, and the corresponding text features.…”