Uncertainty-Aware Machine Support for Paper Reviewing on the Interspeech 2019 Submission Corpus

Abstract

The evaluation of scientific submissions through peer review is both the most fundamental component of the publication process, as well as the most frequently criticised and questioned. Academic journals and conferences request reviews from multiple reviewers per submission, which an editor, or area chair aggregates into the final acceptance decision. Reviewers are often in disagreement due to varying levels of domain expertise, confidence, levels of motivation, as well as due to the heavy workload and the different interpretations by the reviewers of the score scale. Herein, we explore the possibility of a computational decision support tool for the editor, based on Natural Language Processing, that offers an additional aggregated recommendation. We provide a comparative study of state-of-the-art text modelling methods on the newly crafted, largest review dataset of its kind based on Interspeech 2019, and we are the first to explore uncertainty-aware methods (soft labels, quantile regression) to address the subjectivity inherent in this problem.

Uncertainty Aware Review Hallucination for Science Article Classification

“…In this short paper, we offer solutions to three particularities of this task that the above approaches do not address: a) Often, the recommendations given by the area chair and the reviewers are in disagreement. Whereas previous studies have used either the former (Kang et al, 2018;Wang and Wan, 2018;Ghosal et al, 2019) or a soft label average of the latter (Stappen et al, 2020) for supervision, we show that both signals comprise complementary information. b) Whereas soft labels de-emphasise subjective articles with disagreeing reviews during training (Stappen et al, 2020), we manage to outperform the latter study by explicitly modelling aleatory uncertainty as an auxiliary prediction task.…”

“…Whereas previous studies have used either the former (Kang et al, 2018;Wang and Wan, 2018;Ghosal et al, 2019) or a soft label average of the latter (Stappen et al, 2020) for supervision, we show that both signals comprise complementary information. b) Whereas soft labels de-emphasise subjective articles with disagreeing reviews during training (Stappen et al, 2020), we manage to outperform the latter study by explicitly modelling aleatory uncertainty as an auxiliary prediction task. c) A model that aims to support the editorial decision process should only assume the availability of human review text during training, and be able to make recommendations in their absence.…”

“…Small available dataset size is a limitation known to the community working on this domain (Kang et al, 2018;Ghosal et al, 2019) -we use the largest database of its kind (Stappen et al, 2020), i. e., the 2 179 preprocessed academic submissions, 5 842 reviews, with corresponding acceptance decisions and reviewer scores from the submission system of Interspeech 2019, shared with us by the technical chairs of the conference. After data cleaning and removal of corrupt entries, the accepted and rejected classes are well-balanced: 50.2 % acceptances, and 49.8 % rejections.…”

“…Natural Language Understanding (NLU) could also offer decision support to the area chair, as argued in (Ghosal et al, 2019;Stappen et al, 2020). Such systems jointly model the entire or part of the article and one (Kang et al, 2018;Wang and Wan, 2018;Ghosal et al, 2019), or a variable number…”

“…* * KF and GR contributed equally to this work. of potentially contradicting reviews (Stappen et al, 2020). We adopt the latter, review-aggregating approach, that resembles the editorial process more.…”