Uncertainty Aware Review Hallucination for Science Article Classification

Abstract

The high subjectivity and costs inherent in peer reviewing have recently motivated the preliminary design of machine learning-based acceptance decision methods. However, such approaches are limited in that they: a) do not explore the usage of both the reviewer and area chair recommendations, b) do not explicitly model subjectivity on a per submission basis, and c) are not applicable in realistic settings, by assuming that review texts are available at test time, when these are exactly the inputs that should be considered to be missing in this application. We propose to utilise methods that model the aleatory uncertainty of the submissions, while also exploring different loss importance interpolations between area chair and reviewers' recommendations. We also propose a modality hallucination approach to impute review representations at test time, providing the first realistic evaluation framework for this challenging task. * * KF and GR contributed equally to this work. of potentially contradicting reviews (Stappen et al., 2020). We adopt the latter, review-aggregating approach, that resembles the editorial process more.

Disentangling Uncertainty in Machine Translation Evaluation

“…Ye et al (2021) models uncertainty in performance prediction of NLP systems. Mielke et al (2019) applies heteroscedastic models to assess language difficulty, whereas Friedl et al (2021) estimates aleatoric uncertainty in scientific peer reviewing. While our paper focus on a regression task, some of our techniques might apply more broadly to these problems.…”