How coherent are neural models of coherence?

Abstract

Despite the recent advances in coherence modelling, most such models including state-of-the-art neural ones, are evaluated on either contrived proxy tasks such as the standard order discrimination benchmark, or tasks that require special expert annotation. Moreover, most evaluations are conducted on small newswire corpora. To address these shortcomings, in this paper we propose four generic evaluation tasks that draw on different aspects of coherence at both the lexical and document levels, and can be applied to any corpora. In designing these tasks, we aim at capturing coherence-specific properties, such as the correct use of discourse connectives, lexical cohesion, as well as the overall temporal and causal consistency among events and participants in a story. Importantly, our proposed tasks either rely on automatically-generated data, or data annotated for other purposes, hence alleviating the need for annotation specifically targeted to the task of coherence modelling. We perform experiments with several existing state-of-the-art neural models of coherence on these tasks, across large corpora from different domains, including newswire, dialogue, as well as narrative and instructional text. Our findings point to a strong need for revisiting the common practices in the development and evaluation of coherence models.

Rethinking Self-Supervision Objectives for Generalizable Coherence Modeling

“…They call for more comprehensive evaluations of coherence models. Pishdad et al (2020) also reached a similar conclusion. They retrained several neural coherence models for tasks analogous to coherence modeling such as detecting connective substitution and topic switching.…”

“…With the advancements of neural methods in recent years, claims of fluency in summarization (Liu et al, 2017;Celikyilmaz et al, 2018), language modeling (Radford et al, 2019;Brown et al, 2020), response generation (Zhang et al, 2020;Hosseini-Asl et al, 2020) and human parity in machine translation (Hassan et al, 2018) have led to calls for finer-grained discourse-level evaluations (Läubli et al, 2018;Sharma et al, 2019;Popel et al, 2020), since traditional metrics such as BLEU and ROUGE are unable to measure text quality and readability (Paulus et al, 2018;Reiter, 2018). Coherence models that can evaluate machine-generated text have become the need of the hour.…”

Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models

“…Many of these are also mentioned by coherence evaluation studies, nonetheless they mostly revert to the use of some form of sentence-order variations (Chen et al, 2019;Moon et al, 2019;Mesgar et al, 2020). While some progress has been made towards incorporating more linguistically motivated test sets (Chen et al, 2019;Mohammadi et al, 2020;Pishdad et al, 2020), most evaluation studies focus on models trained specifically on coherence classification and prediction tasks.…”

“…It does not pinpoint the qualities that make the shuffled text incoherent, it does not tell us which linguistic devices are at fault, emphasising the need to move beyond this technique. This paper aims to add to the growing body of research stressing the need for more qualitative evaluations of text coherence (See et al, 2019;Mohammadi et al, 2020;Pishdad et al, 2020). We design different test suites created semiautomatically from existing corpora.…”

“…A common approach to coherence evaluation consists in shuffling the sentence order of a text, thereby creating incoherent text samples that need to be discriminated from the original (Barzilay and Lapata, 2008). While this approach to creating incoherent test data is intuitive enough, recent studies suggest that it paints only a partial picture of what constitutes coherence (Lai and Tetreault, 2018;Mohammadi et al, 2020;Pishdad et al, 2020). It does not pinpoint the qualities that make the shuffled text incoherent, it does not tell us which linguistic devices are at fault, emphasising the need to move beyond this technique.…”