Dialogue Coherence Assessment Without Explicit Dialogue Act Labels

Abstract

Recent dialogue coherence models use the coherence features designed for monologue texts, e.g. nominal entities, to represent utterances and then explicitly augment them with dialogue-relevant features, e.g., dialogue act labels. It indicates two drawbacks, (a) semantics of utterances is limited to entity mentions, and (b) the performance of coherence models strongly relies on the quality of the input dialogue act labels. We address these issues by introducing a novel approach to dialogue coherence assessment. We use dialogue act prediction as an auxiliary task in a multi-task learning scenario to obtain informative utterance representations for coherence assessment. Our approach alleviates the need for explicit dialogue act labels during evaluation. The results of our experiments show that our model substantially (more than 20 accuracy points) outperforms its strong competitors on the Dai-lyDialogue corpus, and performs on par with them on the SwitchBoard corpus for ranking dialogues concerning their coherence. We release our source code 1 .Recent approaches to dialogue coherence modeling use the coherence features designed for monologue texts, e.g. entity transitions (Barzilay and Lapata, 2005), and augment them with dialoguerelevant features, e.g., DA labels (Cervone et al., 2018). These DA labels are provided by human annotators or DA prediction models. Such coherence models suffer from the following drawbacks: (a) they curb semantic representations of utterances to entities, which are sparse in dialogue because of short utterance lengths, and (b) their performance relies on the quality of their input DA labels.

DynaEval: Unifying Turn and Dialogue Level Evaluation

“…Very few have considered customizing their dialogue coherence models for evaluating the performance of dialogue systems. It is common to leverage supervised approaches (Higashinaka et al, 2014;Gandhe and Traum, 2016;Cervone et al, 2018;Yi et al, 2019), that is closely linked to modeling with entities and dialogue acts (Cervone and Riccardi, 2020;Zhou et al, 2019;Mesgar et al, 2020).…”

“…For fair comparison, we apply the same procedure described in Section 3.1 to derive the sentence embedding of an utterance in CoSim. S-DiCoh (Mesgar et al, 2020) is a recent state-of-the-art dialogue coherence model. It models a dialogue with a neural network framework consisting of two bidrectional LSTM layers with attention mechanism at both the token and utterance level.…”

Improving Factual Consistency Between a Response and Persona Facts

“…Topical coherence sub-reward (R 2 ) Topical coherence is a crucial property of high-quality dialogues (See et al, 2019;Mesgar et al, 2020). We capture the topical coherence of response r to the last utterance u T 1 in dialogue history by representing them using an average pooling layer over their token representations obtained by BERT.…”

Is Incoherence Surprising? Targeted Evaluation of Coherence Prediction from Language Models

“…There exists a large body of work in linguistics regarding different notions of coherence, such as the influence of coreference (Hobbs, 1979;Barzilay and Lapata, 2008, inter alia), Centering theory (Grosz et al, 1995), discourse structure (Mann and Thompson, 1987;Webber et al, 2003), and phenomena that connect utterances in dialogue, such as conversational maxims (Grice, 1975) or speaker interaction (Lascarides and Asher, 2009). Many of these are also mentioned by coherence evaluation studies, nonetheless they mostly revert to the use of some form of sentence-order variations (Chen et al, 2019;Moon et al, 2019;Mesgar et al, 2020). While some progress has been made towards incorporating more linguistically motivated test sets (Chen et al, 2019;Mohammadi et al, 2020;Pishdad et al, 2020), most evaluation studies focus on models trained specifically on coherence classification and prediction tasks.…”

“…Since sequentiality is central to the language modelling task, models successfully distinguish between both versions. This shuffling technique has been widely applied in the evaluation of coherence models (Barzilay and Lapata, 2008;Chen et al, 2019;Moon et al, 2019;Mesgar et al, 2020). We include it as baseline for our method, in order to contrast how more fine-grained notions of coherence compare to this broad approach.…”