FaithDial: A Faithful Benchmark for Information-Seeking Dialogue

Abstract

The goal of information-seeking dialogue is to respond to seeker queries with natural language utterances that are grounded on knowledge sources. However, dialogue systems often produce unsupported utterances, a phenomenon known as hallucination. Dziri et al. ( 2022)'s investigation of hallucinations has revealed that existing knowledgegrounded benchmarks are contaminated with hallucinated responses at an alarming level (>60% of the responses) and models trained on this data amplify hallucinations even further (>80% of the responses). To mitigate this behavior, we adopt a data-centric solution and create FAITHDIAL, a new benchmark for hallucination-free dialogues, by editing hallucinated responses in the Wizard of Wikipedia (WOW) benchmark. We observe that FAITHDIAL is more faithful than WoW while also maintaining engaging conversations. We show that FAITHDIAL can serve as training signal for: i) a hallucination critic, which discriminates whether an utterance is faithful or not, and boosts the performance by 21.1 F1 score on the BEGIN benchmark compared to existing datasets for dialogue coherence; ii) high-quality dialogue generation. We benchmark a series of state-of-the-art models and propose an auxiliary contrastive objective that achieves the highest level of faithfulness and abstractiveness based on several automated metrics. Further, we find that the benefits of FAITHDIAL generalize to zero-shot transfer on other datasets, such as CMU-DOG and TOPICALCHAT. Finally, human evaluation reveals that responses generated by models trained on FAITHDIAL are perceived as more interpretable, cooperative, and engaging.

Proceedings of the 4th New Frontiers in Summarization Workshop

“…Previous work reported the percentage of novel n-grams or the notion of Coverage (Grusky et al, 2018) as a proxy for abstractiveness. These metrics have been adopted in other areas such as dialog (Dziri et al, 2022) to inspect the qualities and characteristics of datasets. Despite being convenient, these measures do not enable fine-grained analyses of multi-sentence aggregation.…”

Target-Guided Dialogue Response Generation Using Commonsense and Data Augmentation

“…Advancements in adding a moral dimension to KGs, and extending them with intuition of morality (such as crime is bad), can enable generation of morally correct knowledge paths. Furthermore, imbuing conversational systems with empathy (Ma et al, 2020), moral discretion (Ziems et al, 2022) and factual correctness (Gupta et al, 2021b;Dziri et al, 2022) will improve users' experience and trust in the system.…”

Graph vs. Sequence: An Empirical Study on Knowledge Forms for Knowledge-Grounded Dialogue

“…experimented with an Encoder-Decoder model on multiple datasets but ignored the Dual-Encoders are more compatible with the task. Dziri et al (2022a) experimented on sentencelevel knowledge datasets, neglecting the effect of knowledge size and granularity. Nevertheless, comparisons from multiple perspectives are needed to support the superiority and contribution of the model theoretically.…”