Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue

Abstract

Video-grounded Dialogue (VGD) aims to decode an answer sentence to a question regarding a given video and dialogue context. Despite the recent success of multi-modal reasoning to generate answer sentences, existing dialogue systems still suffer from a text hallucination problem, which denotes indiscriminate text-copying from input texts without an understanding of the question. This is due to learning spurious correlations from the fact that answer sentences in the dataset usually include the words of input texts, thus the VGD system excessively relies on copying words from input texts by hoping those words to overlap with ground-truth texts. Hence, we design Text Hallucination Mitigating (THAM) framework, which incorporates Text Hallucination Regularization (THR) loss derived from the proposed information-theoretic text hallucination measurement approach. Applying THAM with current dialogue systems validates the effectiveness on VGD benchmarks (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows enhanced interpretability.

HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

“…HEAR shows state-of-the-art performances on all the metrics compared to previous works (i.e., please refer to Related Work for their detailed captions.). Our baseline DLM is T5 Transformer (Raffel et al, 2020), which is the same baseline (i.e., T5RLM) of THAM (Yoon et al, 2022c), but here, our proposed SAL shows more gains, and further improvements are also shown by applying RLE. As our proposed HEAR is performed in a model-agnostic manner, we also validate other VGD models with HEAR in Table 2.…”

“…For the sensible decision in SAL, we introduce two technical contributions: (1) Keyword-based Audio Sensing and (2) Semantic Neural Estimator. HEAR is applied on current runner models (Hori et al, 2019a;Yoon et al, 2022c;Li et al, 2021b) in a model-agnostic manner, where the effectiveness is validated on VGD dataset (i.e., AVSD@DSTC7, AVSD@DSTC8) with steady performance gains on natural language generation metrics.…”

“…Graph representations (Kim et al, 2021;Pham et al, 2022; were also popular solutions for holding semantic commonalities between the dialogue and video. There have been novel structures (Hori et al, 2019b; to enhance the multi-modal representation and frameworks (Le and Chen, 2020;Yoon et al, 2022c) to improve the quality of responses in terms of bias or word selection. As such, many advances have been made in the multi-modal understanding of VGD systems, but mainly between video and language.…”

“…This symptom of ignoring input audio in responding to the question will be referred to as the 'deaf response'. Figure 1 represents examples of deaf responses of existing VGD systems (Yoon et al, 2022c;Li et al, 2021a). To the question "Does the video have sound?"…”