UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems

Abstract

Knowledge bases (KBs) are usually essential for building practical dialogue systems. Recently we have seen rapidly growing interest in integrating knowledge bases into dialogue systems. However, existing approaches mostly deal with knowledge bases of a single modality, typically textual information. As today's knowledge bases become abundant with multimodal information such as images, audios and videos, the limitation of existing approaches greatly hinders the development of dialogue systems. In this paper, we focus on task-oriented dialogue systems and address this limitation by proposing a novel model that integrates external multimodal KB reasoning with pre-trained language models. We further enhance the model via a novel multi-granularity fusion mechanism to capture multi-grained semantics in the dialogue history. To validate the effectiveness of the proposed model, we collect a new large-scale (14K) dialogue dataset MMDialKB, built upon multimodal KB. Both automatic and human evaluation results on MMDialKB demonstrate the superiority of our proposed framework over strong baselines.

Cross-modal text and visual generation: A systematic review. Part 1: Image to text

“…There has been some effort to incorporate knowledge broader than what can be learned from the training dataset itself. One particular area of application is visual dialogue, where external knowledge bases have been proposed [160],…”

Dialogue Logic Aware and Key Utterance Decoupling Model for Multi-Party Dialogue Reading Comprehension

“…W. Wei et al [28] think dialogue reading comprehension is also used for intelligent human-computer interaction systems. Compared to the more mature two-party dialogue MRC [29], [30], one would expect applications such as dialogue systems to be able to handle more complex multi-party dialogue MRC. Due to the excellent performance of PrLMs in text-level NLP tasks (Section II-A), PrLMs have been widely used in the processing of multi-party dialogue MRC in earlier studies [31], [32].…”

Comparison of Features in Content Coverage and Presentation of Complex Numbers in Textbooks of China, Japan and Singapore

“…Since knowledge plays a vital role in the response generation of task-oriented dialog systems, we first conduct the knowledge acquisition for the given multimodal context. Considering the semantic knowledge is pivotal to capturing the user's intentions [33,36,37], we focus on selecting two kinds of semantic knowledge: attribute knowledge and relation knowledge. Thereinto, the attribute knowledge, which is widely used, refers to the attribute-value pairs of entities mentioned directly in the context.…”