Ordinal and Attribute Aware Response Generation in a Multimodal Dialogue System

Abstract

Multimodal dialogue systems have opened new frontiers in the traditional goal-oriented dialogue systems. The state-of-the-art dialogue systems are primarily based on unimodal sources, predominantly the text, and hence cannot capture the information present in the other sources such as videos, audios, images etc. With the availability of large scale multimodal dialogue dataset (MMD) (Saha et al., 2018) on the fashion domain, the visual appearance of the products is essential for understanding the intention of the user. Without capturing the information from both the text and image, the system will be incapable of generating correct and desirable responses. In this paper, we propose a novel position and attribute aware attention mechanism to learn enhanced image representation conditioned on the user utterance. Our evaluation shows that the proposed model can generate appropriate responses while preserving the position and attribute information. Experimental results also prove that our proposed approach attains superior performance compared to the baseline models, and outperforms the state-of-the-art approaches on text similarity based evaluation metrics.

More to diverse: Generating diversified responses in a task oriented multimodal dialog system

“…The user utterances play a significant role in building the dialogue context for generating coherent and appropriate responses, in accordance to the user demands. We focus on generating the textual responses only in a similar manner as [8,12,13]. Here, the task of multi-modal dialog generation is defined as follows: we consider both the modalities, i.e.…”

“…Earlier works on the MMD dataset reported in [12,13,44] used the hierarchical encoder-decoder model to generate responses by capturing information from text, images and the knowledge base. Recently, [8] proposed attribute and position-aware attention for generating textual responses. The authors in [45] used an hierarchical attention mechanism for generating responses on the MMD dataset.…”

“…In this model, we concatenate pairwise text and image features U f and I f obtained after parallel co-attention mechanism as input to the MFB module for better interaction between the modalities as used in [8].…”

“…In Table 2, we provide the results of our proposed framework in comparison to the existing state-of-the-art methods. For response generation in multi-modal systems, we compare our current work with the existing systems [8,13]. For a fair comparison with the existing systems, the data used for experimentation should be similar in terms of its structure, genre as well as annotation in order to draw correct conclusions on the results obtained.…”

“…Lately, several works on multimodal dialogue systems [7][8][9] have encouraged research in this direction by combining information from the different modalities, such as texts, audios, videos and images. Multimodal conversational systems provide completeness to the existing dialogue systems by providing necessary information that lacks in unimodal systems.…”