More to diverse: Generating diversified responses in a task oriented multimodal dialog system

Abstract

Multimodal dialogue system, due to its many-fold applications, has gained much attention to the researchers and developers in recent times. With the release of large-scale multimodal dialog dataset Saha et al. 2018 on the fashion domain, it has been possible to investigate the dialogue systems having both textual and visual modalities. Response generation is an essential aspect of every dialogue system, and making the responses diverse is an important problem. For any goal-oriented conversational agent, the system’s responses must be informative, diverse and polite, that may lead to better user experiences. In this paper, we propose an end-to-end neural framework for generating varied responses in a multimodal dialogue setup capturing information from both the text and image. Multimodal encoder with co-attention between the text and image is used for focusing on the different modalities to obtain better contextual information. For effective information sharing across the modalities, we combine the information of text and images using the BLOCK fusion technique that helps in learning an improved multimodal representation. We employ stochastic beam search with Gumble Top K-tricks to achieve diversified responses while preserving the content and politeness in the responses. Experimental results show that our proposed approach performs significantly better compared to the existing and baseline methods in terms of distinct metrics, and thereby generates more diverse responses that are informative, interesting and polite without any loss of information. Empirical evaluation also reveals that images, while used along with the text, improve the efficiency of the model in generating diversified responses.

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…Sampling from discrete distributions can be achieved with (among others) inverse transform sampling, or the Gumbel-max trick [4] (see Section 4.1.1) and extensions thereof (see Section 4.3). Gumbel-based sampling algorithms have, for example, been used for (discrete) action selection in a multi-armed bandit setting [10], for sampling data points in active learning [11], for text generation in dialog systems [12], or in translation tasks [13], [14].…”

A Review of the Gumbel-max Trick and its Extensions for Discrete Stochasticity in Machine Learning

“…Sampling from discrete distributions can be achieved with (among others) inverse transform sampling, or the Gumbel-max trick [4] (see Section 4.1.1) and extensions thereof (see Section 4.3). Gumbel-based sampling algorithms have for example been used for (discrete) action selection in a multi-armed bandit setting [10], for sampling data points in active learning [11], for text generation in dialog systems [12] or in translation tasks [13], [14].…”

Aspect-Aware Response Generation for Multimodal Dialogue System

“…The authors in [12] used a hierarchical attention mechanism for generating responses on the MMD dataset. In [20], the authors proposed a stochastic method for generating diverse responses in a multimodal dialogue setup. Multi-domain multi-modal aspect controlled response generation task was introduced in [21].…”