Towards Building Large Scale Multimodal Domain-Aware Conversation Systems

Abstract

While multimodal conversation agents are gaining importance in several domains such as retail, travel etc., deep learning research in this area has been limited primarily due to the lack of availability of large-scale, open chatlogs. To overcome this bottleneck, in this paper we introduce the task of multimodal, domain-aware conversations, and propose the MMD benchmark dataset. This dataset was gathered by working in close coordination with large number of domain experts in the retail domain. These experts suggested various conversations flows and dialog states which are typically seen in multimodal conversations in the fashion domain. Keeping these flows and states in mind, we created a dataset consisting of over 150K conversation sessions between shoppers and sales agents, with the help of in-house annotators using a semi-automated manually intense iterative process. With this dataset, we propose 5 new sub-tasks for multimodal conversations along with their evaluation methodology. We also propose two multimodal neural models in the encode-attend-decode paradigm and demonstrate their performance on two of the sub-tasks, namely text response generation and best image response selection. These experiments serve to establish baseline performance and open new research directions for each of these sub-tasks. Further, for each of the sub-tasks, we present a 'per-state evaluation' of 9 most significant dialog states, which would enable more focused research into understanding the challenges and complexities involved in each of these states.

CAISE: Conversational Agent for Image Search and Editing

“…Multimodal Dialogue. Multimodal dialogue has been actively studied in previous works (Das et al 2017;De Vries et al 2017;Mostafazadeh et al 2017;Saha, Khapra, and Sankaranarayanan 2017;Pasunuru and Bansal 2018;Alamri et al 2019;Haber et al 2019;Kim et al 2019;Moon et al 2020;Shuster et al 2020;Cheng et al 2020). Although all these works involve interesting task setups (question answering/generation, object discovery, shopping, collaborative drawing, response retrieval/generation, image identification/generation, etc.)…”

Polite Task-oriented Dialog Agents: To Generate or to Rewrite?

“…Hence, we follow a similar line of work and propose to enrich fashion dialog agents with politeness. Saha et al (2018) introduced a large-scale multimodal fashion dialog dataset (MMD) built semi-automatically, using field experts, accompanied by two RNN (Cho et al, 2014) models capable of emulating the system responses in a multimodal scenario. Due to its domain, it carries mainly neutral and polite dialog.…”

“…Polite-RL uses the politeness score of a sampled utterance as a measure of politeness that acts as the Reinforcement Learning component of the loss function (see appendix A.4), to guide the generation towards a more polite output. We focused on improving the used embeddings to include a novel lexicon, given that the fashion domain (Saha et al, 2018) differs significantly from the training data (Danescu-Niculescu-Mizil et al, 2013), making out-of-vocabulary situations a major issue. Originally, this model uses embeddings initialized using a Word2Vec model trained on the Google News dataset (Mikolov et al, 2013).…”

“…Originally, this model uses embeddings initialized using a Word2Vec model trained on the Google News dataset (Mikolov et al, 2013). Despite its vocabulary size, the dataset's vocabulary can still leave out a significant portion of the terms used in the fashion-specific datasets (mainly clothes' names and attributes), due to its highly specific domain (Saha et al, 2018).…”

“…• Explored politeness on a novel domain, conversational assistants on the fashion domain (Saha et al, 2018), exposing the opportunities for improving politeness.…”