Image-Chat: Engaging Grounded Conversations

Abstract

To achieve the long-term goal of machines being able to engage humans in conversation, our models should captivate the interest of their speaking partners. Communication grounded in images, whereby a dialogue is conducted based on a given photo, is a setup naturally appealing to humans (Hu et al., 2014). In this work we study large-scale architectures and datasets for this goal. We test a set of neural architectures using state-of-the-art image and text representations, considering various ways to fuse the components. To test such models, we collect a dataset of grounded human-human conversations, where speakers are asked to play roles given a provided emotional mood or style, as the use of such traits is also a key factor in engagingness (Guo et al., 2019). Our dataset, Image-Chat, consists of 202k dialogues over 202k images using 215 possible style traits. Automatic metrics and human evaluations of engagingness show the efficacy of our approach; in particular, we obtain state-of-the-art performance on the existing IGC task, and our best performing model is almost on par with humans on the Image-Chat test set (preferred 47.7% of the time).

Augmenting Transformers with KNN-Based Composite Memory for Dialog

“…We augment Transformer sequence to sequence (seq2seq) networks on the encoder side with KIF to improve generative dialog models. We experiment on two dialog tasks, Wizard of Wikipedia and Engaging ImageChat (Shuster et al, 2020). In both datasets, models must leverage information external to the dialog history alone-in Wizard of Wikipedia, the chat requires access to knowledgeable facts and in Engaging ImageChat, discussion about a specific image.…”

“…Agents are assigned one of 215 personalities (e.g., sweet, caring, excited) to increase engagingness. Previous work (Shuster et al, 2020 identified that both crowdworkers and models, when provided with personalities, produced more diverse, interesting responses, as evaluated by humans.…”

“…We use a multimodal neural network designed to handle both image input and text input. Following Shuster et al (2020), the images are encoded using a pre-trained ResNeXt network (Xie et al, 2017). To extract the final image representation, we project the 2048-dimensional output of the image encoder to 512-dimensions using a deep multilayer perceptron with ReLU activation units.…”

“…The conversation history, which includes the one-word personality, is encoded with a Transformer encoder network. The image and conversation are integrated using the Multimodal-Sum-Combiner module proposed in Shuster et al (2020).…”

“…In the other scenario, Engaging ImageChat (Shuster et al, 2020), crowdworkers are provided with images and asked to have a conversation inspired by or about the image. In this case, the retrieved external knowledge is images and their associated dialogs.…”