Visual Dialogue State Tracking for Question Generation

Abstract

GuessWhat?! is a visual dialogue task between a guesser and an oracle. The guesser aims to locate an object supposed by the oracle oneself in an image by asking a sequence of Yes/No questions. Asking proper questions with the progress of dialogue is vital for achieving successful final guess. As a result, the progress of dialogue should be properly represented and tracked. Previous models for question generation pay less attention on the representation and tracking of dialogue states, and therefore are prone to asking low quality questions such as repeated questions. This paper proposes visual dialogue state tracking (VDST) based method for question generation. A visual dialogue state is defined as the distribution on objects in the image as well as representations of objects. Representations of objects are updated with the change of the distribution on objects. An object-difference based attention is used to decode new question. The distribution on objects is updated by comparing the question-answer pair and objects. Experimental results on GuessWhat?! dataset show that our model significantly outperforms existing methods and achieves new state-of-the-art performance. It is also noticeable that our model reduces the rate of repeated questions from more than 50% to 21.9% compared with previous state-of-the-art methods.

They Are Not All Alike: Answering Different Spatial Questions Requires Different Grounding Strategies

“…After the introduction of the supervised baseline models (de Vries et al, 2017), several models have been proposed for the Questioner, which are mostly based on reinforcement learning (Sang-Woo et al, 2019;Zhang et al, 2018b;Zhao and Tresp, 2018;Zhang et al, 2018a;Gan et al, 2019;Pang and Wang, 2020). For these models, the role of the Oracle is even more salient than for models based on supervised or cooperative learning since they are reinforced to ask those questions the Oracle is good at answering.…”

“…!, instead, work has been done mostly, if not only, on the questioner. Current models trained with reinforcement learning achieve high task success; they adapt to the oracle limitations and end-up asking questions that are linguistically simpler than those asked by humans Pang and Wang, 2020).…”

On the role of effective and referring questions in GuessWhat?!

“…In this paper we are particularly interested in the GuessWhat?! models that generate questions explicitly modelling the dialogue history Shukla et al, 2019;Pang and Wang, 2020). 1…”

“…The need of going beyond this metric to evaluate the quality of the dialogues has already been observed. So far attention has been put on the linguistic skills of the models (Shukla et al, 2019;Shekhar et al, 2019) and their dialogue strategies (Shekhar et al, 2018;Pang and Wang, 2020). But still the models are evaluated without considering how much each question contributes to the goal.…”

A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

“…Recent years have witnessed an increasing attention in visually grounded dialogues (Zarrieß et al, 2016;de Vries et al, 2018;Alamri et al, 2019;Narayan-Chen et al, 2019). Despite the impressive progress on benchmark scores and model architec-tures (Das et al, 2017b;Wu et al, 2018;Kottur et al, 2018;Gan et al, 2019;Shukla et al, 2019;Niu et al, 2019;Zheng et al, 2019;Kang et al, 2019;Murahari et al, 2019;Pang and Wang, 2020), there have also been critical problems pointed out in terms of dataset biases (Goyal et al, 2017;Chattopadhyay et al, 2017;Massiceti et al, 2018;Chen et al, 2018;Kottur et al, 2019;Kim et al, 2020;Agarwal et al, 2020) which obscure such contributions. For instance, Cirik et al (2018) points out that existing dataset of reference resolution may be largely solvable without recognizing the full referring expressions (e.g.…”