Deep Bayesian Network for Visual Question Generation

Abstract

Generating natural questions from an image is a semantic task that requires using vision and language modalities to learn multimodal representations. Images can have multiple visual and language cues such as places, captions, and tags. In this paper, we propose a principled deep Bayesian learning framework that combines these cues to produce natural questions. We observe that with the addition of more cues and by minimizing uncertainty in the among cues, the Bayesian network becomes more confident. We propose a Minimizing Uncertainty of Mixture of Cues (MUMC), that minimizes uncertainty present in a mixture of cues experts for generating probabilistic questions. This is a Bayesian framework and the results show a remarkable similarity to natural questions as validated by a human study. We observe that with the addition of more cues and by minimizing uncertainty among the cues, the Bayesian framework becomes more confident. Ablation studies of our model indicate that a subset of cues is inferior at this task and hence the principled fusion of cues is preferred. Further, we observe that the proposed approach substantially improves over state-of-the-art benchmarks on the quantitative metrics (BLEU-n, METEOR, ROUGE, and CIDEr). Here we provide project link for Deep Bayesian VQG https: //delta-lab-iitk.github.io/BVQG/.

Guiding Visual Question Generation

“…We compare our models with four recently proposed VQG models Information Maximising VQG (IMVQG) (Krishna, Bernstein, and Fei-Fei 2019), What BERT Sees (WBS) (Scialom et al 2020), Deep Bayesian Network (DBN) (Patro et al 2020), and Category Consistent Cyclic VQG (C3VQG) (Uppal et al 2020). Out of these four papers, IMVQG's training and evaluation setup is the most similar to ours.…”

“…Jain, Zhang, and Schwing (2017) proposed a model using a VAE instead of a GAN, however their improved results require the use of a target answer during inference. To overcome this requirement, Krishna, Bernstein, and Fei-Fei (2019) Other work, such as Patro et al (2018), Patro et al (2020) and Uppal et al (2020), either do not include BLEU scores higher than BLEU-1, which is not very informative, or address variants of the VQG task. In the latter case the models fail to beat previous SoTA on BLEU-4 for standard VQG.…”

Latent Variable Models for Visual Question Answering

“…In general, Bayesian approach considerably outperforms the quantitative metrics in state-of-the-art benchmarks. There has been some work on exploring Bayesian and latent variable methods for Visual Question Generation (Patro et al, 2020;Krishna et al, 2019). However, in our work, we frame VQA under the variational inference framework where we approximate both the variational and generative distribution during training.…”

Goal-Driven Visual Question Generation from Radiology Images

“…In contrast to answering visual questions about images, generating questions has received little attention so far. A few recent works have attempted to generate questions from images in the open domain [24][25][26]. However, the task of VQG in the medical domain has not been well-studied.…”