Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer

Kang, Gi-Cheon; Park, Junseok; Lee, Hwaran; Zhang, Byoung-Tak; Kim, Jin-Hwa

doi:10.18653/v1/2021.findings-emnlp.31

Cited by 6 publications

(5 citation statements)

References 33 publications

(44 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recent studies finetune the VisDial models on the densely annotated labels 3 in the validation dataset and evaluate the models on the test dataset. By following the method applied in SGL+KT [20], we also train the student model on the dense labels. In Table 8, the models like VisDial-BERT and VD-BERT show huge improvements on NDCG and counter-effect on other metrics when utilizing the dense labels.…”

Section: Discussionmentioning

confidence: 99%

“…Prior work has developed a variety attention mechanisms [5][6][7][8][9][10][11][12][13]18] considering the interactions among the image, dialog history, and question. Some studies [14,20] have attempted to discover the semantic structures of the dialog in the context of graph neural networks [64] using the soft attention mechanisms [65]. From the learning algorithm perspective, all of them have relied on supervised learning on VisDial data.…”

Section: Related Workmentioning

confidence: 99%

“…Most of the previous approaches in VisDial [5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20] have trained the dialog agents solely on VisDial data via supervised learning. More recent studies [21][22][23] have employed self-supervised pre-trained models such as BERT [24] or ViLBERT [25] and finetuned them on VisDial data.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training

Kang¹,

Kim²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Visual dialog (VisDial) is a task of answering a sequence of questions grounded in an image, using the dialog history as context. Prior work has trained the dialog agents solely on VisDial data via supervised learning or leveraged pre-training on related vision-and-language datasets. This paper presents a semi-supervised learning approach for visually-grounded dialog, called Generative Self-Training (GST), to leverage unlabeled images on the Web. Specifically, GST first retrieves in-domain images through out-of-distribution detection and generates synthetic dialogs regarding the images via multimodal conditional text generation. GST then trains a dialog agent on the synthetic and the original VisDial data. As a result, GST scales the amount of training data up to an order of magnitude that of VisDial (1.2M → 12.9M QA data). For robust training of the generated dialogs, we also propose perplexity-based data selection and multimodal consistency regularization. Evaluation on VisDial v1.0 and v0.9 datasets shows that GST achieves new stateof-the-art results on both datasets. We further observe strong performance gains in the low-data regime (up to 9.35 absolute points on NDCG). * Equal contribution Preprint. Under review.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training

Kang¹,

Kim²,

Kim³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Then the high-level representation of cross-modal information is used to generate visually and contextually coherent responses. By leveraging a new structural loss function, Sparse Graph Learning (SGL) [69] learns the semantic relationships among dialog rounds and predicts sparse structures of the visually-grounded dialog. To identify multiple possible answers, the Knowledge Transfer (KT) method is also proposed to extract the soft scores of each candidate answer.…”

Section: Graph-based Semantic Relationmentioning

confidence: 99%

Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review

Wang¹,

Guo²,

Zeng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The intelligent dialogue system, aiming at communicating with humans harmoniously with natural language, is brilliant for promoting the advancement of human-machine interaction in the era of artificial intelligence. With the gradually complex human-computer interaction requirements (e.g., multimodal inputs, time sensitivity), it is difficult for traditional text-based dialogue system to meet the demands for more vivid and convenient interaction. Consequently, Visual-Context Augmented Dialogue System (VAD), which has the potential to communicate with humans by perceiving and understanding multimodal information (i.e., visual context in images or videos, textual dialogue history), has become a predominant research paradigm. Benefiting from the consistency and complementarity between visual and textual context, VAD possesses the potential to generate engaging and context-aware responses. For depicting the development of VAD, we first characterize the concepts and unique features of VAD, and then present its generic system architecture to illustrate the system workflow. Subsequently, several research challenges and representative works are detailed investigated, followed by the summary of authoritative benchmarks. We conclude this paper by putting forward some open issues and promising research trends for VAD, e.g., the cognitive mechanisms of human-machine dialogue under cross-modal dialogue context, and knowledge-enhanced cross-modal semantic interaction.CCS Concepts: • Human-centered computing → HCI theory, concepts and models; • Computing methodologies → Discourse, dialogue and pragmatics.

show abstract

“…In the generatrive setting, we consider recent generative baselines, including KBGN [17], DAM [19], LTMI [28], MITVG [9], GOG-Multi-Gen [7], and LTMI-LG [8]. In the discriminative setting, we consider state-of-the-art discriminative baselines, including KBGN [17], Modality-Balanced [21], FGA [35], Du-alVD [18], MCA [1], P1+P2 [32], SGL [20], CARE [25], LTMI-GOG-Multi [7], VisDial-BERT [27], VD-BERT [37], and VD-PCR [42].…”

Section: Baselinesmentioning

confidence: 99%

Unified Multimodal Model with Unlikelihood Training for Visual Dialog

Wang

Jiang

2022

Proceedings of the 30th ACM International Conference on Multimedia

View full text Add to dashboard Cite

The task of visual dialog requires a multimodal chatbot to answer sequential questions from humans about image content. Prior work performs the standard likelihood training for answer generation on the positive instances (involving correct answers). However, the likelihood objective often leads to frequent and dull outputs and fails to exploit the useful knowledge from negative instances (involving incorrect answers). In this paper, we propose a Unified Multimodal Model with UnLikelihood Training, named UniMM-UL, to tackle this problem. First, to improve visual dialog understanding and generation by multi-task learning, our model extends ViLBERT from only supporting answer discrimination to holding both answer discrimination and answer generation seamlessly by different attention masks. Specifically, in order to make the original discriminative model compatible with answer generation, we design novel generative attention masks to implement the autoregressive Masked Language Modeling (autoregressive MLM) task. And to attenuate the adverse effects of the likelihood objective, we exploit unlikelihood training on negative instances to make the model less likely to generate incorrect answers. Then, to utilize dense annotations, we adopt different fine-tuning methods for both generating and discriminating answers, rather than just for discriminating answers as in the prior work. Finally, on the VisDial dataset, our model achieves the best generative results (69.23 NDCG score). And our model also yields comparable discriminative results with the state-of-the-art in both single-model and ensemble settings (75.92 and 76.17 NDCG scores).

show abstract

Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer

Cited by 6 publications

References 33 publications

The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training

The Dialog Must Go On: Improving Visual Dialog via Generative Self-Training

Enabling Harmonious Human-Machine Interaction with Visual-Context Augmented Dialogue System: A Review

Unified Multimodal Model with Unlikelihood Training for Visual Dialog

Contact Info

Product

Resources

About