Multi-grained Attention with Object-level Grounding for Visual Question Answering

Abstract

Attention mechanisms are widely used in Visual Question Answering (VQA) to search for visual clues related to the question. Most approaches train attention models from a coarsegrained association between sentences and images, which tends to fail on small objects or uncommon concepts. To address this problem, this paper proposes a multi-grained attention method. It learns explicit wordobject correspondence by two types of wordlevel attention complementary to the sentenceimage association. Evaluated on the VQA benchmark, the multi-grained attention model achieves competitive performance with stateof-the-art models. And the visualized attention maps demonstrate that addition of objectlevel groundings leads to a better understanding of the images and locates the attended objects more precisely.

Language-Mediated, Object-Centric Representation Learning

“…Visual concept learning. Learning visual concepts from language and other forms of supervision provides useful representations for various downstream tasks, such as image captioning (Yin and Ordonez, 2017;Wang et al, 2018), visual-question answering (Yi et al, 2018;Huang et al, 2019), shape differentiation (Achlioptas et al, 2019), image classification (Mu et al, 2020), and scene manipulation (Prabhudesai et al, 2020). Previous work has been focusing on various types of representations (Ren et al, 2016;Wu et al, 2017), training algorithms (Faghri et al, 2018;Morgado et al, 2020) and supervision (Johnson et al, 2016;Yang et al, 2018).…”

Language-Mediated, Object-Centric Representation Learning

“…Visual concept learning. Learning visual concepts from language and other forms of supervision provides useful representations for various downstream tasks, such as image captioning (Yin and Ordonez, 2017;Wang et al, 2018), visual-question answering (Yi et al, 2018;Huang et al, 2019), and scene manipulation (Prabhudesai et al, 2020). Previous work has been focusing on various types of representations (Ren et al, 2016;Wu et al, 2017), training algorithms (Faghri et al, 2018) and supervision (Johnson et al, 2016;Yang et al, 2018).…”

An animated picture says at least a thousand words: Selecting Gif-based Replies in Multimodal Dialog

“…Because many of these systems are designed to support voicebased dialog, they overlook non-textual forms of interaction used in social media conversations. In parallel, multimodal NLP systems have been developed for image data, often focusing on image-totext tasks such as image captioning (Melas-Kyriazi et al, 2018;Sharma et al, 2018) and visual question answering (Antol et al, 2015;Huang et al, 2019;Khademi, 2020). More recent work has focused on the reverse text-to-image dimension, such as generating an image from a description (Niu et al, 2020;Ramesh et al, 2021).…”