SA-VQA: Structured Alignment of Visual and Semantic Representations for Visual Question Answering

Abstract

Visual Question Answering (VQA) attracts much attention from both industry and academia. As a multi-modality task, it is challenging since it requires not only visual and textual understanding, but also the ability to align crossmodality representations. Previous approaches extensively employ entity-level alignments, such as the correlations between the visual regions and their semantic labels, or the interactions across question words and object features. These attempts aim to improve the cross-modality representations, while ignoring their internal relations. Instead, we propose to apply structured alignments, which work with graph representation of visual and textual content, aiming to capture the deep connections between the visual and textual modalities. Nevertheless, it is nontrivial to represent and integrate graphs for structured alignments. In this work, we attempt to solve this issue by first converting different modality entities into sequential nodes and the adjacency graph, then incorporating them for structured alignments.As demonstrated in our experimental results, such a structured alignment improves reasoning performance. In addition, our model also exhibits better interpretability for each generated answer. The proposed model, without any pretraining, outperforms the state-of-the-art methods on GQA dataset, and beats the non-pretrained state-of-the-art methods on VQA-v2 dataset.

GFSNet: Gaussian Fourier with Sparse Attention Network for Visual Question Answering

“…When the value exceeds t i , we record the position (i, j) in the position matrix P . The entire process is mathematically described in equations ( 11) and (12). By adopting this approach, we can effectively concentrate on critical information while also considering the positional context of the selected elements, leading to an improved feature representation and a deeper understanding of the underlying patterns in the image.…”

“…Other research endeavors concentrate on modeling and understanding image information to achieve a profound comprehension of image content and address intricate reasoning questions related to the queries. In the realm of semantic relationship modeling in images, researchers primarily employ graph neural network methods [10][11][12][13] for explicit and implicit reasoning. The objective of these methods is to capture semantic relationships within images, thereby enabling the model to better understand the connection between image content and questions.…”

DSGEM: Dual scene graph enhancement module‐based visual question answering

“…To introduce more explicit and explainable information to represent the complex relationships, the visual scene graph [10] that contains more edge semantics than the classic Laplacian graph is introduced into the VQA task. Visual scene graph used in VQA methods is mainly constructed according to three types of explicable structural information: relative spatial information [11,28,29], visual relationship information [30][31][32] and KG information [18,33,34].…”