Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using Transformer Encoders

Abstract

Despite the evolution of deep-learning-based visual-textual processing systems, precise multi-modal matching remains a challenging task. In this work, we tackle the problem of accurate cross-media retrieval through image-sentence matching based on word-region alignments using supervision only at the global imagesentence level. In particular, we present an approach called Transformer Encoder Reasoning and Alignment Network (TERAN). TERAN enforces a fine-grained match between the underlying components of images and sentences, i.e., image regions and words, respectively, in order to preserve the informative richness of both modalities. The proposed approach obtains state-of-the-art results on the image retrieval task on both MS-COCO and Flickr30k. Moreover, on MS-COCO, it defeats current approaches also on the sentence retrieval task. Given our long-term interest in scalable cross-modal information retrieval, TERAN is designed to keep the visual and textual data pipelines well separated. In fact, cross-attention links invalidate any chance to separately extract visual and textual features needed for the online search and the offline indexing steps in large-scale retrieval systems. In this respect, TERAN merges the information from the two domains only during the final alignment phase, immediately before the loss computation. We argue that the fine-grained alignments produced by TERAN pave the way towards the research for effective and efficient methods for large-scale cross-modal information retrieval. We compare the effectiveness of our approach against the best eight methods in this research area. On the MS-COCO 1K test set, we obtain an improvement of 3.5% and 1.2% respectively on the image and the sentence retrieval tasks on the Recall@1 metric. The code used for the experiments is publicly available on GitHub at https://github.com/mesnico/TERAN.

Keep the Caption Information: Preventing Shortcut Learning in Contrastive Image-Caption Retrieval

“…Recently, there has been a shift to transformer-based [52] network architectures for both the image and caption encoder. Messina et al [38,39] introduce a transformer-based network architecture solely trained for the ICR task. Since then, several transformerbased methods have been introduced [8,22,28,31,35]; some of them combine the image and caption encoder into one unified architecture.…”

“…Both the images and captions are mapped into a shared latent space by two encoders, which correspond to the two modalities. These encoders are typically optimized with contrastive loss functions [4,11,12,22,27,29,34,38,39,54,55,59]. Shortcut learning.…”

WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training

“…Most existing multi-modal pre-training models, especially those with the single-tower architecture [20,36,26,39,9,11,41,22,7,14], take an assumption that there exists strong semantic correlation between the input imagetext pair. With this strong assumption, the interaction between image-text pairs can thus be modeled with crossmodal transformers.…”

Recurrent Vision Transformer for Solving Visual Reasoning Problems

“…Initially developed for solving natural language processing tasks, it found its way into the computer vision world, capturing the interest of the whole community. These Transformer-based architectures already proved their effectiveness in many image and video processing tasks [23,24,11,7,2]. The Transformer's success is mainly due to the power of the self-attention mechanism, which can relate every visual token with all the others, creating a powerful relational understanding pipeline.…”