Attend, Correct And Focus: A Bidirectional Correct Attention Network For Image-Text Matching

Abstract

Image-text matching task aims to learn the fine-grained correspondences between images and sentences. Existing methods use attention mechanism to learn the correspondences by attending to all fragments without considering the relationship between fragments and global semantics, which inevitably lead to semantic misalignment among irrelevant fragments. To this end, we propose a Bidirectional Correct Attention Network (BCAN), which leverages global similarities and local similarities to reassign the attention weight, to avoid such semantic misalignment. Specifically, we introduce a global correct unit to correct the attention focused on relevant fragments in irrelevant semantics. A local correct unit is used to correct the attention focused on irrelevant fragments in relevant semantics. Experiments on Flickr30K and MSCOCO datasets verify the effectiveness of our proposed BCAN by outperforming both previous attention-based methods and state-of-the-art methods. Code can be found at: https://github.com/liuyyy111/BCAN.

Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval

“…SCAN has been used as a baseline for many methods and has led to technological developments since its proposal. Examples include the bidirectional focal attention network (BFAN) [15] and the position focused attention network (PFAN) [16]. In BFAN, irrelevant image regions and words cause deterioration in the correspondence between images and text; thus, they are removed.…”

Ask&Confirm: Active Detail Enriching for Cross-Modal Retrieval with Partial Query

“…Most text-based image retrieval approaches are based on deep neural networks [38,16,18,10,34,5]. The main ob-jective of the retrieval system is to accurately measure the similarity between the inputs from two different modalities.…”

“…Cross-Modal Projection Learning (CMPL) [38] is proposed to pull image and text embeddings into an aligned space. To further enhance the retrieval performance in a fine-grained way, [16,18,10,34] proposed different attention-based approaches, applying visual attention between every image region and word.…”

“…Whereas the others consist of different objects like trees and buses. If the retrieval model receives a complete description including all objects, existing methods [16,18,34] would perform excellently. Due to the partial query focuses only on a single region therefore lacking differentiating information, many scenes may fit its description thus become false positives.…”

“…However, we discover that some cost-free information is always ignored. In detail, most text-image matching methods [16,18,34] extract image features by an object detector [1], but they never leverage the ready-made information of object categories. Another discovery is that the distribution of object categories is not uniform.…”