CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval

Abstract

Text-image cross-modal retrieval is a challenging task in the field of language and vision. Most previous approaches independently embed images and sentences into a joint embedding space and compare their similarities. However, previous approaches rarely explore the interactions between images and sentences before calculating similarities in the joint space. Intuitively, when matching between images and sentences, human beings would alternatively attend to regions in images and words in sentences, and select the most salient information considering the interaction between both modalities. In this paper, we propose Crossmodal Adaptive Message Passing (CAMP), which adaptively controls the information flow for message passing across modalities. Our approach not only takes comprehensive and fine-grained cross-modal interactions into account, but also properly handles negative pairs and irrelevant information with an adaptive gating scheme. Moreover, instead of conventional joint embedding approaches for textimage matching, we infer the matching score based on the fused features, and propose a hardest negative binary crossentropy loss for training. Results on COCO and Flickr30k significantly surpass state-of-the-art methods, demonstrating the effectiveness of our approach.

Context-Aware Multi-View Summarization Network for Image-Text Matching

“…According to the granularity of representation, studies on imagetext matching can be categorized into two groups: 1) global embedding based methods [5,6,34,44], and 2) local inference based methods [3,16,18,21,26,37]. The former ones first embed the whole images and sentences into a joint embedding space, and then calculate the visual-semantic similarity.…”

“…For instance, Lee et al [16] presented a cross attention to align both image regions and words to infer image-text similarity. Wang et al [37] proposed a crossmodal adaptive message passing method to perform fine-grained interaction and filter irrelevant information with a gating strategy. Liu et al [21] designed a focal attention network comprising the preassigning and re-assigning attention, which focuses on eliminating irrelevant fragments.…”

“…To justify the effectiveness of our proposal, we compared our proposed CAMERA with the following state-of-the-art baselines: VQA [20], sm-LSTM [11], VSE++ [5], CMPM + CMPC [44], SCAN [16], SCG [31], TIMAM [30], SAEM [40], CAMP [37], BFAN [21], PFAN [36], SAN [12], VSRN [17], and SGM [35]. Note that we directly quoted the results from their original papers.…”

“…As a fundamental task of multimodal interaction, image-text matching, focusing on measuring the semantic similarity between an image and a text, has attracted extensive research attention. It indeed facilitates various applications, such as cross-modal retrieval [3,37], visual question answering (VQA) [2,24] and multimedia understanding [10,38,39].…”

“…However, it would be difficult for such a compact representation to capture fine-grained semantic details in texts and images. Afterwards, to further exploit the fine-grained relationships between images and texts, several studies [16,18,37] have adopted the attention mechanism to perform semantic alignments between visual regions and words. Particularly, they aggregate fragment features guided by local cross-modal affinities, i.e., region-word similarities, to obtain global representations.…”