Dual-path Convolutional Image-Text Embeddings with Instance Loss

Abstract

Matching images and sentences demands a fine understanding of both modalities. In this article, we propose a new system to discriminatively embed the image and text to a shared visual-textual space. In this field, most existing works apply the ranking loss to pull the positive image/text pairs close and push the negative pairs apart from each other. However, directly deploying the ranking loss on heterogeneous features (i.e., text and image features) is less effective, because it is hard to find appropriate triplets at the beginning. So the naive way of using the ranking loss may compromise the network from learning inter-modal relationship. To address this problem, we propose the instance loss, which explicitly considers the intra-modal data distribution. It is based on an unsupervised assumption that each image/text group can be viewed as a class. So the network can learn the fine granularity from every image/text group. The experiment shows that the instance loss offers better weight initialization for the ranking loss, so that more discriminative embeddings can be learned. Besides, existing works usually apply the off-the-shelf features, i.e., word2vec and fixed visual feature. So in a minor contribution, this article constructs an end-to-end dual-path convolutional network to learn the image and text representations. End-to-end learning allows the system to directly learn from the data and fully utilize the supervision. On two generic retrieval datasets (Flickr30k and MSCOCO), experiments demonstrate that our method yields competitive accuracy compared to state-of-the-art methods. Moreover, in language-based person retrieval, we improve the state of the art by a large margin. The code has been made publicly available.

Hybrid Attention Network for Language-Based Person Search

“…Similarly, Zheng et al put forward a method using CNN to learn both text features and image features, using classification loss function to pre-train distinguishing features, and then using ranking function to train features matching. The method achieved good person search performance [ 17 ]. Chen et al proposed a text and image block matching method to catch the local similarity [ 32 ].…”

“…To deal with the challenge of cross-modality, we proposed a cross-modal attention mechanism and joint loss function. Most of the existing works have independently extracted the features of each modality, and then measured the cross-modal correlation and similarity of the features [ 7 , 17 ]. However, the features of different modalities are quite different and noisy, and the correlation between features is not strong because of cross-modal heterogeneity.…”

Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding

“…where Φ cnn , Φ lstm and Φ linear are learning parameters of the CNN, LSTM and linear transformation, respectively. During learning, we follow (Faghri et al 2017;Zheng et al 2017) to exploit a two-step learning strategy to prevent the over-fitting problem. In the first step, we fix the parameters of pretrained residual CNN, and only optimize the parameters of LSTM and linear transformation.…”

Multiple Saliency and Channel Sensitivity Network for Aggregated Convolutional Feature

“…Zheng et al (Zheng et al 2017) proposed the instance loss for instance-level image-text matching. Based on the assumption that each image/text group is distinct, they viewed each image/text group as a class.…”

“…a very limited training dataset containing about one-seventh of (Gordo et al 2016) with only image level labels. For the unsupervised mode, motivated by the instance loss for image-text matching (Zheng et al 2017), we leverage it to train our multiple saliency block (MSB) without any supervised information. For the supervised one, we use standard classification loss and triplet loss with batch hard mining (Hermans, Beyer, and Leibe 2017) to fine-tune the whole network for two-stage learning with only image-level labels.…”