Feature Fusion for Image Retrieval With Adaptive Bitrate Allocation and Hard Negative Mining

Abstract

By combining Convolutional Neural Network (CNN) descriptor and Compact Descriptors for Visual Search (CDVS), the visual search performance can be boosted. However, some redundancies still exist in the CDVS representation and the hard negative mining is not very accurate when training CNN embeddings. In this paper, we propose a high performance image retrieval scheme based on descriptor fusion. In detail, we first propose a more compact CDVS descriptor database building scheme through bitrate allocation, which can reduce information redundancy and boost image retrieval performance. We then propose a highly accurate CDVS-guided hard negative mining scheme when training CNN embeddings. In the hard negative selection, the CDVS descriptor and the CNN embedding are adaptively weighted together to achieve more precise decisions. Finally, the retrieval result is further refined through CDVS local descriptor matching by removing the irrelevant targets from the top positions. Extensive experimental results show that the proposed method outperforms the recent hybrid method and several other anchors remarkably, and produces better visual search performance. Codes and some models are available at https://github.com/WendyDong/ImageRetrieval_DF_CDVS. INDEX TERMS Image retrieval, CDVS, CNN, hard negative mining, deep learning.

Construct Informative Triplet with Two-stage Hard-sample Generation

“…where i is the i th query index and N is the number of total queries [3]. M i relevant is the number of relevant images corresponding to the i th query, r is the rank and P (r) is the precision at the cut-off rank of r. The employed Recall@K metric is determined by the existence of at least one correct retrieved sample in the K nearest neighbors [26].…”

“…Deep metric learning aims to learn an embedding that can decrease the distance between matching pairs and increase the distance between non-matching pairs [3,28]. To learn such an embedding, many works are proposed, such as the traditional contrastive loss [12,29] under Siamese network and triplet loss [13,25].…”

“…Millions of images are produced every day from these types of digital equipment, and thus how to accurately search images from a large dataset poses a great challenge. To address this problem, the content-based image retrieval (CBIR) technique is proposed to search for images representing the same object or scene as the one depicted in a query image [2,3].…”