CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples

Abstract

Convolutional Neural Networks (CNNs) achieve state-of-theart performance in many computer vision tasks. However, this achievement is preceded by extreme manual annotation in order to perform either training from scratch or fine-tuning for the target task. In this work, we propose to fine-tune CNN for image retrieval from a large collection of unordered images in a fully automated manner. We employ state-of-the-art retrieval and Structure-from-Motion (SfM) methods to obtain 3D models, which are used to guide the selection of the training data for CNN fine-tuning. We show that both hard positive and hard negative examples enhance the final performance in particular object retrieval with compact codes.

SIFT Meets CNN: A Decade Survey of Instance Retrieval

“…pass to the CNN model. The visual representations exhibit improved discriminative ability [17], [24].…”

“…Usually, spatial verification [12] is employed for re-ranking and obtaining the ROIs from which the local features undergo average pooling. AQE is used by many later works [10], [17], [24] as a standard tool. The recursive AQE and the scale-band recursive QE are effective improvement but incur more computational cost [100].…”

“…The Landmark dataset is later used by Gordo et al [17] for fine-tuning, after an automatic cleaning approach based on SIFT matching. In [24], Radenović et al employ the retrieval and StructureFrom-Motion methods to build 3D landmark models so that images depicting the same architecture can be grouped. Using this labeled dataset, the linear discriminative projections (denoted as L w in Table 5) outperform the previous whitening technique [53].…”

“…A standard siamese network based on AlexNet and the contrastive loss is employed in [143]. In [24], Radenović et al propose to replace the FC layers with a MAC layer [10]. Moreover, with the 3D architecture models built in [24], training pairs can be mined.…”

“…In [24], Radenović et al propose to replace the FC layers with a MAC layer [10]. Moreover, with the 3D architecture models built in [24], training pairs can be mined. Positive image pairs are selected based on the number of co-observed 3D points (matched SIFT features), while hard negatives are defined as those with small distances in their CNN descriptors.…”