Learning Fine-Grained Image Similarity with Deep Ranking

Wang, Jiang; Song, Yang; Leung, Thomas; Rosenberg, Chuck; Wang, Jingbin; Philbin, James; Chen, Bin; Wu, Ying

doi:10.1109/cvpr.2014.180

Cited by 1,139 publications

(886 citation statements)

References 18 publications

Supporting

Mentioning

882

Contrasting

Unclassified

Order By: Relevance

“…Previous works employ standard classification [8], pairwise-loss [24] or Triplet-loss [165], [17] CNN models for fine-tuning. The introduction of Faster R-CNN to instance retrieval is a promising starting point towards more accurate object localization [17].…”

Section: Towards Generic Instance Retrievalmentioning

confidence: 99%

SIFT Meets CNN: A Decade Survey of Instance Retrieval

Zheng

Yang

Tian

2018

IEEE Trans. Pattern Anal. Mach. Intell.

616

319

View full text Add to dashboard Cite

Abstract-In the early days, content-based image retrieval (CBIR) was studied with global features. Since 2003, image retrieval based on local descriptors (de facto SIFT) has been extensively studied for over a decade due to the advantage of SIFT in dealing with image transformations. Recently, image representations based on the convolutional neural network (CNN) have attracted increasing interest in the community and demonstrated impressive performance. Given this time of rapid evolution, this article provides a comprehensive survey of instance retrieval over the last decade. Two broad categories, SIFT-based and CNN-based methods, are presented. For the former, according to the codebook size, we organize the literature into using large/medium-sized/small codebooks. For the latter, we discuss three lines of methods, i.e., using pre-trained or fine-tuned CNN models, and hybrid methods. The first two perform a single-pass of an image to the network, while the last category employs a patch-based feature extraction scheme. This survey presents milestones in modern instance retrieval, reviews a broad selection of previous works in different categories, and provides insights on the connection between SIFT and CNN-based methods. After analyzing and comparing retrieval performance of different categories on several datasets, we discuss promising directions towards generic and specialized instance retrieval.

show abstract

Section: Towards Generic Instance Retrievalmentioning

confidence: 99%

SIFT Meets CNN: A Decade Survey of Instance Retrieval

Zheng

Yang

Tian

2018

IEEE Trans. Pattern Anal. Mach. Intell.

616

319

View full text Add to dashboard Cite

show abstract

“…The first approach consists of submitting pairs of training images (X, Y ) to the network, telling it if they are similar or not [19]. The second approach is to use triplets of images (A, B, C) telling the network that d(f (A), f(B)) should be smaller than d(f (A), f(C)) (where d is a distance function and f the embedding function) [38]. Since we are interested in making a ranking system, the order of proximity is what is important to us and the second approach then better suited.…”

Section: Fine-tuning the Networkmentioning

confidence: 99%

Visual Link Retrieval in a Database of Paintings

Séguin

Striolo

diLenardo

et al. 2016

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. This paper examines how far state-of-the-art machine vision algorithms can be used to retrieve common visual patterns shared by series of paintings. The research of such visual patterns, central to Art History Research, is challenging because of the diversity of similarity criteria that could relevantly demonstrate genealogical links. We design a methodology and a tool to annotate efficiently clusters of similar paintings and test various algorithms in a retrieval task. We show that pretrained convolutional neural network can perform better for this task than other machine vision methods aimed at photograph analysis. We also show that retrieval performance can be significantly improved by fine-tuning a network specifically for this task.

show abstract

“…For example, triplet loss is widely used in many deep verification [33,29] or ranking [40,46,36,31] networks. It is adopted here to enforce the ranking between a query sketch and a pair of positive and negative photos.…”

Section: Shortcuts and Layer Fusion In Deep Learningmentioning

confidence: 99%

“…It is adopted here to enforce the ranking between a query sketch and a pair of positive and negative photos. In the vast majority of cases [40,46,36,31] Euclidean distancebased, or other first-order energy functions are used in the loss formulation. They are first order in the sense that only element-wise comparisons are made, making it sensitive to feature misalignment and meaning that no cross-feature correlation can be exploited in the similarity.…”

Section: Shortcuts and Layer Fusion In Deep Learningmentioning

confidence: 99%

Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval

Song

et al. 2017

2017 IEEE International Conference on Computer Vision (ICCV)

210

217

View full text Add to dashboard Cite

Human sketches are unique in being able to capture both the spatial topology of a visual object, as well as its subtle appearance details. Fine-grained sketch-based image retrieval (FG-SBIR) importantly leverages on such fine-grained characteristics of sketches to conduct instancelevel retrieval of photos. Nevertheless, human sketches are often highly abstract and iconic, resulting in severe misalignments with candidate photos which in turn make subtle visual detail matching difficult. Existing FG-SBIR approaches focus only on coarse holistic matching via deep cross-domain representation learning, yet ignore explicitly accounting for fine-grained details and their spatial context. In this paper, a novel deep FG-SBIR model is proposed which differs significantly from the existing models in that: (1) It is spatially aware, achieved by introducing an attention module that is sensitive to the spatial position of visual details; (2) It combines coarse and fine semantic information via a shortcut connection fusion block; and (3) It models feature correlation and is robust to misalignments between the extracted features across the two domains by introducing a novel higher-order learnable energy function (HOLEF) based loss. Extensive experiments show that the proposed deep spatial-semantic attention model significantly outperforms the state-of-the-art.

show abstract

Learning Fine-Grained Image Similarity with Deep Ranking

Abstract: Abstract

Cited by 1,139 publications

References 18 publications

SIFT Meets CNN: A Decade Survey of Instance Retrieval

SIFT Meets CNN: A Decade Survey of Instance Retrieval

Visual Link Retrieval in a Database of Paintings

Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval

Contact Info

Product

Resources

About