Distributed Kd-Trees for Ultra Large Scale Object Recognition

Abstract

Distributed Kd-Trees is a method for building image retrieval systems that can handle hundreds of millions of images. It is based on dividing the Kd-Tree into a "root subtree" that resides on a root machine, and several "leaf subtrees", each residing on a leaf machine. The root machine handles incoming queries and farms out feature matching to an appropriate small subset of the leaf machines. Our implementation employs the MapReduce architecture to efficiently build and distribute the Kd-Tree for millions of images. It can run on thousands of machines, and provides orders of magnitude more throughput than the state-of-the-art, with better recognition performance. We show experiments with up to 100 million images running on 2048 machines, with run time of a fraction of a second for each query image.

Parallel similarity search based on the dimensions value cardinalities of image descriptor vectors

“…(2) KD-Trees [1]: Following the evaluation strategy of [1], we used the parameter of backtracking steps, denoting the fixed budget for doing backtracking steps for every dimension which is shared among all the KD-Trees searched for this dimension. The backtracking steps for KD-Trees 10 were varied in (0.5%, 1%, 3%, 5%, 7%, 9%, 10%, 11%, 12%), expressed as a percentage of the total dataset size N .…”

“…For instance, Aly et al [1] used the Map-Reduce architecture to efficiently build and parallelize the KD-Tree index. A single KD-Tree is considered, where the top of the tree is located on a single root-node and the bottom part of the tree is divided into several machines, called leafnodes.…”

“…Then, similarity search is performed into the leafnodes, whereas the root-node aggregates the results of the leaf-nodes and returns the top-k results. The disadvantage of the aforementioned similarity search strategy of [1] is that a high preprocessing cost is required to construct both global and local complex index structures per machine.…”

“…In line 1, a set V pk is generated (|V pk | N ), which consists of the descriptor vectors with primary key pk equal to vqp 1 …”

“…Given (a) M machines, (b) the set V of the N D-dimensional image descriptor vectors, and (c) the descriptor vector vq of1 The initial value vi,j has been reordered based on p. a query image q 2 , P-DVC supports the following functionalities: (a) parallel preprocessing of the N descriptor vectors to generate the global double linked list L with the logical sorted positions of all N descriptor vectors based on DVC, (b) insertion of the query vq to global double linked list L in real-time, by computing the correct position posq, and (c) parallel query processing to retrieve the top-k similar results to query vq in a result set R based on a predefined constant range 2W , expressed as a percentage of the dataset N .4.2 Preprocessing AlgorithmsRole: To generate the global double linked list L with the logical sorted positions of all N descriptor vectors. The basic components are: (a) the M Dimension Value Cardinality Extractors (Section 4.2.1), (b) the Priority Index (Section 4.2.2), (c) the M Image Sorters and the Global Image Sorter (Section 4.2.3).…”