Permutation search methods are efficient, yet faster search is possible

Abstract

We survey permutation-based methods for approximate knearest neighbor search. In these methods, every data point is represented by a ranked list of pivots sorted by the distance to this point. Such ranked lists are called permutations. The underpinning assumption is that, for both metric and non-metric spaces, the distance between permutations is a good proxy for the distance between original points. Thus, it should be possible to efficiently retrieve most true nearest neighbors by examining only a tiny subset of data points whose permutations are similar to the permutation of a query. We further test this assumption by carrying out an extensive experimental evaluation where permutation methods are pitted against state-of-the art benchmarks (the multi-probe LSH, the VP-tree, and proximity-graph based retrieval) on a variety of realistically large data set from the image and textual domain. The focus is on the highaccuracy retrieval methods for generic spaces. Additionally, we assume that both data and indices are stored in main memory. We find permutation methods to be reasonably efficient and describe a setup where these methods are most useful. To ease reproducibility, we make our software and data sets publicly available.

Accurate and Fast Retrieval for Complex Non-metric Data via Neighborhood Graphs

“…objectives of this experimental series is to demonstrate that unlike some prior work [24,25] we deal with substantially non-symmetric data. In the second series, we carry out a fully-fledged retrieval experiment using SW-graph [22] with different index-and query-time symmetrization approaches.…”

“…Malkov and Yashunin proposed an efficient multi-layer neighborhood-graph method called a Hierarchical Navigable Small World (HNSW) [23]. It is a generalization and improvement of the previously proposed method navigable Small World (SW-graph) [22], which has been shown to be quite efficient in the past [22,24] Although there are different approaches to construct a neighborhood graphs, all retrieval strategies known to us rely on a simple semi-greedy graph-traversal algorithm with (possibly) multiple restarts. Such an algorithm keeps a priority queue of elements, Table 1: Data sets which ranks candidates in the order of increasing distance to the query.…”

“…Also note that KL-divergence can be symmetrized with little loss in accuracy, i.e., on the histogramlike data KL-divergence is only mildly non-symmetric. There is prior work on non-metric k-NN search that demonstrated good results specifically for KL-divergence [24,25] for Wiki-d and RCV-d data sets. However, as our experiments clearly show, this work does not use a substantially non-symmetric distance.…”

Accessing very high dimensional spaces in parallel

“…For each query, a candidate list is built with the objects that cannot be discarded when compared with the pivots so they have to be compared directly with the query object. For example, permutation methods [19] are a kind of pivot method with dimension reduction.…”

“…In [19] we can find a very good comparative analysis of several state-of-theart methods for approximate k-NN search. Experiments are performed with a large number of varied datasets represented in both metric and non-metric spaces.…”