Fast Distributed k NN Graph Construction Using Auto-tuned Locality-sensitive Hashing

Abstract

The k -nearest-neighbors ( k NN) graph is a popular and powerful data structure that is used in various areas of Data Science, but the high computational cost of obtaining it hinders its use on large datasets. Approximate solutions have been described in the literature using diverse techniques, among which Locality-sensitive Hashing (LSH) is a promising alternative that still has unsolved problems. We present Variable Resolution Locality-sensitive Hashing, an algorithm that addresses these problems to obtain an approximate k NN graph at a significantly reduced computational cost. Its usability is greatly enhanced by its capacity to automatically find adequate hyperparameter values, a common hindrance to LSH-based methods. Moreover, we provide an implementation in the distributed computing framework Apache Spark that takes advantage of the structure of the algorithm to efficiently distribute the computational load across multiple machines, enabling practitioners to apply this solution to very large datasets. Experimental results show that our method offers significant improvements over the state-of-the-art in the field and shows very good scalability as more machines are added to the computation.

Scalable feature selection using ReliefF aided by locality‐sensitive hashing

“…An additional advantage of VRLSH is that the estimation procedure finds suitable values for the hyperparameters of the method. It does this by efficiently searching for resolution values and hash function hyperparameters that yield adequately sized buckets to start computing the graph (a process described in detail in Reference [34]). With this automated procedure, the user only needs to provide the data set and indicate the desired number of neighbors to be obtained in the graph.…”

“…Variable resolution LSH (VRLSH) 34,37 is the most recent algorithm that uses the LSH approach. As shown in Algorithm 1 and as described in detail below, it takes iterative LSH steps to obtain an approximate kNN ‐graph.…”

“…Algorithms that obtain approximate graphs-as opposed to exact graphs-have been based on divide-andconquer approaches 31 and local search. 32 However, LSH 33,34 is the approach that has shown most success to date.…”

“…LSH maps data of any size into a smaller space using a hashing function that maximizes the possibility of assigning the same value to elements that are similar. It has been used, for instance, selection 35 and also has been successfully employed for kNN-graph construction 33,34 and for the related problem of kNN search. 36 Solutions are based on isolating small clusters of similar elements, used to calculate small isolated subgraphs that are then merged to obtain the complete approximate graph.…”

Toward more efficient locality‐sensitive hashing via constructing novel hash function cluster

“…Le and Lauw 35 propose a framework named SRPR, which factors in the stochasticity of LSH hash functions when learning real-valued user and item latent vectors, eventually improving the recommendation accuracy after LSH indexing. In paper, 36 Eiras-Franco et al propose an algorithm that creates an approximate kNN graph at a significantly reduced computational cost based on variable resolution LSH.…”