Optimization of All Pairs Similarity Search

Chen, Yuechen; Tang, Xinhuai; Liu, Bing; Chen, Delai

doi:10.1109/csci.2015.16

Cited by 1 publication

(1 citation statement)

References 23 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditional NNG construction methods could not scale to sets of object this large. Given the growing popularity of cloud computing, some of the traditional NNS methods were ported to cloud programming frameworks developed for dealing with big data (e.g., Hadoop, Spark) [1,2,14,18,31,38,39,43,46]. Most of the solutions use the MapReduce [20] framework and can be split into two categories.…”

Section: Related Workmentioning

confidence: 99%

Parallel cosine nearest neighbor graph construction

Anastasiu

Karypis²

2019

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

The nearest neighbor graph is an important structure in many data mining methods for clustering, advertising, recommender systems, and outlier detection. Constructing the graph requires computing up to n 2 similarities for a set of n objects. This high complexity has led researchers to seek approximate methods, which find many but not all of the nearest neighbors. In contrast, we leverage shared memory parallelism and recent advances in similarity joins to solve the problem exactly. Our method considers all pairs of potential neighbors but quickly filters pairs that could not be a part of the nearest neighbor graph, based on similarity upper bound estimates. The filtering is data dependent and not easily predicted, which poses load balance challenges in parallel execution. We evaluated our methods on several real-world datasets and found they work up to two orders of magnitude faster than existing methods, display linear strong scaling characteristics, and incur less than 1% load imbalance during filtering.

show abstract