Projection Based Large Scale High-Dimensional Data Similarity Join Using MapReduce Framework

Abstract

Similarity join has been widely used in many data analysis and data mining applications, we mainly focus on the scalability and performance problem of similarity join query on massive highdimensional data set. p-stable distribution based projection scheme can implement dimension reduction effectively. Three novel approaches based on projection scheme are proposed to deal with massive highdimensional data similarity join problem: Single projection method, Multiple projection method and Projection space partitioning method. Comprehensive experimental tests were performed to evaluate the performance of the above approaches. The experimental results show that the proposed approaches in this paper have good performance and scalability.

Scalable resource description framework clustering: A distributed approach for analyzing knowledge graphs using minHash locality sensitive hashing

“…Approximate similarity join 25 needs two datasets. It then compares the entities in these datasets and returns pairs of entities whose distance is less than the threshold ( t ) provided by the user.…”

MapReduce-Based Dynamic Partition Join with Shannon Entropy for Data Skewness

“…To handle the problem that existing MapReducebased filtering methods require multiple MapReduce jobs to improve the join performance, the adaptive filter-based join algorithm is proposed [17]. e MapReduce framework is also suitable to handle large-scale high-dimensional data similarity joins [18]. For example, spatial join is able to perform data analysis in geospatial applications, which contain massive geographical information in a high-dimensional form [19].…”