Similarity joins for high‐dimensional data using Spark

Abstract

Similarity join on high-dimensional data is a primitive operation. It is used to find all data pairs that with distance no more than from the given data set according to a specific distance measure.As the data set scale and dimension increase, computation cost increases vastly. Hadoop and Spark have become the popular platforms for big-data analysis. Because Spark has native advantages in iterative computations, we adopted it as our platform to perform similarity joins on high-dimensional data sets. In order to resolve problems such as data imbalance, data duplication, and redundant computation of existing works, we have proposed a new algorithm based on Symbolic aggregation and vertical decomposition. We first conduct dimension-reduction using symbolic aggregation method. Then, we applied vertical partition operation on processed data.The join operations are performed on each vertical partition in parallel manner and the proposed new filters are utilized to prune false positives in early stage. Finally, the partial results generated from each partition will be aggregated and verified to get final results. Our proposed algorithm can significantly improve the efficiency of similarity joins on high-dimensional data. In order to verify the efficiency and scalability of our methods, we implemented it using MapReduce and Spark. We compared our methods with existing works on public data sets, and the experimental results showed that the new methods were more efficient and scalable under different running environments. KEYWORDShigh-dimensional data, piecewise aggregation, similarity join, symbolic aggregation, Spark, vertical partition INTRODUCTIONIn this era of big data, data-acquisition occurs ever more quickly, the scale of data is increasing rapidly, and the types of data are complex and diverse. This brings new challenges to data analysis and processing. As a basic operation, the similarity join has been applied widely in many fields, such as friend recommendations, 1 pattern recognition, 2 clustering, 3 image similarity matching, 4 outlier detection, 5 and spatial databases. 6 A similarity join is essentially a pair of comparisons. It has high computationally complexity and is intensive. Data-processing time increases exponentially with data volume. To improve the execution efficiency of algorithms, more efficient methods are needed to reduce unnecessary operations in large-scale data processing. Most traditional algorithms use a spatial index, such as a B+tree, R-tree, or z-order curve, to improve the performance of a similarity join, but traditional algorithms do not apply to large-scale, high-dimensional dalg:1ata sets. The current MapReduce 7 framework based on Hadoop has emerged as the primary choice for big-data processing. MapReduce is a programming model that can easily develop scalable parallel applications for large-scale data. For computationally intensive operations such as a similarity join, scholars recently have proposed parallel knn-join algorithms using MapReduce, such as H-BNLJ, H-BRJ, 8 and PGBJ...

Projection Based Large Scale High-Dimensional Data Similarity Join Using MapReduce Framework

“…Compared with Hadoop based solutions, Chen et al [32] has faster computing speed and better scalability. Rong et al [33] proposed a new similarity join algorithm called symbolic aggregation and vertical decomposition(SAVD) using Spark.…”

Parallel time series join using spark

“…Several works have explored the efficient way to perform set similarity joins. The parallel theta join using MapReduce to join two data sets like in relational databases is explored by Okcan and Riedewald and by Zhang et al Similarity joins on high dimensional data using Spark are studied exploiting data representation and vertical partition techniques . Our approach, however, aims to find correlated segments from two time series.…”

“…The parallel theta join using MapReduce to join two data sets like in relational databases is explored by Okcan and Riedewald 26 and by Zhang et al 26,27 Similarity joins on high dimensional data using Spark are studied exploiting data representation and vertical partition techniques. 28 Our approach, however, aims to find correlated segments from two time series. Bendre and Manthalkar 29 introduced predictive analytics approaches The FFT algorithm is widely used in many industries for data transformation.…”

Performance enhancement for iterative data computing with in‐memory concurrent processing

“…They adopted dimension reduction with symbolic aggregation method and vertical partition operation to implement join operations for high‐dimensional data. Spark caches the intermediate results to reduce data input or output so that the efficiency of iteration is improved . However, how to select the right RDDs to cache the partitions in limited memory is an open issue.…”

“…Spark caches the intermediate results to reduce data input or output so that the efficiency of iteration is improved. 29,33 However, how to select the right RDDs to cache the partitions in limited memory is an open issue. The shuffle operations are still required for the iterative applications.…”