A Utility-Based Distributed Pattern Mining Algorithm With Reduced Shuffle Overhead

Kumar, Sunil; Mohbey, Krishna Kumar

doi:10.1109/tpds.2022.3221210

Cited by 3 publications

(2 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Te proposed algorithm reduces the number of transactions and the time spent processing the dataset. Castro et al [33] compared alternative Apriori [35], and it succeeded in reducing the communication cost during the shufe process by allocating the tasks across cluster nodes in a fair and efective way using search space division strategy. It showed better performance in running time, memory usage, and scalability.…”

Section: Related Workmentioning

confidence: 99%

ASCF: Optimization of the Apriori Algorithm Using Spark-Based Cuckoo Filter Structure

Alrahwan,

Farouk

2024

International Journal of Intelligent Systems

View full text Add to dashboard Cite

Data mining is the process used for extracting hidden patterns from large databases using a variety of techniques. For example, in supermarkets, we can discover the items that are often purchased together and that are hidden within the data. This helps make better decisions which improve the business outcomes. One of the techniques that are used to discover frequent patterns in large databases is frequent itemset mining (FIM) that is a part of association rule mining (ARM). There are different algorithms for mining frequent itemsets. One of the most common algorithms for this purpose is the Apriori algorithm that deduces association rules between different objects which describe how these objects are related together. It can be used in different application areas like market basket analysis, student’s courses selection process in the E-learning platforms, stock management, and medical applications. Nowadays, there is a great explosion of data that will increase the computational time in the Apriori algorithm. Therefore, there is a necessity to run the data-intensive algorithms in a parallel-distributed environment to achieve a convenient performance. In this paper, optimization of the Apriori algorithm using the Spark-based cuckoo filter structure (ASCF) is introduced. ASCF succeeds in removing the candidate generation step from the Apriori algorithm to reduce computational complexity and avoid costly comparisons. It uses the cuckoo filter structure to prune the transactions by reducing the number of items in each transaction. The proposed algorithm is implemented on the Spark in-memory processing distributed environment to reduce processing time. ASCF offers a great improvement in performance over the other candidate algorithms based on Apriori, where it achieves a time of only 5.8% of the state-of-the-art approach on the retail dataset with a minimum support of 0.75%.

show abstract

Section: Related Workmentioning

confidence: 99%

ASCF: Optimization of the Apriori Algorithm Using Spark-Based Cuckoo Filter Structure

Alrahwan,

Farouk

2024

International Journal of Intelligent Systems

View full text Add to dashboard Cite

show abstract

“…Li et al [12] studied the data skew in the Shuffle stage and proposed a Shuffle phase dynamic balance partitioning method based on reservoir sampling to sample and preprocess the intermediate data, predict the overall data skew and provide the overall partitioning strategy for application implementation, thus reducing the impact of data skew on the Spark performance. Kumar et al [13] studied the search space partitioning strategy of data parallelism. Based on the communication cost-effectiveness pattern mining algorithm, tasks can be allocated fairly and effectively among cluster nodes to reduce the communication cost generated during Shuffle.…”

Section: Related Workmentioning

confidence: 99%

Optimization of the Join between Large Tables in the Spark Distributed Framework

Wang

2023

Applied Sciences

View full text Add to dashboard Cite

The Join task between Spark large tables takes a long time to run and produces a lot of disk I/O, network I/O and disk occupation in the Shuffle process. This paper proposes a lightweight distributed data filtering model that combines broadcast variables and accumulators using RoaringBitmap. When the data in the two tables are not exactly matched, the dimension table Key is collected through the accumulator, compressed by RoaringBitmap and distributed to each node using broadcast variables. The distributed fact table data can be pre-filtered on the local server, which effectively reduces the data transmission and disk reading and writing in the Shuffle phase. Experimental results show that this optimization method can reduce disk usage, shorten the running time and reduce network I/O and disk I/O for Spark Join tasks in the case of massive data, and the effect is more obvious when the two tables have a higher incomplete matching degree or a fixed matching degree but a larger amount of data. This optimization scheme has the advantages of being easy to use, being easy to maintain and having an obvious effect, and it can be applied to many development scenarios.

show abstract

Trust factor-based analysis of user behavior using sequential pattern mining for detecting intrusive transactions in databases

Singh

Jindal

2023

J Supercomput

View full text Add to dashboard Cite

A Utility-Based Distributed Pattern Mining Algorithm With Reduced Shuffle Overhead

Cited by 3 publications

References 54 publications

ASCF: Optimization of the Apriori Algorithm Using Spark-Based Cuckoo Filter Structure

ASCF: Optimization of the Apriori Algorithm Using Spark-Based Cuckoo Filter Structure

Optimization of the Join between Large Tables in the Spark Distributed Framework

Trust factor-based analysis of user behavior using sequential pattern mining for detecting intrusive transactions in databases

Contact Info

Product

Resources

About