Research on Load Balancing MapReduce Equivalent Join Based on Intelligent Sampling and Multi Knapsack Algorithm

Abstract

Background: With the rapid development of science, more data are produced in people's life. Therefore, the storage and calculation of big data has become the focus of scientific research. MapReduce performs well in big data processing. However, it is prone to data skew. Which affects the overall efficiency of the data processing cluster. Objective: Aiming at the low efficiency of MapReduce data join, this paper proposes an intelligent data join load balancing algorithm based on dynamic programming. The algorithm introduces data sampling and partition algorithm. Due to the high performance of dynamic programming in data constraint problem, it is used to intelligently solve the data skew problem. Methods: Firstly, the causes of data skew are analyzed and the data partition method is improved. The algorithm introduces a data sampling method. In the task allocation stage, the multidimensional knapsack algorithm is used. Different key values are evenly divided to each computing node through the load cost. Finally, The performance of the improved algorithm is verified by experiments. Results: The experimental results show that compared with the traditional load balancing algorithm and the existing improved algorithm, the new algorithm improves the data processing efficiency, reduces the data skew problem and better solves the problem of data load imbalance. Conclusion: A two table equivalent join load balancing algorithm based on key cost is proposed. The algorithm creatively combines dynamic programming with intelligent data sampling, which greatly improves the efficiency and quality of data processing. The algorithm is worthy of popularization and application.