Optimizing data partition for scaling out NoSQL cluster

Abstract

Data partition impacts the performance of Not Only SQL (NoSQL) systems significantly. Nowadays, many of the peer-to-peer NoSQL systems use consistent hashing to partition data automatically. These systems use virtual nodes and random data placement methods to divide the consistent hashing ring, which may lead to imbalanced data partition and degrade the overall system performance. The problem is prominent especially for scaling out heterogeneous clusters. Considering the capacity of each node, an imbalance coefficient of data distribution for a cluster is proposed firstly in this paper. Based on the imbalance coefficient, we propose a dynamic programming algorithm to calculate the position of the new coming node in the consistent hashing ring, which expands the consistent hashing ring more evenly without re-shuffling the entire datasets.Simulations and experiments on Cassandra with Yahoo! Cloud Serving Benchmark (YCSB) benchmark show our algorithm is better than the state-of-the-art work. . load balance, and (3) traffic load balance. The last two dimensions can be solved by replication mechanism. Our focus in this paper is the nodes distribution over the ring space.Virtual node is proposed in [10][11][12] to improve scalability and load balanced. Chen et al.[13] propose a hybrid data partitioning strategy for heterogeneous cluster load balance, and divide a node into many virtual nodes according to the node ability for the data storage and processing. However, although each node can store the amount of data, which is proportional to its capacity

An evaluation of relational and NoSQL distributed databases on a low-power cluster

“…Other authors attained significant throughput results for on smaller datasets [ 6 , 10 – 12 , 42 ] or low-power devices [ 23 ]. On the other hand, we had latency results comparable to related papers running Cassandra on virtualized and native high-end platforms [ 43 – 45 ].…”

Big data cloud and the frontier of computer science and technology

“…Conventional radon virtual nodes and manual configuration methods for consistent hashing can significantly lead to imbalanced data partition. Huang et al study the performance degradation problem caused by the imbalanced data partition. They first propose a novel imbalance coefficient of data distribution.…”

“…Important advances have been provided by Wei et al , by Li et al , by Hong et al , by Xu et al , by Yao et al , by Huang et al , by Zhu et al , and by Tang et al . These contributions focus on the following.…”