Function of Content Defined Chunking Algorithms in Incremental Synchronization

Abstract

Data chunking algorithms divide data into several small data chunks in a certain way, thus transforming the operation of data into the one of multiple small data chunks. Data chunking algorithms have been widely used in duplicate data detection, parallel computing and other fields, but it is seldom used in data incremental synchronization. Aiming at the characteristics of incremental data synchronization, this paper proposes a novel data chunking algorithm. By dividing two data that need synchronization into small data chunks, comparing the contents of these small data chunks, different ones are the incremental data that need to be found. The new algorithm determines to set a cut-point based on the number of 1 contained in the binary format of all bytes in an interval. Thus it improves the resistance against the byte shifting problem at the expense of the chunk size stability, which makes it more suitable for the incremental data synchronization. Comparing this algorithm with several known classical or state of art algorithms, experiments show that the incremental data found by this algorithm can be reduced by 32%∼57% compared to the others with same changes between two data. The experimental results based on real-world datasets show that PCI improves the calculation speed of classic Rsync algorithm up to 70%, however, with a drawback of increasing the Transmission compression rate up to 11.8%. INDEX TERMS Data synchronization, chunking algorithm, data backup, increment.

A Design of Parallel Content-Defined Chunking System Using Non-Hashing Algorithms on FPGA

“…If the cutoff condition is not satisfied, slide the window to the next byte and re-perform the comparison of the sum of 1s with the threshold. This process is repeated until all the cut points and chunks of a data stream are found [14]. The algorithm's operation is shown in Fig.…”

“…In the worst case that a byte is added, deleted, or changed near the chunk boundary, that chunk and the subsequent one are affected. One of the PCI drawbacks is the varying size of chunks due to the geometric distribution [14]. Besides, the ability to eliminate low entropy strings is another aspect to consider.…”

“…In the publication [12], Y. Zhang and colleagues gave some important criteria used to evaluate the CDC algorithm and were widely used in later studies [13], [14]. The evaluation criteria include:…”

“…By eliminating the sliding window, AE and RAM reduce the great number of comparisons. In contrast, the lately Parity Check of Interval (PCI) algorithm [14], which examines the number of 1s in a sliding window and compares that value with a preset threshold to determine a cut point, gives very positive results.…”

SimpleSync: A parallel delta synchronization method based on Flink

“…Therefore, when the CDC algorithms are applied to delta synchronization, we need to focus on the byte-shifting ability, which affects the stability of the cut-point and then affects the accuracy of the differential data. To better apply the CDC algorithm to the differential data acquisition, we proposed the PCI algorithm 17 in the previous study, which enhanced the ability to resist byte-shifting through small window pattern matching, The accuracy of differential data acquisition is improved.…”