SimpleSync: A parallel delta synchronization method based on Flink

Abstract

Cloud storage service has been in full swing in the industry. Delta synchronization technology as a key technology of cloud storage services has not made a key breakthrough. Almost all the existing researches are based on the synchronization process proposed by the Rsync algorithm and mix some optimization appropriately, but the particularity of cloud storage service is not fully considered. This paper proposes a new incremental synchronization method SimpleSync, which makes use of the characteristic that the server does not actively modify the backup files in the cloud storage service, removes the redundant steps in Rsync, and enables the synchronization between the client and the server only through a single communication. Besides, according to the server-side synchronization request processing logic, this paper puts forward the design idea of parallel processing with the Flink framework, to the best of our knowledge, for the first time. After the server receives the synchronization request, SimpleSync first puts it into Kafka for buffering and then uses Flink to process the synchronization request in parallel. In the experimental part, a large number of experiments are designed to compare SimpleSync with other delta synchronization algorithms. Experimental results show that SimpleSync has obvious advantages in synchronization performance. Meanwhile, experiments show that SimpleSync has correctness.