Improving Restore Performance for In-Line Backup System Combining Deduplication and Delta Compression

Zhang, Yucheng; Yuan, Ye; Feng, Dan; Wang, Chunzhi; Wu, Xinyun; Yan, Lingyu; Pan, Deng; Wang, Shuanghong

doi:10.1109/tpds.2020.2991030

Cited by 17 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, as the number of backups increases, chunks in a backup are scattered in different containers, also known as chunk fragmentation. 7,8,[35][36][37][38] This means that locality preserved in containers degrades with the increase of the number of the backup version, which leads to a decreased number of detected similar chunks as well as a loss of compression. The compression loss can be up to 27% when the backup version number increases to 49, and this loss would keep increasing as the number increases.…”

Section: Problem Statementmentioning

confidence: 99%

A cost‐efficient resemblance detection scheme for post‐deduplication delta compression in backup systems

Wang

Yan

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

Delta compression, which is efficient in removing repeated string among similar chunks, can be used as a complement to data deduplication in backup storage for extra space savings. The process of detecting similar candidates to use as the base for delta compression is called resemblance detection. Several indexes are required for resemblance detection. Maintaining them in RAM would limit the system scalability and increase system cost. Storing them on the disk suffers from low throughput due to poor random I/O performance of the disk. In this article, we present the history-aware resemblance detection (HARD), a cost-efficient resemblance detection approach that captures most of the similar chunks with a limited memory footprint. HARD is based on the observation that, for chunks in a backup, most of their similar chunks can be found in the most recent backups. HARD thus only indexes super-features in the most recent backups for resemblance detection to reduce the memory footprint of resemblance indexes while captures most of the potential similar chunks for delta compression.Experimental results based on three real-world datasets show that HARD achieves higher compression than the state-of-the-art approach.

show abstract

Section: Problem Statementmentioning

confidence: 99%

A cost‐efficient resemblance detection scheme for post‐deduplication delta compression in backup systems

Wang

Yan

et al. 2021

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

show abstract

“…In addition to optimizing the collection of differential data, to reduce the cost during the synchronization, scholars have proposed delta compression and delayed synchronization. Zhang, Yucheng et al 18 proposed that some chunks are the basic ones, and the others are represented by the incremental encoding of the basic chunks, which makes full use of the similarity between chunks, thus reducing the overall transmission data. To avoid frequent network communication caused by the real-time update, Saiz-Laudo R et al proposed Rate-Based Sync Deferment (RSD).…”

Section: F I G U R E 1 Communication Flow Of Rsync Algorithmmentioning

confidence: 99%

SimpleSync: A parallel delta synchronization method based on Flink

Zhang

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Cloud storage service has been in full swing in the industry. Delta synchronization technology as a key technology of cloud storage services has not made a key breakthrough. Almost all the existing researches are based on the synchronization process proposed by the Rsync algorithm and mix some optimization appropriately, but the particularity of cloud storage service is not fully considered. This paper proposes a new incremental synchronization method SimpleSync, which makes use of the characteristic that the server does not actively modify the backup files in the cloud storage service, removes the redundant steps in Rsync, and enables the synchronization between the client and the server only through a single communication. Besides, according to the server-side synchronization request processing logic, this paper puts forward the design idea of parallel processing with the Flink framework, to the best of our knowledge, for the first time. After the server receives the synchronization request, SimpleSync first puts it into Kafka for buffering and then uses Flink to process the synchronization request in parallel. In the experimental part, a large number of experiments are designed to compare SimpleSync with other delta synchronization algorithms. Experimental results show that SimpleSync has obvious advantages in synchronization performance. Meanwhile, experiments show that SimpleSync has correctness.

show abstract

“…Hence delta compression has attracted a lot of attention in data storage and network transmission as an excellent complementary technique after data deduplication. Delta compression can not only further reduce storage space after data deduplication [12,13], but it is also wellsuitable for mobile device storage [14] and remote file transfer scenarios [15][16][17].…”

Section: Introductionmentioning

confidence: 99%

Chunk2vec: A novel resemblance detection scheme based on Sentence‐BERT for post‐deduplication delta compression in network transmission

Wang,

et al. 2024

IET Communications

Self Cite

View full text Add to dashboard Cite

Delta compression, as a complementary technique for data deduplication, has gained widespread attention in network storage systems. It can eliminate redundant data between non‐duplicate but similar chunks that cannot be identified by data deduplication. The network transmission overhead between servers and clients can be greatly reduced by using data deduplication and delta compression techniques. Resemblance detection is a technique that identifies similar chunks for post‐deduplication delta compression in network storage systems. The existing resemblance detection approaches fail to detect similar chunks with arbitrary similarity by setting a similarity threshold, which can be suboptimal. In this paper, the authors propose Chunk2vec, a resemblance detection scheme for delta compression that utilizes deep learning techniques and Approximate Nearest Neighbour Search technique to detect similar chunks with any given similarity range. Chunk2vec uses a deep neural network, Sentence‐BERT, to extract an approximate feature vector for each chunk while preserving its similarity with other chunks. The experimental results on five real‐world datasets indicate that Chunk2vec improves the accuracy of resemblance detection for delta compression and achieves higher compression ratio than the state‐of‐the‐art resemblance detection technique.

show abstract

Improving Restore Performance for In-Line Backup System Combining Deduplication and Delta Compression

Cited by 17 publications

References 10 publications

A cost‐efficient resemblance detection scheme for post‐deduplication delta compression in backup systems

A cost‐efficient resemblance detection scheme for post‐deduplication delta compression in backup systems

SimpleSync: A parallel delta synchronization method based on Flink

Chunk2vec: A novel resemblance detection scheme based on Sentence‐BERT for post‐deduplication delta compression in network transmission

Contact Info

Product

Resources

About