WAN-optimized replication of backup datasets using stream-informed delta compression

Abstract

Replicating data off-site is critical for disaster recovery reasons, but the current approach of transferring tapes is cumbersome and error-prone. Replicating across a wide area network (WAN) is a promising alternative, but fast network connections are expensive or impractical in many remote locations, so improved compression is needed to make WAN replication truly practical. We present a new technique for replicating backup datasets across a WAN that not only eliminates duplicate regions of files (deduplication) but also compresses similar regions of files with delta compression, which is available as a feature of EMC Data Domain systems.Our main contribution is an architecture that adds stream-informed delta compression to already existing deduplication systems and eliminates the need for new, persistent indexes. Unlike techniques based on knowing a file's version or that use a memory cache, our approach achieves delta compression across all data replicated to a server at any time in the past. From a detailed analysis of datasets and hundreds of customers using our product, we achieve an additional 2X compression from delta compression beyond deduplication and local compression, which enables customers to replicate data that would otherwise fail to complete within their backup window.

An Effective Data Reduction in a Twin Cloud Environment using an Authorized De-dup. Technique with DARE Scheme

“…of these approaches for likeness detection needs high overheads of computation and categorisation. Shilane et al projected a stream-informed delta compression (SIDC) approach utilized in a WAN surroundings for reducing similar information transmission and so fast information replication [9]. This approach is superfeature primarily based and enhances the block-level deduplication by solely police work resemblance among nonduplicate blocks within the cache that preserves the backup stream locality.…”

DARE: A Deduplication-Aware Resemblance Detection and Elimination Scheme for Data Reduction with Low Overheads

“…Data deduplication is an efficient data reduction approach that not only reduces storage space [4], [5], [6], [7], [8], [9], [10] by eliminating duplicate data but also mini-mizes the transmission of redundant data in low-bandwidth network environments [11], [12], [13], [14]. In general, a chunk-level data deduplication scheme splits data blocks of a data stream (e.g., backup files, databases, and virtual machine images) into multiple data chunks that are each uniquely identified and duplicate-detected by a secure SHA-1 or MD5 hash signature (also called a fingerprint) [5], [11].…”

“…While data deduplication has been widely deployed in storage systems for space savings, the fingerprint-based dedu-plication approaches have an inherent drawback: they often fail to detect the similar chunks that are largely identical except for a few modified bytes, because their secure hash digest will be totally different even only one byte of a data chunk was changed [4], [5], [12], [15], [16]. It becomes a big challenge when applying data deduplication to storage data-sets and workloads that have frequently modified data, which demands an effective and efficient way to eliminate redun-dancy among frequently modified and thus similar data.…”

“…Delta compression, an efficient approach to removing redundancy among similar data chunks has gained increas-ing attention in storage systems [12], [17], [18], [19], [20]. For example, if chunk A2 is similar to chunk A1 (the base-chunk), the delta compression approach calculates and then only stores the differences (delta) and mapping relation between A2 and A1.…”

“…The state-of-the-art solutions [12], [15], [16], [17] detect similarity for delta compression by computing sev-eral Rabin fingerprints as features and grouping them into superfingerprints, also referred to as super-features (SF) (detailed in Section 3.3). Nevertheless, to index a dataset of 80 TB and assuming an average chunk size of 8 KB and 16 bytes per index entry, for example, about 200 GB worth of super-feature index entries must be generated, which will still be too large to fit in memory [12]. Since the random accesses to ondisk index are much slower than that to RAM, the frequent accesses to on-disk super-features will cause the system throughput to become unacceptably low for the users [6], [12], [21].…”