DAM: A DataOwnership-Aware Multi-layered De-duplication Scheme

Abstract

Beyond the storage savings brought by chunk-level de-duplication in backup and archiving systems, a prominent challenge facing this technology is how to efficiently and effectively identify the duplicate chunks. Most of the chunk fingerprints used to identify individual chunks are stored on disks due to the limited main memory capacity. Checking for chunk fingerprint match on disk for every input chunk is known to be a severe performance bottleneck for the backup process. On the other hand, our intuitions and analyses of real backup data both indicate that duplicate chunks tend to strongly concentrate according to the data ownership. Motivated by this observation and to avoid or alleviate the aforementioned backup performance bottleneck, we propose DAM, a dataownership-aware multi-layered de-duplication scheme that exploits the data chunks' ownership and uses a tri-layered de-duplication approach to narrow the search space for duplicate chunks to reduce the total disk accesses. Our experimental results with real world datasets on DAM show it reduces the disk accesses by an average of 60.8% and shortens the de-duplication time by an average of 46.3%.

Improvement of Deduplication Efficiency by Two‐Layer Deduplication System

“…On the other hand, in two-layer deduplication using a first chunk size of 16 KB, an improvement of the SDR from 0.3 to 0.44 results in an NRC increase by a factor of only 4.5 at most. In this example, the combination (8,2) achieves the maximum SDR, with a smaller NRC than (16,2). As mentioned above, the optimal combination of chunk sizes to minimize the NRC varies with the data properties.…”

“…As mentioned above, the optimal combination of chunk sizes to minimize the NRC varies with the data properties. In addition, as can be seen by comparing (8,2) or (16,2) to (4), the SDR can be improved without increasing the NRC due to two-layer deduplication.…”

“…In addition, the duplication factor producing the maximum NRC decreases. Thus, with a duplication factor of 0.8, the (8,2) and (16,2) systems prove best for dirty lengths of 5 KB and 32 KB, respectively. In addition, the NRC reduction effect of two-layer system increases over a wide range of duplication factors.…”

“…In addition, the NRC reduction effect of two-layer system increases over a wide range of duplication factors. For example, the improvement rate is 41% for (8, 2) at a dirty length of 5 KB, and 71% for (16,2) at 32 KB. This is because at larger dirty lengths, successive chunks are more likely to be eliminated efficiently, while the deduplication performance improves at the first stage so that fewer chunks are eliminated at the second stage.…”

“…The dashed line represents single-layer deduplication. As the chunk size decreases from (16) to (8) to (4) to (2), the NRC grows faster and faster. Here, the SDR is improved from 0.3 to 0.44 as the chunk size is reduced from 16 KB to 8 KB, 4 KB, and 2 KB, and the NRC increases by factors of 3, 5, and 11, respectively.…”