Chunk Fragmentation Level: An Effective Indicator for Read Performance Degradation in Deduplication Storage

Abstract

Data deduplication has recently become commonplace in most secondary storage and even in some primary storage for the capacity optimization purpose. Aside from its write performance, read performance of the deduplication storage has been gaining in significance with a wide range of its deployments. In this paper, we emphasize the importance of read performance in reconstituting a data stream from its unique and shared chunks physically dispersed over deduplication storage. We newly introduce a read performance indicator called Chunk Fragmentation Level (CFL). We also validate that the CFL is very effective to indicate read performance of deduplication storage through a developed theoretical performance model and extensive experiments. Finally, we articulate further research issues.While the data growth in our digital universe becomes explosive, it has been observed that the 70% of the worldwide digital data are duplicated. Certainly, data duplication [1] (briefly deduplication) would be extremely usefuly to mitigate demanded storage capacity by eliminating any duplicate data from a large set of data streams. A typical chunkingbased deduplication process divides a given data stream into smaller chunks of variable/fixed lengths and computes a hash value of each chunk with a cryptographic hash function, such as SHA-1 or MD5. Generally, dynamic chunking (variablelength chunks) outperforms static chunking (fixed-length chunks). A hash index table should effectively maintain a large number of hash values with their storage locations. Only if the hash value of a chunk is not found in the hash index table, the deduplicated (unique or unshared) chunk is written into the deduplication storage. Otherwise, the shared chunk will be eliminated from further physical write.Recent deduplication design has mainly focused on its write performance by: (1) improving the duplication detection and chunk existence querying efficiency with efficient chunking, faster hash indexing [2], locality-preserving index caching [1], and efficient bloom filters [3]; (2) compressing the unique chunks and performing (fixed-size) large writes through containers [4], [1] or similar structures [5].However, read performance of deduplication storage does matter. Traditionally, the read performance (mainly throughput) of deduplication storage, such as backup systems, has not gained in its significance because people thinks that the occurrence of reads is much smaller than that of writes. Having said that, the read performance becomes extremely critical in the event of entire system restoration from crash [1]. Higher read performance can shorten a recovery window time, resulting in higher system availability. In addition, since the deduplication storage has a limited capacity, it is periodically necessary to stage the backed up data streams for migration to archive storage, such as virtual tapes. This requires reconstruction of their original data streams, because the archive storage operations are typically stream-based. In fact, this staging frequency is ...

Assuring Demanded Read Performance of Data Deduplication Storage with Backup Datasets

“…The OCF follows the same definition in our previous work [9]. However, the definition of the CCF is significantly different from its previous definition in [9]. That is, the previous CCF did not consider a read cache that is closely related to the read performance.…”

“…The CFL is composed of two parameters: optimal chunk fragmentation (for short, OCF) and cache-aware current chunk fragmentation (for short, CCF). The OCF follows the same definition in our previous work [9]. However, the definition of the CCF is significantly different from its previous definition in [9].…”

“…However, the read performance becomes extremely critical when it comes to restoring the entire system from crash [2], [9]. Higher read performance can significantly save the recovery time while accomplishing higher system availability.…”

Survey on Data Deduplication for Cloud Storage to Reduce Fragmentation

“…Y. Nam et al proposed deduplication process which implanted with some kind of chunk fragmentation optimization. It associates each data stream with its own open container in the memory, so that the unique chunks from different streams can be stored into different chunk containers [7].…”

Survey on Fragmentation for Deduplication in Backup Storage

“…A read performance of the deduplication storage has been gaining great significance. In [6], author introduced called Chunk Fragmentation Level (CFL) as an indicator for read performance.…”