A Survey and Classification of Storage Deduplication Systems

Abstract

The automatic elimination of duplicate data in a storage system, commonly known as deduplication, is increasingly accepted as an effective technique to reduce storage costs. Thus, it has been applied to different storage types, including archives and backups, primary storage, within solid-state drives, and even to random access memory. Although the general approach to deduplication is shared by all storage types, each poses specific challenges and leads to different trade-offs and solutions. This diversity is often misunderstood, thus underestimating the relevance of new research and development.The first contribution of this article is a classification of deduplication systems according to six criteria that correspond to key design decisions: granularity, locality, timing, indexing, technique, and scope. This classification identifies and describes the different approaches used for each of them. As a second contribution, we describe which combinations of these design decisions have been proposed and found more useful for challenges in each storage type. Finally, outstanding research challenges and unexplored design points are identified and discussed.

High-speed data deduplication using parallelized cuckoo hashing

“…In contrast, a distributed deduplication includes multiple nodes to perform deduplication, thus parallelizing the deduplication process. Some distributed deduplication systems have separate indexing structures [22]. Some deduplication systems have a centralized index structure.…”

Survey on Data Deduplication for Cloud Storage to Reduce Fragmentation

“…This policy allows separate file sets for applications with different performance norms, as some may not allow the performance consequence introduced by deduplication. And performing effective lookup and update operations [9].…”

“…On the other side, offline deduplication may introduce extra reads from the storage, it requires more storage space, and increases concurrency issues, and increases the complexity of the deduplication process. These problems driven to the development [9]. …”

Deriving and comparing deduplication techniques using a model-based classification

“…Paulo and Pereira present a broad survey of different deduplication systems [20]. The authors set the systems in relation to each other and show common properties.…”