Block locality caching for data deduplication

Abstract

Data deduplication systems discover and remove redundancies between data blocks by splitting the data stream into chunks and comparing a hash of each chunk with all previously stored hashes. Storing the corresponding chunk index on hard disks immediately limits the achievable throughput, as these devices are unable to support the high number of random IOs induced by this index. Several approaches to overcome this chunk lookup disk bottleneck have been proposed. Often, the approaches try to capture the locality information of a backup run and use this in the next backup run to predict future chunk requests. However, often this locality is only captured by a surrogate, e.g., the order of the chunks in containers. [37]. Furthermore, some approaches degenerate slowly when the systems operate over months and years because the locality information becomes outdated.We propose a novel approach, called Block Locality Cache (BLC), that captures the previous backup run significantly better than existing approaches and also always uses up-todate locality information and which is, therefore, less prone to aging.We evaluate the approach using a trace-based simulation of multiple real-world backup datasets. The simulation compares the Block Locality Cache with the approach of Zhu et al. [37] and provides a detailed analysis of the behavior and IO pattern. Furthermore, a prototype implementation is used to validate the simulation.

Brief Review of De Duplication over Cloud for Optimizing Storage

“…Fatema Rashid et al [2] proposed a framework that enables the file to be divided first into smaller units and then those blocks gets encrypted using an effective encryption technique however their framework wasn't tested in real cloud environment, AvaniWildaniet. al [3] demonstrated the effectiveness of their approach employing a straightforward neighborhood grouping that needs solely timestamp and block number, that makes it better to be used with various kind of storage systems without the need to modify host file systems.Dirk Meister, et.al [4] projected a way within which earlier backup information was used to predict the future backup. This technique increase the lookup performance.…”

HyCache+: Towards Scalable High-Performance Caching Middleware for Parallel File Systems

“…Block Locality Caching (BLC) [24] captures the backup and always uses the latest locality information to achieve better performance for data deduplication systems. The File Access corRelation Mining and Evaluation Reference model (FARMER) [42] optimizes the large scale file system by correlating access patterns and semantic attributes.…”

eWave

“…Currently, the eWave cache implements a simple LRU eviction policy. Anyway, other policies can be introduced to improve superchunk caching effectiveness [18,19].…”