ADMAD: Application-Driven Metadata Aware De-duplication Archival Storage System

Abstract

There is a huge amount of duplicated or redundant data in current storage systems. So Data Deduplication, which uses lossless data compression schemes to minimize the duplicated data at the interfile level, has been receiving broad attention in recent years. But there are still research challenges in current approaches and storage systems, such as: how to chunking the files more efficiently and better leverage potential similarity and identity among dedicated applications; how to store the chunks effectively and reliably into secondary storage devices. In this paper, we propose ADMAD: an Application-Driven Metadata Aware De-duplication Archival Storage System, which makes use of certain meta-data information of different levels in the I/O path to direct the file partitioning into more Meaningful data Chunks (MC) to maximally reduce the inter-file level duplications. However, the chunks may be with different lengths and variable sizes, storing them into storage devices may result in a lot of fragments and involve a high percentage of random disk accesses, which is very inefficient. Therefore, in ADMAD, chunks are further packaged into fixed sized Objects as the storage units to speed up the I/O performance as well as to ease the data management.Preliminary experiments have demonstrated that the proposed system can further reduce the required storage space when compared with current methods (from 20% to near 50% according to several datasets), and largely improves the writing performance (about 50%-70% in average).

Effective content-based video caching with cache-friendly encoding and media-aware chunking

“…The are a few works [25,27] that attempt to adjust the chunking algorithm to the content type for improved redundancy exposure. [27] employs specialized chunking for flash videos (FLV) and MP3 files and improves the data compression rate by up to 27% compared with that of Rabin's fingerprinting.…”

“…[27] employs specialized chunking for flash videos (FLV) and MP3 files and improves the data compression rate by up to 27% compared with that of Rabin's fingerprinting. [25] improves the NRE performance by adjusting the chunk size to the characteristics of the content.…”

DAM: A DataOwnership-Aware Multi-layered De-duplication Scheme

“…LBFS [25] is the first file system that breaks files into variable sized chunks using content defined chunking algorithm instead of fixed chunk size, which overcomes the shifting problem and can identify much more redundant data. To achieve a higher compression ratio, other chunking methods, such as fingerdiff [26], TTTD [27], ADMAD [28], provide more flexibility on the variability of chunk sizes. Fuzzy block matching [29] combines shingling and error-correcting to reduce file transmission costs.…”

Multi-level comparison of data deduplication in a backup scenario

“…Even if this chunking method is not restricted to Rabin Fingerprints, CDC and Rabin Fingerprints are sometimes used interchangeably [14].…”

“…by splitting email archives exactly at mail boundaries. The ADMAD deduplication system incorporates application-specific chunking methods, which are automatically selected based on file metadata [14]. It achieves higher compression ratios for MP3 files (43.9%), Flash video files (27%), HTML files (48%), and Email (14.3%) in comparison to Content-defined Chunking using Rabin Fingerprints.…”