BloomStore: Bloom-Filter based memory-efficient key-value store for indexing of data deduplication on flash

Lu, Guanlin; Nam, Young Jin; Du, David H. C.

doi:10.1109/msst.2012.6232390

Cited by 60 publications

(52 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This can be solved by organizing the index as a log, appending new entries sequentially, as in ChunkStash [Debnath et al 2010]. In fact, the challenges for key-value stores on FLASH memory are widely researched and are present in a wider range of systems unrelated to deduplication [Anand et al 2010;Debnath et al 2011;Lim et al 2011;Lu et al 2012]. Moreover, both dedupv1 and ChunkStash explore spatial locality as the original DDFS work does.…”

Section: Backup and Archival Storagementioning

confidence: 99%

A Survey and Classification of Storage Deduplication Systems

2014

View full text Add to dashboard Cite

The automatic elimination of duplicate data in a storage system, commonly known as deduplication, is increasingly accepted as an effective technique to reduce storage costs. Thus, it has been applied to different storage types, including archives and backups, primary storage, within solid-state drives, and even to random access memory. Although the general approach to deduplication is shared by all storage types, each poses specific challenges and leads to different trade-offs and solutions. This diversity is often misunderstood, thus underestimating the relevance of new research and development.The first contribution of this article is a classification of deduplication systems according to six criteria that correspond to key design decisions: granularity, locality, timing, indexing, technique, and scope. This classification identifies and describes the different approaches used for each of them. As a second contribution, we describe which combinations of these design decisions have been proposed and found more useful for challenges in each storage type. Finally, outstanding research challenges and unexplored design points are identified and discussed.

show abstract

Section: Backup and Archival Storagementioning

confidence: 99%

A Survey and Classification of Storage Deduplication Systems

2014

View full text Add to dashboard Cite

show abstract

“…After then, the Bloom Filter based index becomes very popular for space-efficient and fast matching, such as deduplication of enterprise backup systems [9][10][11], fast matching in IP routing [12][13][14], element membership query in distributed database [15,16], and many other fast-matching demanded network applications [2,17].…”

Section: Related Workmentioning

confidence: 99%

“…Compared with S q , S b in memory can be neglected. Whereas, Bloomstore [11] only keeps a write buffer related BF in memory, and stores the other BFs in flash to minimize memory overhead (S b is at the same level as S q ).…”

Section: Parameter Initializationsmentioning

confidence: 99%

Par-BF: A Parallel Partitioned Bloom Filter for Dynamic Data Sets

Liu

et al. 2014

2014 International Workshop on Data Intensive Scalable Computing Systems

View full text Add to dashboard Cite

Compared with a hash table, a Bloom Filter (BF) is more space-efficient for supporting fast matching though resulting in a controllable and acceptable false positive probability. The space size of the basic BF is predetermined based on the expected number of elements to be stored. However, we cannot predict the scale of a BF space for dynamic sets. The two existing solutions for supporting dynamic sets, Scalable BF (SBF) and Dynamic BF (DBF), still face some challenges on system performance and memory overhead.This paper presents a new BF for dynamic data sets, called Partitioned BF (Par-BF). Compared with DBF and SBF, the size and the range of the false positive probability can be calculated by a group of formulas to leverage a sweet spot between high-performance and low-overhead. Moreover, Par-BF supports parallel fast matching which can improve the overall throughput. From our trace-driven experimental results, the IOPS of Par-BF outperforms that of DBF and SBF from 6X to 10X, and from 2X to 4X, respectively. Meanwhile, through our proposed garbage collection policy, the memory overhead of Par-BF is less than half of the memory usage of SBF.

show abstract

“…Lu et.al. [23] has proposed a technique called bloomstore, a bloom filter based memory efficient key-value pairs for indexing of data de-duplication on flash. They proposed an efficient KV store on flash with a bloom filter [7] based index structure called bloomstore.…”

Section: Flash Based De-duplication Systemsmentioning

confidence: 99%

A Study on Various Data De-duplication Systems

Vikraman¹,

Abirami²

2014

IJCA

View full text Add to dashboard Cite

Data is the heart of any organization; hence it is necessary to protect it. For doing so, it is the needed to implement a good backup and recovery plan. But the redundant nature of the backup data makes the storage a concern; hence it is necessary to avoid the redundant data present in the backup. Data deduplication is one such solution that discovers and removes the redundancies among the data blocks.

show abstract

BloomStore: Bloom-Filter based memory-efficient key-value store for indexing of data deduplication on flash

Cited by 60 publications

References 13 publications

A Survey and Classification of Storage Deduplication Systems

A Survey and Classification of Storage Deduplication Systems

Par-BF: A Parallel Partitioned Bloom Filter for Dynamic Data Sets

A Study on Various Data De-duplication Systems

Contact Info

Product

Resources

About