A General-Purpose Counting Filter

Pandey, Prashant; Bender, Michael A.; Johnson, Rob; Patro, Rob

doi:10.1145/3035918.3035963

Cited by 120 publications

(131 citation statements)

References 25 publications

Supporting

Mentioning

129

Contrasting

Unclassified

Order By: Relevance

“…We compare our GPU SQF and RSQF with a variety of other AMQs: Costa et al's BloomGPU filter [13], Pandey et al's multithreaded counting quotient filter (CQF) [6], Arash Partow's Bloom filter [16], and our own CPU SQF implementation, which uses the serial operations described in Section III. We also modified the BloomGPU filter to create a version ("BloomGPU 1-bit") using only one bit per element of the filter bit array, rather than an entire byte.…”

Section: Resultsmentioning

confidence: 99%

“…Dutta et al [14] implement a parallel version of their streaming quotient filter, an AMQ designed for removing duplicates from streams of data. Pandey et al [6] also implement a multithreaded version of their counting quotient filter, which uses the same structure as their rank-andselect-based quotient filter, described in Section III-B. Their implementation depends on per-thread locking that does not scale to the parallelism of a GPU.…”

Section: Related Workmentioning

confidence: 99%

“…Pandey et al [6] designed the RSQF to improve upon the SQF by increasing lookup performance at high load factors and reducing the number of metadata bits. 1 Their filter stores the remainders using the same slot locations and order as the SQF, but it uses a different metadata scheme for locating items within the filter.…”

Section: B Rank-and-select-based Quotient Filtersmentioning

confidence: 99%

“…To accomplish this, we partition the filter into insert regions, and assign one thread to each region. This is similar to the partitioning for multi-threaded inserts devised by Pandey [6], but because the GPU has many more threads than a CPU, our implementation uses smaller regions than theirs. Initially, regions are one block, and as more items have been inserted, the regions increase in size (and, consequently, decrease in number).…”

Section: B Insertsmentioning

confidence: 99%

“…The quotient filter [5] is designed to maintain locality of data, and beyond supporting all the functionality of the BF, it also supports deletions and the merging of filters. Additionally, the QF can be extended to include counters [6]. We implement two versions of the QF on the GPU, the standard quotient filter (SQF) and the rank-and-select-based quotient filter (RSQF), and compare their relative strengths and weaknesses on this massively parallel architecture.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Quotient Filters: Approximate Membership Queries on the GPU

Geil

Farach-Colton

Owens

2018

2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Abstract-In this paper, we present our GPU implementation of the quotient filter, a compact data structure designed to implement approximate membership queries. The quotient filter is similar to the more well-known Bloom filter; however, in addition to set insertion and membership queries, the quotient filter also supports deletions and merging filters without requiring rehashing of the data set. Furthermore, the quotient filter can be extended to include counters without increasing the memory footprint. This paper describes our GPU implementation of two types of quotient filters: the standard quotient filter and the rankand-select-based quotient filter. We describe the parallelization of all filter operations, including a comparison of the four different methods we devised for parallelizing quotient filter construction. In solving this problem, we found that we needed an operation similar to a parallel scan, but for non-associative operators. One outcome of this work is a variety of methods for computing parallel scan-type operations on a non-associative operator.For membership queries, we achieve a throughput of up to 1.13 billion items/second for the rank-and-select-based quotient filter: a speedup of 3x over the BloomGPU filter. Our fastest filter build method achieves a speedup of 2.1-3.1x over BloomGPU, with a peak throughput of 621 million items/second, and a rate of 516 million items/second for a 70% full filter. However, we find that our filters do not perform incremental updates as fast as the BloomGPU filter. For a batch of 2 million items, we perform incremental inserts at a rate of 81 million items/second -a 2.5x slowdown compared to BloomGPU's throughput of 201 million items/second. The quotient filter's memory footprint is comparable to that of a Bloom filter.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: B Rank-and-select-based Quotient Filtersmentioning

confidence: 99%

Section: B Insertsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Quotient Filters: Approximate Membership Queries on the GPU

Geil

Farach-Colton

Owens

2018

2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

show abstract

Stretching your data with taffy filters

Apple

2022

Softw Pract Exp

View full text Add to dashboard Cite

Popular approximate membership query structures such as Bloom filters and cuckoo filters are widely used in databases, security, and networking. These structures represent sets approximately, and support at least two operations-insert and lookup; lookup always returns true on elements in the structure; it also returns true with some probability 𝜀 on elements not in the structure. These latter elements are called false positives. Compensatory for these false positives, filters can be much smaller than hash tables that represent the same set. However, unlike hash tables, cuckoo filters, and Bloom filters must be initialized with the intended number of maximum entries, and cannot grow larger-inserts beyond this number fail or significantly increase the false positive probability. This article presents designs and implementations of filters than can grow without inserts failing and without meaningfully increasing the false positive probability, even if the filters are created with a small initial size. The resulting code is available on GitHub under a permissive open source license.

show abstract

Fast Wavelet Tree Construction in Practice

Kaneta

2018

String Processing and Information Retrieval

View full text Add to dashboard Cite

A General-Purpose Counting Filter

Cited by 120 publications

References 25 publications

Quotient Filters: Approximate Membership Queries on the GPU

Quotient Filters: Approximate Membership Queries on the GPU

Stretching your data with taffy filters

Fast Wavelet Tree Construction in Practice

Contact Info

Product

Resources

About