Quotient Filters: Approximate Membership Queries on the GPU

Geil, Afton; Farach-Colton, Martı́n; Owens, John D.

doi:10.1109/ipdps.2018.00055

Cited by 9 publications

(5 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The QF uses a quotient as a bucket index for storing the remainder. Mapping an element to a slot may lead to a collision when two different fingerprints are mapped to the same quotient (Geil, Farach-Colton & Owens, 2018).…”

Section: The Quotient Filtermentioning

confidence: 99%

An efficient filter with low memory usage for multimedia data of industrial Internet of Things

Goudarzi

Rahmani

2021

PeerJ Computer Science

View full text Add to dashboard Cite

One of the essential concerns of Internet of Things (IoT) is in industrial systems or data architecture to support the evolutions in transportation and logistics. Considering the Industrial IoT (IIoT) openness, the need for accessibility, availability, and searching of data has rapidly increased. The primary purpose of this research is to propose an Efficient Two-Dimensional Filter (ETDF) to store multimedia data of IIoT applications in a specific format to achieve faster response and dynamic updating. This filter consists of a two-dimensional array and a hash function integrated into a cuckoo filter for efficient use of memory. This study evaluates the scalability of the filter by increasing the number of requests from 10,000 to 100,000. To assess the performance of the proposed filter, we measure the parameters of access time and lookup message latency. The results show that the proposed filter improves the access time by 12%, compared to a Fast Two-Dimensional Filter (FTDF). Moreover, it improves memory usage by 20% compared to FTDF. Experiments indicate a better access time of the proposed filter compared to other filters (i.e., Bloom, quotient, cuckoo, and FTD filters). Insertion and deletion times are essential parameters in comparing filters, so they are also analyzed.

show abstract

Section: The Quotient Filtermentioning

confidence: 99%

An efficient filter with low memory usage for multimedia data of industrial Internet of Things

Goudarzi

Rahmani

2021

PeerJ Computer Science

View full text Add to dashboard Cite

show abstract

“…Whereas the Bloom filter supports only a limited set of operations (no deletions) and no resizing, modern filters have shown how to adapt spaceefficient hash tables in order to construct practical space-efficient filters that support a richer set of operations. Quotient filters and variants (e.g., counting and vector) [15,15,46,55,113,114] are built on the idea of storing small fingerprints via ordered linear probing [4]; and cuckoo [48] and Morton filters [22] are based on the idea of storing these fingerprints via cuckoo hashing [109,110]. There has also been an effort to push forward the theoretical frontiers of what guarantees a filter can offer (see, e.g., Pagh's single-hash filter [105] as well as more recent results [14,82]).…”

Section: Related Workmentioning

confidence: 99%

“…The fact that many filters are implemented on top of hash-table designs means that improvements to hash table performance directly results in improvements to filter performance. For example, our techniques for improving linear probing can immediately be applied to linear-probing based filters [15,46,55,113,114].…”

Section: Related Workmentioning

confidence: 99%

Linear Probing Revisited: Tombstones Mark the Death of Primary Clustering

Bender,

Kuszmaul,

Kuszmaul

2021

Preprint

View full text Add to dashboard Cite

First introduced in 1954, the linear-probing hash table is among the oldest data structures in computer science, and thanks to its unrivaled data locality, linear probing continues to be one of the fastest hash tables in practice. It is widely believed and taught, however, that linear probing should never be used at high load factors; this is because of an effect known as primary clustering which causes insertions at a load factor of 1 − 1/x to take expected time Θ(x 2 ) (rather than the intuitive running time of Θ(x)). The dangers of primary clustering, first discovered by Knuth in 1963, have now been taught to generations of computer scientists, and have influenced the design of some of the most widely used hash tables in production.We show that primary clustering is not the foregone conclusion that it is reputed to be. We demonstrate that seemingly small design decisions in how deletions are implemented have dramatic effects on the asymptotic performance of insertions: if these design decisions are made correctly, then even if a hash table operates continuously at a load factor of 1 − Θ(1/x), the expected amortized cost per insertion/deletion is Õ(x). This is because the tombstones left behind by deletions can actually cause an anti-clustering effect that combats primary clustering. Interestingly, these design decisions, despite their remarkable effects, have historically been viewed as simply implementation-level engineering choices.We also present a new variant of linear probing (which we call graveyard hashing) that completely eliminates primary clustering on any sequence of operations: if, when an operation is performed, the current load factor is 1 − 1/x for some x, then the expected cost of the operation is O(x). Thus we can achieve the data locality of traditional linear probing without any of the disadvantages of primary clustering. One corollary is that, in the external-memory model with a data blocks of size B, graveyard hashing offers the following remarkably strong guarantee: at any load factor 1 − 1/x satisfying x = o(B), graveyard hashing achieves 1 + o(1) expected block transfers per operation. In contrast, past external-memory hash tables have only been able to offer a 1 + o(1) guarantee when the block size B is at least Ω(x 2 ).Our results come with actionable lessons for both theoreticians and practitioners, in particular, that welldesigned use of tombstones can completely change the asymptotic landscape of how the linear probing behaves (and even in workloads without deletions).

show abstract

“…They also mention the possibility for concurrent access using an external array of locks (see Section 5 for results). Recently, there was also a GPU-based implementation of quotient filters [9]. This is another indicator that there is a lot of interest in concurrent AMQs even in these highly parallel scenarios.…”

Section: Related Workmentioning

confidence: 99%

Concurrent Expandable AMQs on the Basis of Quotient Filters

Maier¹,

Sanders²,

Williger³

2019

Preprint

View full text Add to dashboard Cite

A quotient filter is a cache efficient Approximate Membership Query (AMQ) data structure. Depending on the fill degree of the filter most insertions and queries only need to access one or two consecutive cache lines. This makes quotient filters very fast compared to the more commonly used Bloom filters that incur multiple cache misses depending on the false positive rate. However, concurrent Bloom filters are easy to implement and can be implemented lock-free while concurrent quotient filters are not as simple. Usually concurrent quotient filters work by using an external array of locks -each protecting a region of the table. Accessing this array incurs one additional cache miss per operation. We propose a new locking scheme that has no memory overhead. Using this new locking scheme we achieve 1.8 times higher speedups than with the common external locking scheme.Another advantage of quotient filters over Bloom filters is that a quotient filter can change its size when it is becoming full. We implement this growing technique for our concurrent quotient filters and adapt it in a way that allows unbounded growing while keeping a bounded false positive rate. We call the resulting data structure a fully expandable quotient filter. Its design is similar to scalable Bloom filters, but we exploit some concepts inherent to quotient filters to improve the space efficiency and the query speed.Additionally, we propose several quotient filter variants that are aimed to reduce the number of status bits (2-status-bit variant) or to simplify concurrent implementations (linear probing quotient filter). The linear probing quotient filter even leads to a lock-free concurrent filter implementation. This is especially interesting, since we show that any lockfree implementation of another common quotient filter variant would incur significant overheads in the form of additional data fields or multiple passes over the accessed data.

show abstract

Quotient Filters: Approximate Membership Queries on the GPU

Cited by 9 publications

References 12 publications

An efficient filter with low memory usage for multimedia data of industrial Internet of Things

An efficient filter with low memory usage for multimedia data of industrial Internet of Things

Linear Probing Revisited: Tombstones Mark the Death of Primary Clustering

Concurrent Expandable AMQs on the Basis of Quotient Filters

Contact Info

Product

Resources

About