Cardinality estimation and dynamic length adaptation for Bloom filters

Abstract

Bloom filters are extensively used in distributed applications, especially in distributed databases and distributed information systems, to reduce network requirements and to increase performance. In this work, we propose two novel Bloom filter features that are important for distributed databases and information systems. First, we present a new approach to encode a Bloom filter such that its length can be adapted to the cardinality of the set it represents, with negligible overhead with respect to computation and false positive probability. The proposed encoding allows for significant network savings in distributed databases, as it enables the participating nodes to optimize the length of each Bloom filter before sending it over the network, for example, when executing Bloom joins. Second, we show how to estimate the number of distinct elements in a Bloom filter, for situations where the represented set is not materialized. These situations frequently arise in distributed databases, where estimating the cardinality of the represented sets is necessary for constructing an efficient query plan. The estimation is highly accurate and comes with tight probabilistic bounds. For both features we provide a thorough probabilistic analysis and extensive experimental evaluation which confirm the effectiveness of our approaches.Note: This is a preprint. The final version is available at http://www.springerlink. com/

When private set intersection meets big data

“…The resulting Bloom filter has no false negative, which means the query result of any element y ∈ S 1 ∩ S 2 against BF S 1 ∩S 2 is always true. The false positive probability of the resulting Bloom filter is no higher than either of the constituent Bloom filter [38]. Note that due to collisions, it is possible that the jth bit is set in BF S 1 by an element in S 1 − S 1 ∩ S 2 and jth bit is set in BF S 2 by an element in S 2 − S 1 ∩ S 2 .…”

BSI: Bloom Filter-Based Semantic Indexing for Unstructured P2P Networksh

“…These bit arrays are similar to those employed in traditional Bloom filters and is supported by a sufficiently large body of research work [14], [16], [17] that allows us to estimate number of documents reachable for a multi-concept query solely based on these bit arrays. Similar to level 1, level 2(TSBF 2,P ) also contains multiple bit arrays each representing different multi-concept queries that whose concepts have C as the least common ancestor in the ontology hierarchy for which P has at least one qualified document in its local document collection (TSBF 2,P (C)).…”

“…Research community has proposed many works to estimate the cardinality(i.e. number of elements) of an original set solely based on its Bloom filter bit array [14], [16], [17]. For our work we used the work presented by authors of [16].…”

“…Given a set S and its Bloom Filter BFS with m bits out of which t are true bits, and k hash functions [16] defines the cardinality of a set is as follows:…”

Approximate Two-Party Privacy-Preserving String Matching with Linear Complexity

“…A single Bloom filter is used for all grams generated by a single string S. Papapetrou et al [22] conclude, that the optimal number of hash functions to do cardinality estimation using Bloom filters is 1. Based on this we fix k = 1 and only use a single hash function to build and query Bloom filters throughout the rest of the paper.…”