kBF: A Bloom Filter for key-value storage with an application on approximate state machines

Abstract

Abstract-Key-value (k-v) storage has been used as a crucial component for many network applications, such as social networks, online retailing, and cloud computing. Such storage usually provides support for operations on key-value pairs, and can be stored in memory to speed up responses to queries. So far, existing methods have been deterministic: they will faithfully return previously inserted key-value pairs. Providing such accuracy, however, comes at the cost of memory and CPU time. In contrast, in this paper, we present an approximate k-v storage that is more compact than existing methods. The tradeoff is that it may, theoretically, return a null value for a valid key with a low probability, or return a valid value for a key that was never inserted. Its design is based on the probabilistic data structure called the "Bloom Filter", which was originally developed to test element membership in sets. In this paper, we extend the bloom filter concept to support key-value operations, and demonstrate that it still retains the compact nature of the original bloom filter. We call the resulting design as the kBF (key-value bloom filter), and systematically analyze its performance advantages and design tradeoffs. Finally, we apply the kBF to a practical problem of implementing a state machine in network intrusion detection to demonstrate how the kBF can be used as a building block for more complicated software infrastructures.

Optimizing Bloom Filter: Challenges, Solutions, and Comparisons

“…kBF. Instead of saving the KV pairs directly, kBF [19] converts the values into fixed-length string bits with respect to given constraints. kBF consists of cells, each of which has two components, i.e., the counter to track the number of encodings into the cell, and the encoding field records either an original encoding, or the XOR results of the encodings that are mapped to this cell.…”

“…Multi-class BF [3] Optihash [67] FPF-MBF [43] Retouched BF [75] MPCBF [77] Ternary BF [134] Generalized BF [76] Cross-checking BF [72] Complement BF [73] Yes-no BF [74] VI-CBF [83] FP-CBF [84] Selected Hash [78] [79] Space-code BF [88] Adaptive BF [114] Spectral BF [112] Loglog BF [113] Dynamic BF [89] Variable length signatures [117] Scalable BF [118] DBA [90] Par-BF [91] Weighted BF [119] Popularity conscious BF [120] BloomStore [17] kBF [19] IBLT [21] k-mer BF [131] Spatial BF [17] CBF [36] Deletable BF [133] Distance-sensitive BF [143] Locality-sensitive BF [144] Less hash [85] DLB-BF [96] Combinatorial BF [97] Bloom-1 [92] OMASS [93] Compressed BF [99] Compacted BF [100] Forest-structured BF [106] Stable BF [135] Temporal CBF [136] d-left CBF [101] Memo. Opti.…”

“…EABF [110] L-CBF [111] MLBF [142] Shifting BF [115] ICBF [20] Bloomier filter [95] Parallel BF [94] IBF [145] One hash BF [86] Ultra-fast BF [98] Double Buffering [137] A 2 Buffering [138] Forgetful BF [139] Performance Generalization EGH filter [80] Loglog BF [113] kBF [19] Fig. 13.…”

kBF: Towards Approximate and Bloom Filter based Key-Value Storage for Cloud Computing Systems

“…In our previous work [12], we have investigated the feasibility of this approach with a prototype implementation. Specifically, we proposed a method to encode the values into a special type of binary encodings that can fit into the cells of bloom filters.…”

“…• We extend our method in [12], and present a comprehensive set of programming APIs for cloud computing users to use kBFs as if they were objects in their software implementations. These new APIs include CREATE, JOIN, and COMPRESS, which support operations on multiple kBF objects.…”