CAMP: A New Bitmap Index for Data Retrieval in Traffic Archival

Wu, Yinjun; Chen, Zhen; Cao, Junwei; Li, Haoxun; Li, Chenxing; Wang, Yijie; Zheng, Wenxun; Chang, Jian; Zhou, Jing; Hu, Ziwei; Guo, Jinghong

doi:10.1109/lcomm.2016.2551719

Cited by 4 publications

(2 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Although there are alternatives, the most popular bitmap compression techniques are based on the word‐aligned run‐length encoding compression model inherited from Oracle (byte‐aligned bitmap compression): WAH, Concise, EWAH (used by the Git version control system), COMPAX, VLC, and VAL‐WAH, among others. () On a W ‐bit system, the r bits of the bitset are partitioned into sequences of

W^{'}

consecutive bits, where

W^{'} \approx W

depends on the technique used; for EWAH,

W^{'} = W

; for WAH and many others,

W^{'} = W - 1

. When such a sequence of

W^{'}

consecutive bits contains only 1s or only 0s, we call it a fill word; otherwise, we call it a dirty word.…”

Section: Integer‐set Data Structuresmentioning

confidence: 99%

“…So we use compressed bitmaps. Though there are alternatives [25], the most popular bitmap compression techniques are based on the word-aligned RLE compression model inherited from Oracle (BBC [26]): WAH [27], Concise [28], EWAH [29] (used by the Git [30] version control system), COMPAX [31], VLC [32], VAL-WAH [33], among others [34,35,36]. On a W -bit system, the r bits of the bitset are partitioned into sequences of W consecutive bits, where W ≈ W depends on the technique used; for EWAH, W = W ; for WAH and many others, W = W − 1.…”

Section: Compressed Bitsetsmentioning

confidence: 99%

See 1 more Smart Citation

Roaring bitmaps: Implementation of an optimized software library

Lemire

Kaser

Kurz³

et al. 2018

Softw Pract Exp

View full text Add to dashboard Cite

Compressed bitmap indexes are used in systems such as Git or Oracle to accelerate queries. They represent sets and often support operations such as unions, intersections, differences, and symmetric differences. Several important systems such as Elasticsearch, Apache Spark, Netflix's Atlas, LinkedIn's Pivot, Metamarkets' Druid, Pilosa, Apache Hive, Apache Tez, Microsoft Visual Studio Team Services, and Apache Kylin rely on a specific type of compressed bitmap index called Roaring. We present an optimized software library written in C implementing Roaring bitmaps: CRoaring. It benefits from several algorithms designed for the single-instruction-multiple-data instructions available on commodity processors. In particular, we present vectorized algorithms to compute the intersection, union, difference, and symmetric difference between arrays. We benchmark the library against a wide range of competitive alternatives, identifying weaknesses and strengths in our software. Our work is available under a liberal open-source license.• We present several nontrivial algorithmic optimizations (see Table 1). In particular, we show that a collection of algorithms exploiting SIMD instructions can enhance the performance of a data structure like Roaring in some cases, above and beyond what state-of-the-art optimizing compilers can achieve. To our knowledge, it is the first work to report on the benefits of advanced SIMD-based algorithms for compressed bitmaps.Although the approach we use to compute array intersections using SIMD instructions in Section 4.2 is not new, 22,23 our work on the computation of the union (Section 4.3), difference (Section 4.4), and symmetric difference (Section 4.4) of arrays using SIMD instructions might be novel and of general interest.• We benchmark our C library against a wide range of alternatives in C and C++. Our results provide guidance as to the strengths and weaknesses of our implementation.We focus primarily on our novel implementation and the lessons we learned: we refer to earlier work for details regarding the high-level algorithmic design of Roaring bitmaps. 18,19 Because our library is freely available under a liberal open-source license, we hope that our work will be used to accelerate information systems.

show abstract