Roaring bitmaps: Implementation of an optimized software library

Abstract

Compressed bitmap indexes are used in systems such as Git or Oracle to accelerate queries. They represent sets and often support operations such as unions, intersections, differences, and symmetric differences. Several important systems such as Elasticsearch, Apache Spark, Netflix's Atlas, LinkedIn's Pivot, Metamarkets' Druid, Pilosa, Apache Hive, Apache Tez, Microsoft Visual Studio Team Services, and Apache Kylin rely on a specific type of compressed bitmap index called Roaring. We present an optimized software library written in C implementing Roaring bitmaps: CRoaring. It benefits from several algorithms designed for the single-instruction-multiple-data instructions available on commodity processors. In particular, we present vectorized algorithms to compute the intersection, union, difference, and symmetric difference between arrays. We benchmark the library against a wide range of competitive alternatives, identifying weaknesses and strengths in our software. Our work is available under a liberal open-source license.• We present several nontrivial algorithmic optimizations (see Table 1). In particular, we show that a collection of algorithms exploiting SIMD instructions can enhance the performance of a data structure like Roaring in some cases, above and beyond what state-of-the-art optimizing compilers can achieve. To our knowledge, it is the first work to report on the benefits of advanced SIMD-based algorithms for compressed bitmaps.Although the approach we use to compute array intersections using SIMD instructions in Section 4.2 is not new, 22,23 our work on the computation of the union (Section 4.3), difference (Section 4.4), and symmetric difference (Section 4.4) of arrays using SIMD instructions might be novel and of general interest.• We benchmark our C library against a wide range of alternatives in C and C++. Our results provide guidance as to the strengths and weaknesses of our implementation.We focus primarily on our novel implementation and the lessons we learned: we refer to earlier work for details regarding the high-level algorithmic design of Roaring bitmaps. 18,19 Because our library is freely available under a liberal open-source license, we hope that our work will be used to accelerate information systems.

HDTQ: Managing RDF Datasets in Compressed Space

“…The HDTQ prototype 9 , built on top of the existing HDT-Java library 10 plements both HDT-AG and HDT-AT approaches using existing compressed bitmaps (called Roaring Bitmaps [15]), which are optimal for sparse bitsequences.…”

“…Table 2 lists the space requirements of the uncompressed RDF datasets in N-Quads notation (column "Size"), in gigabytes, the respective gzipped datasets (column "gzip") and the systems under review, as the ratio between the size for the required space and the uncompressed size. The numbers reported for HDT-AG and HDT-AT include the size of HDTQ and the additional HDT indexes (created with HDT-FoQ [16]) needed to resolve all quad patterns 15 . Note that Virtuoso was not capable of importing the BEAR-A dataset due to a persistent error when inserting large quad data.…”

“…Note that Virtuoso was not capable of importing the BEAR-A dataset due to a persistent error when inserting large quad data. In fact, a similar bug in the current Java implementation of Roaring Bitmaps [15] made us use Bitmap375 [11] for this particular scenario, as the HDTQ prototype supports both implementations interchangeably, being transparent to users consuming/querying HDTQ. As expected, gzip achieves large space savings that outperform the space needs of the RDF stores.…”

“…However, in practice, HDT-AT and HDT-AG can be implemented with compressed bitmaps [15] to minimize the size of the bitsequences.…”

“…The main reason lies in the compressed implementation of the bitmaps [15] that exploits consecutive runs of 0's or 1's to achieve further compression. In most of the cases, longer runs are produced when annotating the triples per graph (HDT-AG), than viceversa (HDT-AT).…”