Optimizing Druid with Roaring bitmaps

Abstract

In the current Big Data era, systems for collecting, storing and efficiently exploiting huge amounts of data are continually introduced, such as Hadoop, Apache Spark, Dremel, etc. Druid is one of theses systems especially designed to manage such data quantities, and allows to perform detailed real-time analysis on terabytes of data within sub-second latencies. One of the important Druid 's requirements is fast data filtering. To insure that, Druid makes an extensive use of bitmap indexes. Previously, we introduced a new compressed bitmap index scheme called Roaring bitmap that has shown interesting results when compared to the bitmap compression scheme adopted by Druid : Concise. Since, Roaring bitmap has been integrated to Druid as an indexing solution. In this work, we produce an extensive series of experiments in order to compare Roaring bitmap and Concise time-space performances when used to accelerate Druid 's OLAP queries and other kinds of operations Druid realizes on bitmaps, like: retrieving set bits from bitmaps, computing bitmap complements, aggregating several bitmaps with logical ORs and ANDs operations. Roaring bitmap has shown to improve up to ≈ 5× analytical queries response times under Druid compared to Concise.

Challenging SQL-on-Hadoop Performance with Apache Druid

“…Although other benchmarks are available, to the best of our knowledge, they do not evaluate such diverse set of tools. For Druid, although a very promising tool, few works have studied this technology and most of them do not use significant data volumes [8], [11] or do not compare the results against other relevant technologies, typically used in OLAP workloads on Big Data environments [8]. The work of [12] contributed to this gap by using the SSB to evaluate Druid, also pointing some recommendations regarding performance optimization.…”

Roaring bitmaps: Implementation of an optimized software library

“…These original sets are immutable: they are not modified during the benchmarks. For a validation of Roaring in a database system with external memory processing, see, for example, the work of Chambi et al, where they showed that switching to Roaring from Concise improved system‐level benchmarks.…”

“…For example, if we compute the intersection between 2 sets, we generate a new data structure with the result. This reflects a common usage scenario, eg, Druid relies on collections of immutable and persistent compressed bitmaps to accelerate queries.…”

“…Although there are many ways to compress bitmaps (see Section 2.1), several systems rely on an approach called Roaring including Elasticsearch, Metamarkets' Druid, eBay's Apache Kylin, Netflix's Atlas, LinkedIn's Pinot, Pilosa, Apache Hive, Apache Tez, Microsoft Visual Studio Team Services, and Apache Spark. () In turn, these systems are in widespread use, eg, Elasticsearch provides the search capabilities of Wikipedia .…”

Tree-Encoded Bitmaps

“…Longer runs are mapped to tree nodes closer to the root, and vice versa. have many applications, such as efficiently evaluating predicates [42,45,46] and have been used to accelerate join [44] and aggregation [9,46] queries. For medium or high cardinality columns, bitmap indexes consist of many individual bitmaps that are sparsely populated with 1-bits.…”