ChronicleDB

Abstract

Reactive security monitoring, self-driving cars, the Internet of Things (IoT), and many other novel applications require systems for both writing events arriving at very high and fluctuating rates to persistent storage as well as supporting analytical ad hoc queries. As standard database systems are not capable of delivering the required write performance, log-based systems, key-value stores, and other write-optimized data stores have emerged recently. However, the drawbacks of these systems are a fair query performance and the lack of suitable instant recovery mechanisms in case of system failures. In this article, we present ChronicleDB, a novel database system with a storage layout tailored for high write performance under fluctuating data rates and powerful indexing capabilities to support a variety of queries. In addition, ChronicleDB offers low-cost fault tolerance and instant recovery within milliseconds. Unlike previous work, ChronicleDB is designed either as a serverless library to be tightly integrated in an application or as a standalone database server. Our results of an experimental evaluation with real and synthetic data reveal that ChronicleDB clearly outperforms competing systems with respect to both write and query performance.

Out-of-Order Sliding-Window Aggregation with Efficient Bulk Evictions and Insertions

“…Their work shows how to handle all combinations of order, window kinds, aggregation operations, etc., and is complementary to this paper. ChronicleDB uses a temporal aggregate B+-tree and optimizes writes to persistent storage while handling moderate amounts of out-of-order data by leaving some free space in each block [20]. Hammer Slide uses SIMD instructions to speed up sliding-window aggregation [25]; SlideSide generalizes it to the multi-query case [27]; and LightSaber further generalizes it for parallelism [26].…”

“…Bulk evictions are common in time-based windows, where the arrival of one data item at the youngest end of the window can trigger the eviction of several data items at the oldest end. For example, consider a window of size 60 seconds, with data items at timestamps [0.1,0.2,0.3,0.4,0.5, 10,20,30,40,50,60] seconds. If the next data item to be inserted has timestamp 61, the window must evict the items at timestamps [0.1,0.2,0.3,0.4,0.5].…”

Designing an Event Store for a Modern Three-layer Storage Hierarchy

“…The layout operates on data blocks of fixed size, where the size of such a block is a multiple of the uncompressed size of a TAB + -node. For instance, in our previous work [34], we used 32 KiB blocks for uncompressed 8 KiB TAB + -nodes. Each of these blocks either contains compressed TAB + -nodes or the translation information for a set of TAB + -nodes.…”

“…Hence, instead of a direct replacement of traditional storage or main memory, we will examine PMem as a third layer in the storage hierarchy of an event store. The discussion will be framed as a case study for ChronicleDB [33,34], a special-purpose database system for event streams. Even though we use ChronicleDB as an example, the lessons learned can be applied to both, more specialized (e.g., temporal indexing/storage design) as well as less specialized systems (e.g., ingestion/recovery).…”

Workload-Aware Performance Tuning for Multimodel Databases Based on Deep Reinforcement Learning

“…Tere are hundreds of parameters in a database system, and there are connections between parameters, which makes it difcult for DBAs to complete the tuning of these confguration parameters. Second, the scheme for confguring parameters cannot be reused for database systems deployed in diferent environments (such as local hosts, clouds, or memory) [12][13][14]. It is difcult for DBAs to achieve high database performance by efcient parameter tuning under changing scenarios.…”