Predictable performance for unpredictable workloads

Abstract

This paper introduces Crescando: a scalable, distributed relational table implementation designed to perform large numbers of queries and updates with guaranteed access latency and data freshness. To this end, Crescando leverages a number of modern query processing techniques and hardware trends. Specifically, Crescando is based on parallel, collaborative scans in main memory and so-called "querydata" joins known from data-stream processing. While the proposed approach is not always optimal for a given workload, it provides latency and freshness guarantees for all workloads. Thus, Crescando is particularly attractive if the workload is unknown, changing, or involves many different queries. This paper describes the design, algorithms, and implementation of a Crescando storage node, and assesses its performance on modern multi-core hardware.

High availability, elasticity, and strong consistency for massively parallel scans over relational data

“…As shown in a previous paper [74], Crescando is also much less sensitive to the combination and selectivity of selection predicates than a conventional relational database. This makes the engine especially suited for real-time business intelligence workloads such as our use case, where key-value stores are too restrictive in their consistency and/or data model, and relational databases lack the required performance, elasticity, availability, and robustness.…”

“…As an alternative, we have developed Crescando, a main memory, scan-based engine designed for high throughput and robustness to evolving workloads with unpredictable, multi-dimensional predicates. The engine, published in an earlier paper [74], is now in production use at Amadeus, providing data services that were previously infeasible.…”

“…Owing to this unusual design, a single Crescando instance on a commodity machine can handle thousands of ad hoc, non-key queries and updates per second [74]. The downside is that the turn-around time for individual operations is in the order of seconds, regardless of how few or how many records are touched.…”

“…This design has a few advantages. First, it naturally enables both horizontal parallelism (e.g., parallel join) and vertical parallelism (pipelining) [18,74]. Second, individual operators run in tight loops, which leads to high instruction locality [46].…”

Scaling up analytical queries with column-stores

“…During query execution, it reads pages from the disks in order, but asynchronously, hence aggregating the throughput of the linear scan of each disk. SharedDB [11] utilizes the Crescando [28] storage manager which uses a circular scan of in-memory table partitions, interleaving reads and writes of a batch of queries. Each Crescando scan applies the updates generated by the current batch of queries for each scanned tuple following the queries' arrival order, and then executes the read requests.…”