High availability, elasticity, and strong consistency for massively parallel scans over relational data

Abstract

An elastic and highly available data store is a key component of many cloud applications. Existing data stores with strong consistency guarantees are designed and optimized for small updates, key-value access, and (if supported) small range queries over a predefined key column. This raises performance and availability problems for applications which inherently require large updates, non-key access, and large range queries. This paper presents a solution to these problems: Crescando/RB; a distributed, scan-based, main memory, relational data store (single table) with robust performance and high availability. The system addresses a real, large-scale industry use case: the Amadeus travel management system. This paper focuses on the distribution layer of Crescando/RB, the problem and theory behind it, the rationale underlying key design decisions, and the novel multicast protocol and replication framework it is composed of. Highlighting the key features of the distribution layer, we present experimental results showing that even under permanent node failures and large-scale data repartitioning, Crescando/RB remains fully available and capable of sustaining a heavy query and update load.

AllConcur

“…Travel reservation systems are typical scenarios where updates are preceded by a large number of queries, e.g., clients check many flights before choosing a ticket. To avoid overloading a central server, existing systems either adopt weaker consistency models, such as eventual consistency [18], or partition the state [60], not allowing transactions spanning multiple partitions. ALLCON-CUR offers strong consistency by distributing queries over multiple servers that agree on the entire state.…”

Dare

“…Thus, DARE is intended to store metadata of more complex operations. A strategy to increase scalability would be partitioning data into multiple (reliable) DARE groups and delivering client requests through a routing mechanism [41]. Yet, routing requests that involve multiple groups would require consensus.…”

“…Thus, highly scalable systems typically utilize RSMs only for management tasks and improve overall performance by relaxing request ordering [11] which implicitly shifts the burden of consistency management to the application layer. Yet, many services, such as airline reservation systems, require a consistent view of the complete distributed database at very high request rates [41].…”

On Developing and Operating of Data Elasticity Management Process

“…changes of the QoR metrics associated with data asset. However, current models of data services [1][2][3]5] are not yet associated with QoR that can support the above-mentioned requirements. These challenges motivate us to develop a technique to support generating DEP.…”

“…To provide flexible access to vast amounts of data, several types of Data-as-a-Service (DaaS) have emerged to allow users to execute data analytics atop a vast, rich set of data sources, such as Azure's Data Market 1 , Infochimps 2 , Factual 3 , and a number of research systems [1][2][3]. Many of these systems support the concept of elasticity by scaling data nodes, re-assigning data partitions and reconfiguring data clusters to automatically adapt to dynamic changes in their workload [1][2][3]. Overall, these systems support the elasticity of data services at the infrastructure level.…”