Is Distributed Database Evaluation Cloud-Ready?

Abstract

The database landscape has significantly evolved over the last decade as cloud computing enables to run distributed databases on virtually unlimited cloud resources. Hence, the already non-trivial task of selecting and deploying a distributed database system becomes more challenging. Database evaluation frameworks aim at easing this task by guiding the database selection and deployment decision. The evaluation of databases has evolved as well by moving the evaluation focus from performance to distribution aspects such as scalability and elasticity. This paper presents a cloud-centric analysis of distributed database evaluation frameworks based on evaluation tiers and framework requirements. It analysis eight well adopted evaluation frameworks. The results point out that the evaluation tiers performance, scalability, elasticity and consistency are well supported, in contrast to resource selection and availability. Further, the analysed frameworks do not support cloud-centric requirements but support classic evaluation requirements.

Mowgli

“…Hence, the evaluation approach requires the specification of multi-domain evaluation scenarios as depicted in Figure 1. Each evaluation domain comprises its own set of domain specific constraints, which affect the results for the specified evaluation objectives [35]. Consequently, domain knowledge in each evaluation domain is required, which makes the DBMS evaluation a complex and error prone task.…”

“…Yet, choosing the right DBMS set-up in the jungle of available solutions is a complex task that is not done with the selection of a well-suited DBMS 1 , but continues with the selection of a cloud provider, and ends with the choice of the right size and amount of virtual machines. The three choices influence each other [35], so that making independent decisions may lead to sub-optimal results. Additionally, runtime parameters, including the expected workload, consistency requirements, and availability considerations, are influencing the set-up and depend on each other: for instance, the type of workload can influence whether a user should pay for having a local SSD attached to their virtual machines or not [18].…”

“…While benchmarking is an established approach to select software systems as well as hardware platforms, existing DBMS benchmarking frameworks cannot cope with the volatility of cloud environments [35], particularly as volatile environments demand for reliable and reproducible benchmarking [29]. Based on these observations, we claim that even with the knowledge of the workload and non-functional constraints, a manual selection of DBMS, cloud provider(s), and virtual machine types cannot deliver satisfactory results and that suitable tool support is strongly needed.…”

Towards a framework for orchestrated distributed database evaluation in the cloud

“…Our results show significant differences with respect to elasticity and the need for orchestrated DDBMS evaluation in order to provide adaptive and reproducible evaluation scenarios. Consequently, we analyze existing EFs with the focus on their evaluation scenarios and their consideration of cloud resources [11]. As existing EFs do not yet support orchestrated evaluation scenarios, elasticity and availability evaluation lacks dedicated support.…”

“…Evaluating these requirements of existing DDBMS is a common approach to guide the DDBMS selection process. Yet, current evaluation frameworks (EFs) do not explicitly consider the usage of heterogeneous cloud resources and lack the support for orchestrated evaluation scenarios [11] with respect to scalability, elasticity and availability.…”