A Versatile Framework for Painless Benchmarking of Database Management Systems

Brent, Lexi; Fekete, Alan

doi:10.1007/978-3-030-12079-5_4

Cited by 5 publications

(3 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Evaluation automation support (R4) covering all ETs ensure transparency and reproducibility that are key concepts in DBMS and cloud service evaluations [10], [11]. Moreover, portability support is required by a reasonable level of abstraction across the cloud, DBMS and workload domain [18], [21], i.e. enabling the execution of evaluation scenario on different cloud resources or for different DBMS.…”

Section: B Elasticity Evaluation Requirementsmentioning

confidence: 99%

“…While there is a significant number of DBMS workloads that measure the performance metrics at client side, none of them supports the required adaptions at operator side [18]. Consequently, supportive frameworks are required to automate the evaluations and enable reproducible results [17], [18], [21]. However, DBMS frameworks that also support the operator side are limited [14], [31] and none of them supports the specification and execution of elasticity evaluations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Kaa: Evaluating Elasticity of Cloud-Hosted DBMS

Seybold¹,

Volpert²,

Wesner³

et al. 2019

2019 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)

View full text Add to dashboard Cite

Auto-scaling is able to change the scale of an application at runtime. Understanding the application characteristics, scaling impact as well as the workload, an auto-scaler aligns the acquired resources to match the current workload. For distributed Database Management Systems (DBMS) forming the backend of many large-scale cloud applications, it is currently an open question to what extent they support scaling at run-time. In particular, elasticity properties of existing distributed DBMS are widely unknown and difficult to evaluate and compare. This paper presents a comprehensive methodology for the evaluation of the elasticity of distributed DBMS. On the basis of this methodology, we introduce a framework that automates the full evaluation process. We validate the framework by defining significant elasticity scenarios for a case study that comprises two DBMS for write-heavy and read-heavy workloads of different intensities. The results show that scalable distributed DBMS are not necessarily elastic and that adding more instances to a cluster at run-time may even decrease the experienced performance.

show abstract

Section: B Elasticity Evaluation Requirementsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Kaa: Evaluating Elasticity of Cloud-Hosted DBMS

Seybold¹,

Volpert²,

Wesner³

et al. 2019

2019 IEEE International Conference on Cloud Computing Technology and Science (CloudCom)

View full text Add to dashboard Cite

show abstract

“…To our knowledge, there is no other such tool, c.f. also the studies in Seybold & Domaschka (2017) and Brent & Fekete (2019). There are other tools like Apache JMeter (Java), Ham-merDB (Tcl), Sysbench (LuaJIT), OLTPBench (Java), and BenchBase (Java) that provide very nice features.…”

mentioning

confidence: 97%

DBMS-Benchmarker: Benchmark and Evaluate DBMS in Python

Erdelt¹,

Jestel²

2022

JOSS

View full text Add to dashboard Cite

DBMS-Benchmarker is a Python-based application-level blackbox benchmark tool for Database Management Systems (DBMS). It is intended for reproducible measurement and easy evaluation of the performance the user receives, even in complex benchmark situations. It connects to a given list of DBMS (via JDBC) and runs a given list of (SQL) benchmark queries. Queries can be parametrized and randomized. Results and evaluations are available via a Python interface and can be inspected with standard Python tools like pandas DataFrames. An interactive visual dashboard assists in multi-dimensional analysis of the results.

show abstract

A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance Benchmarking

Erdelt

2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

A Versatile Framework for Painless Benchmarking of Database Management Systems

Cited by 5 publications

References 17 publications

Kaa: Evaluating Elasticity of Cloud-Hosted DBMS

Kaa: Evaluating Elasticity of Cloud-Hosted DBMS

DBMS-Benchmarker: Benchmark and Evaluate DBMS in Python

A Framework for Supporting Repetition and Evaluation in the Process of Cloud-Based DBMS Performance Benchmarking

Contact Info

Product

Resources

About