BenchFoundry: A Benchmarking Framework for Cloud Storage Services

Bermbach, David; Kuhlenkamp, Jörn; Dey, Akon; Ramachandran, Arunmoezhi; Fekete, Alan; Tai, Stefan

doi:10.1007/978-3-319-69035-3_22

Cited by 23 publications

(28 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, each workload implementation provides its own set of workload constraints, which have direct impact on the results. Hence, workload specifications for comparative DBMS evaluations require portable workloads to compare different DBMS against the evaluation objectives [6,29]. DBMS-specific scenarios need to support realistic workloads to enable the in-depth evaluation of DBMS-specific features [6,31].…”

Section: Dbms Workload Domainmentioning

confidence: 99%

“…Hence, workload specifications for comparative DBMS evaluations require portable workloads to compare different DBMS against the evaluation objectives [6,29]. DBMS-specific scenarios need to support realistic workloads to enable the in-depth evaluation of DBMS-specific features [6,31]. Respectively, the workload specification has to abstract common workload constraints to enable comparative and DBMS-specific workload specifications.…”

Section: Dbms Workload Domainmentioning

confidence: 99%

“…Each domain requires detailed knowledge on its own and the entire evaluation process comprises a sequence of multiple interdependent evaluation tasks as depicted in Figure 2. Consequently, a supportive framework is required that automates the evaluation process and fulfils the established requirements of DBMS evaluation [6,7,21,29,35]: Ease of use (R1) : The deployment and configuration of the framework needs to be simple and it needs to provide userfriendly interfaces to specify and execute the evaluation scenarios [6,7,21,29] Portability (R2): In order to execute the evaluation scenarios for different domain properties, the framework has to abstract technical implementations of each evaluation domain and map the high-level evaluation scenarios to concrete technical implementations for each evaluation domain [21,35] Reproducibility (R3): The framework needs to provide comparable evaluation scenario templates, which ensure the deterministic execution for concrete domain constraints [6,29,35] Automation (R4): The automated execution of multi-domain evaluation scenarios requires the orchestration of evaluation tasks across all domains to ensure the reproducibility and portability [35]. In addition, complex evaluation objectives such as elasticity or availability require the DBMS cluster adaptation at evaluation runtime [36].…”

Section: Evaluation Scenario Executionmentioning

confidence: 99%

“…In addition, complex evaluation objectives such as elasticity or availability require the DBMS cluster adaptation at evaluation runtime [36]. Significance (R5): In order to enable significant results by applying realistic domain constraints, the framework needs to support comparative and realistic workloads, commercial cloud resource offerings and relevant DBMS [6,29,35] Extensibility (R6): As each evaluation domain is constantly evolving, the framework needs to provide an extensible architecture and interfaces that allow the easy integration of future domain specific constraints and evaluation objectives [7,21,29,35].…”

Section: Evaluation Scenario Executionmentioning

confidence: 99%

“…While the YCSB workloads target artificial CRUD operations, web-application workloads are presented by OLTP-Bench [14] and BG [3]. BenchFoundry [6] presents a trace-based workload generator for realistic workloads.…”

Section: Benchmarksmentioning

confidence: 99%

See 4 more Smart Citations

Mowgli

Seybold

Keppler

Gründler

et al. 2019

Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Big Data and IoT applications require highly-scalable database management system (DBMS), preferably operated in the cloud to ensure scalability also on the resource level. As the number of existing distributed DBMS is extensive, the selection and operation of a distributed DBMS in the cloud is a challenging task. While DBMS benchmarking is a supportive approach, existing frameworks do not cope with the runtime constraints of distributed DBMS and the volatility of cloud environments. Hence, DBMS evaluation frameworks need to consider DBMS runtime and cloud resource constraints to enable portable and reproducible results. In this paper we present Mowgli, a novel evaluation framework that enables the evaluation of non-functional DBMS features in correlation with DBMS runtime and cloud resource constraints. Mowgli fully automates the execution of cloud and DBMS agnostic evaluation scenarios, including DBMS cluster adaptations. The evaluation of Mowgli is based on two IoT-driven scenarios, comprising the DBMSs Apache Cassandra and Couchbase, nine DBMS runtime configurations, two cloud providers with two different storage backends. Mowgli automates the execution of the resulting 102 evaluation scenarios, verifying its support for portable and reproducible DBMS evaluations. The results provide extensive insights into the DBMS scalability and the impact of different cloud resources. The significance of the results is validated by the correlation with existing DBMS evaluation results.Since the era of RDBMS, their selection is guided by domain-specific benchmarks that have evolved together with distributed DBMSs.

show abstract

Section: Dbms Workload Domainmentioning

confidence: 99%

Section: Dbms Workload Domainmentioning

confidence: 99%

Section: Evaluation Scenario Executionmentioning

confidence: 99%

Section: Evaluation Scenario Executionmentioning

confidence: 99%

Section: Benchmarksmentioning

confidence: 99%

See 3 more Smart Citations

Mowgli

Seybold

Keppler

Gründler

et al. 2019

Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

show abstract

From zero to fog: Efficient engineering of fog‐based Internet of Things applications

2021

Self Cite

View full text Add to dashboard Cite

In Internet of Things (IoT) data processing, cloud computing alone does not suffice due to latency constraints, bandwidth limitations, and privacy concerns. By introducing intermediary nodes closer to the edge of the network that offer compute services in proximity to IoT devices, fog computing can reduce network strain and high access latency to application services. While this is the only viable approach to enable efficient IoT applications, the issue of component placement among cloud and intermediary nodes in the fog adds a new dimension to system design. State‐of‐the‐art solutions to this issue rely on simulation or solving a formalized assignment problem through heuristics only, which both have their drawbacks. In this article, we present a five‐step process for designing practical fog‐based IoT applications that combines best practices, simulation, and testbed analysis to converge towards an efficient system architecture. We then apply this process in a smart factory case study. By deploying filtered options to a physical testbed, we show that each step of our process converges towards more efficient application designs.

show abstract

A Versatile Framework for Painless Benchmarking of Database Management Systems

Brent

Fekete

2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

BenchFoundry: A Benchmarking Framework for Cloud Storage Services

Cited by 23 publications

References 13 publications

Mowgli

Mowgli

From zero to fog: Efficient engineering of fog‐based Internet of Things applications

A Versatile Framework for Painless Benchmarking of Database Management Systems

Contact Info

Product

Resources

About