A configurable method for benchmarking scalability of cloud-native applications

Abstract

Cloud-native applications constitute a recent trend for designing large-scale software systems. However, even though several cloud-native tools and patterns have emerged to support scalability, there is no commonly accepted method to empirically benchmark their scalability. In this study, we present a benchmarking method, allowing researchers and practitioners to conduct empirical scalability evaluations of cloud-native applications, frameworks, and deployment options. Our benchmarking method consists of scalability metrics, measurement methods, and an architecture for a scalability benchmarking tool, particularly suited for cloud-native applications. Following fundamental scalability definitions and established benchmarking best practices, we propose to quantify scalability by performing isolated experiments for different load and resource combinations, which asses whether specified service level objectives (SLOs) are achieved. To balance usability and reproducibility, our benchmarking method provides configuration options, controlling the trade-off between overall execution time and statistical grounding. We perform an extensive experimental evaluation of our method’s configuration options for the special case of event-driven microservices. For this purpose, we use benchmark implementations of the two stream processing frameworks Kafka Streams and Flink and run our experiments in two public clouds and one private cloud. We find that, independent of the cloud platform, it only takes a few repetitions (≤ 5) and short execution times (≤ 5 minutes) to assess whether SLOs are achieved. Combined with our findings from evaluating different search strategies, we conclude that our method allows to benchmark scalability in reasonable time.

Queueing Model based Dynamic Scalability for Containerized Cloud

“…It included the scalability metrics and measurement methods to specify the achievement of the given service level objectives. It also provided the facility for configuring the scalability parameters for getting an efficient response [15]. The author in [16] proposed a container-based autoscaling procedure that used a heuristic technique for utilizing the resources efficiently.…”

“…So, the tasks waiting in the queue can be given as: (14) The CPU utilization can be given as: (15) The waiting time for the tasks at PM is:…”

Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices

“…This chapter builds upon our work previously published in the Empirical Software Engineering journal [HH22a]. It is organized as follows.…”

“…The demand metric describes how the required amount of provisioned resources evolves with increasing load intensities. The capacity metric indicates how the processible load intensity evolves with increasing amounts of provisioned resources This chapter builds upon our work previously published in the Empirical Software Engineering journal [HH22a] and presented at the ACM/SPEC International Conference on Performance Engineering 2021 [HH21a]. It is organized as follows.…”

“…It implements our proposed measurement method and, thus, measures our proposed scalability metrics (see Fig. This chapter builds upon our work previously published in the Empirical Software Engineering journal [HH22a]. It starts by describing our design rationale in Section 8.1.…”