Systems Benchmarking

Kounev, Samuel; Lange, Klaus-Dieter; Kistowski, Jóakim von

doi:10.1007/978-3-030-41705-5

Cited by 39 publications

(23 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, customers can use benchmarks to compare competing products from different suppliers, and researchers use benchmarks to evaluate novel system architectures. 1 However, the redundancy in the benchmark suites will increase the evaluation costs of computer system performance evaluation. 2,3 In addition, when developing, deploying, and operating the SUT, benchmarks can also be used to verify a given hardware and software configuration through simulation, but the simulation time is can be much slower than the actual execution time, 4 for instance, the microservices can be executed within milliseconds in realistic testbed, while the simulation toolkit can consume seconds or minutes for execution.…”

Section: Introductionmentioning

confidence: 99%

“…However, in the past couple of decades, the application scenarios of benchmarking have gradually increased. For example, customers can use benchmarks to compare competing products from different suppliers, and researchers use benchmarks to evaluate novel system architectures 1 . However, the redundancy in the benchmark suites will increase the evaluation costs of computer system performance evaluation 2,3 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

BenchSubset: A framework for selecting benchmark subsets based on consensus clustering

Zhan

Lin

Mao

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

The redundancy in the benchmark suite will increase the time for computer system performance evaluation and simulation. The most typical method to solve this problem is to select subsets based on clustering. However, it is a challenge to validate benchmark subsetting results for unlabeled benchmark suites when using the clustering method, and existing research has not considered this problem. Also, there is no quantitative evaluation method for subsetting which can reflect the universal and the diversity characteristics of the benchmark suite at the same time. To solve the above problems, we propose BenchSubset, a framework for selecting benchmark subsets based on consensus clustering, which includes Group Principal Components Analysis, consensus clustering, and a new evaluation method considering the universal and the diversity characteristics of the benchmark suite. We conducted SPEC CPU2017 subsetting experiments on Huawei's Taishan 200, then verified the effectiveness of Bench-Subset in selecting a benchmark subset. Compared with the mainstream principal components analysis with hierarchical clustering (PCA-H) method, the

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

BenchSubset: A framework for selecting benchmark subsets based on consensus clustering

Zhan

Lin

Mao

et al. 2022

Int J of Intelligent Sys

View full text Add to dashboard Cite

show abstract

“…This procedure is often time-consuming and costly and the results of these benchmarks are seldom repeatable nor portable to other functions [6]. Also other benchmarking characteristics [7,8] like simplicity or efficiency are hard to achieve when designing a benchmark but even more challenging when other researchers want to assess and repeat the experiments since a lot of publications try to "fool the masses with irreproducible results" 4 .…”

Section: Introductionmentioning

confidence: 99%

SeMoDe – Simulation and Benchmarking Pipeline for Function as a Service

Manner¹

2021

Bamberger Beiträge Zur Wirtschaftsinformatik Und Angewandten Informatik

View full text Add to dashboard Cite

Cloud computing started with the promise of delivering computing resources elastically at scale, pay per use and on demand self-service to name a few capabilities. In early 2016, Amazon Web Services (AWS) launched a new product called AWS Lambda which started the so called serverless hype and established a new cloud delivery model, namely Function as a Service (FaaS). FaaS offerings keep the promise of delivering computing resources on demand. They dynamically scale up and down function instances and introduce the most fine-grained billing model across all as-a-service offerings by accounting on a milliseconds basis. Despite this flexibility and the possibility to concentrate on the business functionality, a FaaS user loses operational control. Only a few configuration options remain to tune the functions. The first pay-as-you-go billing model raises new questions for performance-cost trade-offs. In order to choose a suitable configuration dependent on the use case and get a solid understanding of performance impact of FaaS platforms, SeMoDe implements a benchmarking and simulation pipeline. It calibrates a physical developer machine, simulates the function in different settings which are comparable to those of cloud offerings and enables a decision guidance to choose an appropriate configuration when deploying it. Based on a Structured Literature Review (SLR) to show the benchmarking and simulation efforts, I suggest a checklist for conducting fair, repeatable and meaningful benchmarks with a focus on documenting the experiments.

show abstract

“…On the one hand, there are formal models used to analyze performance [3,4]. On the other hand, to describe systems, they use simulation models: QNs [5,6], CPN [5,7] or QPN [8][9][10]. In recent years, most works focused on web systems [4,6,11,12], which are very efficient, and able to handle numerous incoming requests.…”

mentioning

confidence: 99%

Recommendations for Using QPN Formalism for Preparation of Incoming Request Stream Generator in Modeled System

Rak

Rzońca

2021

Applied Sciences

View full text Add to dashboard Cite

Simulation models are elements of science that use software tools to solve complex mathematical problems. They are beneficial in areas such as performance engineering and communications systems. Nevertheless, to achieve more accurate results, researchers should use more detailed models. Having an analysis of the system operations in the early modeling phases could help one make better decisions relating to the solution. In this paper, we introduce the use of the QPME tool, based on queueing Petri nets, to model the system stream generator. This formalism was not considered during the first tool development. As a result of the analysis, an alternative design model is proposed. By comparing the behavior of the proposed generator against the one already developed, a better adjustment of the stream to the customer’s needs was obtained. The study results show that appropriately adjusting queueing Petri net models can help produce better streams of data (tokens).

show abstract

Systems Benchmarking

Cited by 39 publications

References 0 publications

BenchSubset: A framework for selecting benchmark subsets based on consensus clustering

BenchSubset: A framework for selecting benchmark subsets based on consensus clustering

SeMoDe – Simulation and Benchmarking Pipeline for Function as a Service

Recommendations for Using QPN Formalism for Preparation of Incoming Request Stream Generator in Modeled System

Contact Info

Product

Resources

About