Evaluating Load Generation in Virtualized Environments for Software Performance Testing

Abstract

Abstract-Before placing a software system into production, it is necessary to guarantee it provides users with a certain level of Quality-of-Service. Intensive performance testing is then necessary to achieve such a level and the tests require an isolated computing environment. Virtualization can therefore play an important role for saving energy costs by reducing the number of servers required to run performance tests and for allowing performance isolation when executing multiple tests in the same computing infrastructure. Load generation is an important component in performance testing as it simulates users interacting with the target application. This paper presents our experience in using a virtualized environment for load generation aimed at performance testing. We measured several performance metrics and varied system load, number of virtual machines per physical resource, and the CPU pinning schema for comparison of virtual and physical machines. The two main findings from our experiments are that physical machines produced steadier and faster response times under heavy load and that the pinning schema is an important aspect when setting up a virtualized environment for load generation.

A Survey on Load Testing of Large-Scale Software Systems

“…Netto et al [155] and White et al [156] evaluate the stability of the generated load under virtualized environments (e.g., virtual machines). They find that the system throughput sometimes might not produce stable load on virtual machines.…”

Development of a research environment for the operational and computational architecture of central bank digital currency software

“…Для решения поставленной задачи разработаем инфраструктуру среды исследований с применением технологий виртуализации [20].…”

A Case Study on the Stability of Performance Tests for Serverless Applications

“…Costa et al [9] summarized some bad practices of writing microbenchmarks using the JMH framework to mitigate the variation and instability of cloud environments when conducting performance microbenchmarking. Arif et al [4] and Netto et al [43] compared performance metrics generated via performance tests between virtual and physical environments. Their findings highlight the inconsistency between performance testing results in virtual and physical environments.…”