Taming Energy Consumption Variations In Systems Benchmarking

Abstract

The past decade witnessed the inclusion of power measurements to evaluate the energy efficiency of software systems, thus making energy a prime indicator along with performance. Nevertheless, measuring the energy consumption of a software system remains a tedious task for practitioners. In particular, the energy measurement process may be subject to a lot of variations that hinder the relevance of potential comparisons. While the state of the art mostly acknowledged the impact of hardware factors (chip printing process, CPU temperature), this paper investigates the impact of controllable factors on these variations. More specifically, we conduct an empirical study of multiple controllable parameters that one can easily tune to tame the energy consumption variations when benchmarking software systems. To better understand the causes of such variations, we ran more than a 1, 000 experiments on more than 100 nodes with different workloads and configurations. The main factors we studied encompass: experimental protocol, CPU features (C-states, Turbo Boost, core pinning) and generations, as well as the operating system. Our experiments showed that, for some workloads, it is possible to tighten the energy variation by up to 30×. Finally, we summarize our results as guidelines to tame energy consumption variations. We argue that the guidelines we deliver are the minimal requirements to be considered prior to any energy efficiency evaluation. CCS CONCEPTS • Hardware → Power and energy; Power estimation and optimization; Platform power issue; Enterprise level and data centers power issues.

Tales from the Code #1: The Effective Impact of Code Refactorings on Software Energy Consumption

“…We also pay special attention to the commits of each project that exhibit the most energy difference, when exceeding a threshold of 5%. This threshold is computed from the minimum CPU energy consumption variation and the computed standard deviation of the experiments (Ournani et al, 2020).…”

“…We also used OPENJDK, version 1.8.0 242, to run most of our Java experiments-i.e., run both old and recent versions-except for the OkHttp project where we had to use OPENJDK, version 11.0.6. By using the same machine to conduct all the experiments, we guarantee the least energy consumption variation and a controlled impact of the hardware configuration, as recommended in (Ournani et al, 2020), especially to measure small differences in energy consumption. Therefore, the machine has been configured according to guidelines of (Ournani et al, 2020) to mitigate the energy consumption variation to the minimum and produce accurate results that can be faithfully compared.…”

Evaluating the Impact of Java Virtual Machines on Energy Consumption

“…Our experimental protocol enforces that the software under test is the only process executed on the node configured with a very minimal Linux Debian 9 (4.9.0 kernel version). The minimal OS configuration ensures that only mandatory services and daemons are kept active to conduct robust experiments and reduce the factors that can affect the energy consumption measurements during our experiments [15].…”

“…RAPL is one of the most accurate tools to report on the global energy consumption of a processor [6,9]. We note that, due to CPU energy consumption variations issues [15], we used the same node for all our experiments. Moreover, we tried to be very careful, while running our experiments, not to fall in most common benchmarking "crimes" [23].…”

“…Software developers often report that such solutions should come from more energy-efficient hardware components or optimized algorithms [15,24] but, given the complexity of modern software environments, the composition of software layers makes this sustainability objective particularly challenging.…”