A microbenchmark case study and lessons learned

Abstract

The extra abstraction layer posed by the virtual machine, the JIT compilation cycles and the asynchronous garbage collection are the main reasons that make the benchmarking of Java code a delicate task. The primary weapon in battling these is replication: "billions and billions of runs", is phrase sometimes used by practitioners. This paper describes a case study, which consumed hundreds of hours of CPU time, and tries to characterize the inconsistencies in the results we encountered.

Dynamically reconfiguring software microbenchmarks: reducing execution time without sacrificing result quality

“…The mean change rate between the three stoppage criteria and the static approach is~3% or lower for all three. Note that, following a rigorous measurement methodology,~3% could still be caused by JVM instabilities unrelated to our approach [19]. Again, RCIW is the best criterion with 1.4%±3.8%.…”

“…Benchmarks often exhibit multiple steady states resulting in multi-modal distributions, and outliers, due to non-deterministic behavior, might still occur even after stable considered a fork to be in a steady state [19]. Therefore, our approach uses a fixed number of multiple measurement iterations mi (lines 8ś9), as a single measurement iteration would not accurately represent a fork's performance.…”

“…Result quality is impaired by not running enough measurements as well as measurement bias, which requires careful experiment planning and execution [7,11,15,18,19,28,40]. To mitigate measurement bias, Georges et al [18] outlined a rigorous methodology how to asses performance of Java programs, which we base our measurement technique on.…”

RIM4J: An Architecture for Language-Supported Runtime Measurement against Malicious Bytecode in Cloud Computing

“…For each benchmark, it has the shortest reduction of performance at a frequency of 60 s except for the Arith benchmark in Figure 11. For the Arith, RIM4J imposes a 68% reduction at a frequency of 60 s, but a 60% reduction when at a frequency of 5 s. We attribute these small differences mostly to the instabilities of the JVM rather than the impact from our RIM4J [48].…”

DOs and DON'Ts of Conducting Performance Measurements in Java

“…There is much evidence that it is easy to unwittingly conduct performance experiments that produce tricky results [6,7,4,8,21,3,14]. On contemporary execution platforms, even small scale performance experiments are turning from an easy and reliable way of evaluating software performance into a difficult exercise of tracking a multitude of technical details that must be dealt with even in otherwise very simple scenarios.…”