Statistically rigorous java performance evaluation

Abstract

Java performance is far from being trivial to benchmark because it is affected by various factors such as the Java application, its input, the virtual machine, the garbage collector, the heap size, etc. In addition, non-determinism at run-time causes the execution time of a Java program to differ from run to run. There are a number of sources of non-determinism such as Just-In-Time (JIT) compilation and optimization in the virtual machine (VM) driven by timerbased method sampling, thread scheduling, garbage collection, and various system effects.There exist a wide variety of Java performance evaluation methodologies used by researchers and benchmarkers. These methodologies differ from each other in a number of ways. Some report average performance over a number of runs of the same experiment; others report the best or second best performance observed; yet others report the worst. Some iterate the benchmark multiple times within a single VM invocation; others consider multiple VM invocations and iterate a single benchmark execution; yet others consider multiple VM invocations and iterate the benchmark multiple times. This paper shows that prevalent methodologies can be misleading, and can even lead to incorrect conclusions. The reason is that the data analysis is not statistically rigorous. In this paper, we present a survey of existing Java performance evaluation methodologies and discuss the importance of statistically rigorous data analysis for dealing with non-determinism. We advocate approaches to quantify startup as well as steady-state performance, and, in addition, we provide the JavaStats software to automatically obtain performance numbers in a rigorous manner. Although this paper focuses on Java performance evaluation, many of the issues addressed in this paper also apply to other programming languages and systems that build on a managed runtime system.

SCnC: Efficient Unification of Streaming with Dynamic Task Parallelism

“…Therefore graph topology, and in particular the splitting factor at data-parallel split-join nodes, can have an impact on performance. The experimental results were performed according to the methodology suggested by Georges et al [11].…”

Effectively mapping linguistic abstractions for message-passing concurrency to threads on the Java virtual machine

“…We have implemented these four default policies in Panini Capsules and our comparison uses the same Panini program. We measure program runtime and CPU consumption for thread, round-robin, random, work-stealing and our technique when the steady-state performance is reached by following the methodology of Georges et al [13]. We compare program runtime and CPU consumption for these five policies on 2, 4, 8, and 12 cores settings (Linux taskset utility is used for altering core settings on 12-core system).…”

Performance testing in the cloud. How bad is it really?

“…Within each trial, every benchmark (e.g., etcd-1) consists of 50 repeated executions (e.g., using the -i50 parameter of JMH) and every execution produces a single data point, which reports the average execution time in ns. For JMH benchmarks, we also run 10 warmup executions (after which steady-state performance is most likely reached [9]) prior to the test executions. The performance counters originating from warmup iterations are discarded.…”

“…Others report on wrongly quantified experimental evaluations by ignoring uncertainty of measurements through nondeterministic behavior of software systems (e.g., memory placement, dynamic compilation) [15]. Dealing with non-deterministic behavior of dynamically optimized programming languages, Georges et al [9] summarize methodologies to measure languages like Java, and explain statistical methods to use for performance evaluation. All of these studies expect an as stable as possible environment to run performance experiments on.…”