The ghost in the machine : don’t let it haunt your software performance measurements

Abstract

This paper describes pitfalls, issues, and methodology for measuring software performance. Ideally, measurement should be performed and reported in such a way that others will be able to reproduce the results in order to confirm their validity. We aim to motivate scientists to apply the necessary rigor to the design and execution of their software performance measurements to achieve reliable results. Repeatability of experiments, comparability of reported results, and verifiability of claims that are based on such results can be achieved only when measurements and reporting procedures can be trusted. In short, this paper urges the reader to measure the right performance and to measure the performance right.

A Rational Foundation for Software Metrology

“…Then in 1997 the Joint Committee for Guides in Metrology (JCGM) was formed to maintain and promote the two documents, a role which continues to the present day. Lower resolution Higher resolution Figure 2: Resolution, from [1].…”

“…In addition to the primary authors, text and comments were contributed by William F. Guthrie, Frederick (Tim) Boland, Irena Bojanova, Jack Boudreaux, Albert Jones, Jeffrey Horlick, Frederic de Vaulx, Don Tobin, and W. Eric Wong. Figure 1 and Figure 2 were drawn by Vreda Pieterse for [1].…”

Empirically Assessing Algorithm Performance

“…This should be borne in mind whenever percentage information is assessed, and-in the present case-when considering differences between the relatively short times taken for large patterns to be matched. 5 These observations relied on the Activity Monitor utility q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q q parallel on the i7 and i5 machines. In the case of sequential runs, a test harness task would almost always be run on one single core, and hyper-threading would take place to dynamically distribute load between the two virtual cores that made up the single physical core.…”

“…should be used? Pieterse and Flater [5] discuss several of these matters and offer helpful guidelines for carrying out benchmarking. They also offer a fairly comprehensive bibliography of previous studies on benchmarking.…”

MoonlightBox: Mining Android API Histories for Uncovering Release-Time Inconsistencies

“…There are several studies proposed in the literature to devise reliable experimental protocols [28] [29] [30]. As argued by Blackburn et al [28], an unsound empirical finding may misdirect a whole field, encourage the pursuit of unworthy ideas.…”