EMP: execution time measurement protocol for compute‐bound programs

Abstract

Measuring execution time is one of the most used performance evaluation techniques in computer science research. Inaccurate measurements cannot be used for a fair performance comparison between programs. Despite the prevalence of its use, the intrinsic variability in the time measurement makes it hard to obtain repeatable and accurate timing results of a program running on an operating system. We propose a novel execution time measurement protocol (termed EMP) for measuring the execution time of a compute-bound program on Linux, while minimizing that measurement's variability. During the development of execution time measurement protocol, we identified several factors that disturb execution time measurement. We introduce successive refinements to the protocol by addressing each of these factors, in concert, reducing variability by more than an order of magnitude. We also introduce a new visualization technique, what we term 'dual-execution scatter plot' that highlights infrequent, long-running daemons, differentiating them from frequent and/or short-running daemons. Our empirical results show that the proposed protocol successfully achieves three major aspects-precision, accuracy, and scalability-in execution time measurement that can work for open-source and proprietary software. Figure 2. Execution time measurements of an 8-s compute-bound process. This protocol uses the novel device of a dual-execution scatter plot to highlight what we term L-samples, to identify within those samples infrequent, long-running daemons, and thus to determine in a disciplined way cutoffs to remove samples with such daemon executions. We examine how this protocol applies to proprietary as well as open-source programs.Figure 3. Process time versus elapsed time.We evaluate the performance of EMP by rigorous experiments, starting from a simple program in pure-computation mode to a popular CPU-bound benchmark suite, the SPEC benchmark [2]. Our empirical results strongly support the effectiveness and scalability of EMP.The following section discusses the accuracy and precision in execution time and then describes the timing mechanisms in Linux and their limitations. In Section 3, we introduce our EMP. Section 4 explicates factors and presents experimental results detailing the successive refinement of this protocol. The evaluation of the protocol continues with more realistic scenarios. We then review existing literature over the last 30+ years related to execution time measurement. We conclude by discussing future work, including an intriguing phenomenon that our refined protocol uncovered. BACKGROUNDThis section describes an overall background of timing a program on Linux. Specifically, we clarify accuracy and precision in timing and discuss the Linux timing mechanism and some limitations. Accuracy and precision in timingThe concepts of accuracy and precision in time measurement should be carefully differentiated. The accuracy of any measurement is the "closeness of agreement between a measured quantity value and a true quantity value...

Reliable benchmarking: requirements and solutions

“…Using cgroups and namespaces directly minimizes the necessary version requirements, the installa-tion effort, and the necessary access rights, and makes it easy to use a fallback implementation without containers but still with reliable resource measurements due to cgroups if necessary. 30 For example, using Docker would require to give root-level access for the benchmarking machine to all users who want to execute benchmarks. 31 All features of Docker that are necessary for reliable benchmarking (such as process isolation) are available in BenchExec as well and are implemented in a way that after a one-time setup, they only need a few specific privileges that can be granted individually and without a security risk.…”

“…Several strategies have been proposed for dealing with nondeterministic effects that influence measurements by executing runs several times. The execution-time-measurement protocol (EMP) [30] was designed for improving the precision of time measurements by using a sequence of repeated executions and measurements, for example for avoiding the influence of background processes of the operating system. Furthermore, the authors of EMP also identified other sources of measurement errors that the administrator of the benchmarking machine should address.…”