Truth in advertising: Reporting performance of computer programs, algorithms and the impact of architecture

Abstract

<jats:p>The level of detail and precision that appears in the experimental&#x0D; methodology section computer science papers is usually much less&#x0D; than in natural science disciplines. This is partially justified by&#x0D; different nature of experiments. The experimental evidence presented&#x0D; here shows that the time taken by the same algorithm varies so&#x0D; significantly on different CPUs that without knowing the exact model&#x0D; of CPU, it is difficult to compare the results. This is placed in&#x0D; context by analysing a cross-section of experimental results&#x0D; reported in the literature. The reporting of experimental results&#x0D; is sometimes insufficient to allow experiments to be replicated,&#x0D; and in some case is insufficient to support the claims made for the&#x0D; algorithms. New standards for reporting on algorithms results are&#x0D; suggested.</jats:p>

Concept analysis in programming language research: done well it is all right

“…Here, the reality of interest is the software reality, and knowledge is generated by the means of examining or running programs. The most prominent method here is computational experiments-the study of algorithms by exposing their implementations to a wide variety of automatically generated stimuli and measuring the effort expended by the implementation as a function of stimulus parameters [6,28,29,37,48,67,72]; it is relevant to programming language research mostly in the study of implementation techniques.…”

A microbenchmark case study and lessons learned

“…Benchmarking of computer systems is a notoriously delicate task [9]. The increasingly growing abstraction gap between programs and execution environments, most notably virtual machines, forces benchmarking, which could, in the early days, be carried out by counting program instructions, to use methods used in experimental, exact and social sciences.…”