The effect of compiler-flag tuning on SPEC benchmark performance

Abstract

The SPEC CINT92 and CFP92 benchmark suites are application-based system benchmarks primarily intended for workstation-class system performance measurements. The SPEC CPU benchmark results are widely disseminated by system vendors and as such have become the de-facto standard for comparing system performance. Recently, many observers have expressed concerns about the suitability of published SPEC benchmark results in representing application performance on typical systems. The most outspoken concern is that there is too much freedom permitted in the manipulation of compiler flags. This has resulted in revisions to the SPEC reporting procedure.This paper presents and discusses many of the issues concerning the tuning of benchmarks through manipulation of compiler flags. We attempt to quantify the impact of these procedures through controlled experiments. Baseline performance results, using a set of uniform, common optimizations are compared to published data. Further experiments measure the performance of the SPEC benchmarks in the other common usage scenarios. These are a centralized file storage configuration and a system using common binaries among several implementations of the same architecture. Despite the great concern over the use of compiler flags in the SPEC community, our experiments show only a modest impact on performance. The more significant performance differential shown in the other experiments draws into question the utility of current SPEC data to many users.

Using SPEC CPU2006 to evaluate the sequential and parallel code generated by commercial and open-source compilers

“…This makes the search for adequate flags a trial-and-error process. Moreover, as Chan et al pointed out in 1994 [10], it is difficult to ensure that none of the flags chosen has been added to the compiler just to optimize some SPEC program, a situation not allowed by any SPEC benchmark release. Besides this, the base speed metric forces to use the same flags to compile the entire benchmark.…”

Benchmarking

“…This soon prompted the question whether such excessive optimizations are representative for real programming, or whether it would be better to "burn all flags" [13]. The issue was later discussed in academic publications [1,8] and in the trade press. After a considerable debate, the final compromise for SPEC, established in January 1994, was that two metrics were established, "baseline" and "peak": Every SPEC CPU benchmark measurement has to measure performance with a restricted "baseline" set of flags (metric name: SPECint_base2000 or similar), and optionally with a more extended "peak" set of flags.…”

Compile-time support for thread-level speculation

“…This makes the search for adequate flags a trial-and-error process. Moreover, as Chan et al pointed out in 1994 [32], it is difficult to ensure that none of the flags chosen is offered by the compiler just to optimize some SPEC program, a situation not allowed by any released SPEC benchmark. Besides this, the base speed metric forces to use the same flags to compile the entire benchmark.…”