Evaluating Benchmark Subsetting Approaches

Abstract

To reduce the simulation time to a tractable amount or due to compilation (or other related) problems, computer architects often simulate only a subset of the benchmarks in a benchmark suite. However, if the architect chooses a subset of benchmarks that is not representative, the subsequent simulation results will, at best, be misleading or, at worst, yield incorrect conclusions. To address this problem, computer architects have recently proposed several statistically-based approaches to subset a benchmark suite. While some of these approaches are well-grounded statistically, what has not yet been thoroughly evaluated is the: 1) Absolute accuracy, 2) Relative accuracy across a range of processor and memory subsystem enhancements, and 3) Representativeness and coverage of each approach for a range of subset sizes. Specifically, this paper evaluates statistically-based subsetting approaches based on principal components analysis (PCA) and the Plackett and Burman (P&B) design, in addition to prevailing approaches such as integer vs. floating-point, core vs. memory-bound, by language, and at random. Our results show that the two statistically-based approaches, PCA and P&B, have the best absolute and relative accuracy for CPI and energy-delay product (EDP), produce subsets that are the most representative, and choose benchmark and input set pairs that are most well-distributed across the benchmark space. To achieve a 5% absolute CPI and EDP error, across a wide range of configurations, PCA and P&B typically need about 17 benchmark and input set pairs, while the other five approaches often choose more than 30 benchmark and input set pairs.

Characterization of SPEC CPU2006 and SPEC OMP2001: Regression Models and their Transferability

“…[11] applies clustering techniques to architecturally-independent attributes in order to reduce the size of a benchmark suite trace set for simulation. Several studies investigate the application of PCA (Principal Component Analysis) and clustering to the subsetting of benchmarks: [12] focuses on finding similar architectureindependent phases across benchmark-input pairs; [13] analyzes PMU events for SPEC CPU2006 and applies PCA, clustering and other statistical methods to this data to compare and subset the benchmarks; and [14] compares several subsetting approaches and concludes that PCA and Plackett and Burman (P&B) outperform the others. [11] uses SPEC CPU2000 and ICA (Independent Component Analysis) to identify a representative subset of benchmarks for use in simulation studies and demonstrates the improved performance of ICA over PCA and other approaches.…”

An automated framework for characterizing and subsetting GPGPU workloads

“…While previous work has explored dimensionalityreduction techniques such as principal component analysis (PCA) [7], correlation elimination [11], genetic algorithm [11], and Plackett and Burmann (P&B) design technique [10] in combination k-means and hierarchical clustering techniques [7], the combination of PCA and hierarchical clustering has proven to be the most successful [10]. We adopt and apply these best practices in our work.…”

TransPlant: A parameterized methodology for generating transactional memory workloads

“…Q1-1 through Q4-1 show that TransPlant is capable is generating not only outlier programs but also programs with traditional performance characteristics. Further, if a cutoff value is used to choose a subset of programs able to represent the general behavior of all of the benchmarks [32], Q2-1 and Q3-1 are always included.…”