Comparing Benchmarks Using Key Microarchitecture-Independent Characteristics

Abstract

Abstract-Understanding the behavior of emerging workloads is important for designing next generation microprocessors. For addressing this issue, computer architects and performance analysts build benchmark suites of new application domains and compare the behavioral characteristics of these benchmark suites against well-known benchmark suites. Current practice typically compares workloads based on microarchitecture-dependent characteristics generated from running these workloads on real hardware. There is one pitfall though with comparing benchmarks using microarchitecture-dependent characteristics, namely that completely different inherent program behavior may yield similar microarchitecture-dependent behavior.This paper proposes a methodology for characterizing benchmarks based on microarchitecture-independent characteristics. This methodology minimizes the number of inherent program characteristics that need to be measured by exploiting correlation between program characteristics. In fact, we reduce our 47-dimensional space to an 8-dimensional space without compromising the methodology's ability to compare benchmarks. The important benefits of this methodology are that (i) only a limited number of microarchitecture-independent characteristics need to be measured, and (ii) the resulting workload characterization is easy to interpret. Using this methodology we compare 122 benchmarks from 6 recently proposed benchmark suites. We conclude that some benchmarks in emerging benchmark suites are indeed similar to benchmarks from well-known benchmark suites as suggested through a microarchitecture-dependent characterization. However, other benchmarks are dissimilar based on a microarchitecture-independent characterization although a microarchitecture-dependent characterization suggests the opposite to be true.

Milepost GCC: Machine Learning Enabled Self-tuning Compiler

“…Several researchers [24,58,62] attempted to characterize program input in order to predict best code variant at run-time using several machine learning methods, including automatically generated decision trees and statistical modeling. Other works [50,48,33] used machine learning for performance prediction and hardware-software co-design.…”

“…The Milepost project takes an orthogonal approach based on the observation that similar programs may exhibit similar behavior and require similar optimizations so it is possible to correlate program features and optimizations, thereby predicting good transformations for unseen programs based on previous optimization experience [65,27,72,17,48,26,42]. In the current version of Milepost GCC we use static program features (such as the number of instructions in a method, number of branches, etc) to characterize programs and build predictive models.…”

“…In the current version of Milepost GCC we use static program features (such as the number of instructions in a method, number of branches, etc) to characterize programs and build predictive models. Naturally, since static features may not be enough to capture run-time program behavior, we plan to add plugins to improve program and optimization correlation based on dynamic features (performance counters [26], microarchitecture-independent characteristics [48], reactions to transformations [42] or semantically non-equivalent program modifications [41]). The next section describes the overall framework and is followed by a detailed description of Milepost GCC and the Interactive Compiler Interface.…”

Reducing Training Time in a One-Shot Machine Learning-Based Compiler

“…We use ARM assembly instructions as our features to ensure that our technique is properly capturing information about the program and not some facet peculiar to a particular architecture. Hoste and Eeckhout [9] argue that the use of a generic RISC architecture is capable of representing and characterizing program performance and the ARM is used as an approximation a generic RISC core.…”

ISA-independent workload characterization and its implications for specialized architectures

“…This approach is limited because machine-dependent features such as cache size and pipeline depth will strongly impact the workload characterization. To overcome the problem, microarchitectureindependent workload characterization can be employed by profiling instruction traces to collect information such as working set sizes, register traffic, memory locality, and branch predictability [11]. Although this approach removes the effects of microarchitecture-dependent features, some of these analyses depend on the particular ISA with which the trace is represented.…”