SimBench: A portable benchmarking methodology for full-system simulators

Abstract

Abstract-Full-system simulators are increasingly finding their way into the consumer space for the purposes of backwards compatibility and hardware emulation (e.g. for games consoles). For such compute-intensive applications simulation performance is paramount. In this paper we argue that existing benchmark suites such as SPEC CPU2006, originally designed for architecture and compiler performance evaluation, are not well suited for the identification of performance bottlenecks in full-system simulators. While their large, complex workloads provide an indication as to the performance of the simulator on 'real-world' workloads, this does not give any indication of why a particular simulator might run an application faster or slower than another. In this paper we present SimBench, an extensive suite of targeted micro-benchmarks designed to run bare-metal on a fullsystem simulator. SimBench exercises dynamic binary translation (DBT) performance, interrupt and exception handling, memory access performance, I/O and other performance-sensitive areas. SimBench is cross-platform benchmarking framework and can be retargeted to new architectures with minimal effort. For several simulators, including QEMU, Gem5 and SimIt-ARM, and targeting ARM and Intel x86 architectures, we demonstrate that SimBench is capable of accurately pinpointing and explaining real-world performance anomalies, which are largely obfuscated by existing application-oriented benchmarks.

A Retargetable System-level DBT Hypervisor

“…In fact, the extra effort we put into compilation ensures that our code quality surpasses that of Qemu's, as will be demonstrated in Section 3.6. Figure 19 shows that indeed, when using the SimBench micro-benchmark suite [47], the Large-Blocks and Small-Blocks benchmark indicate that our code generation speed is 65% and 85% slower, respectively. These benchmarks are described in Section 3.5.…”

“…As well as using the SPEC benchmark suite, we have also evaluated the performance of both Captive and Qemu using SimBench [47]. This is a targeted suite of micro-benchmarks designed to analyze the performance of full system emulation platforms in a number of categories, such as the performance of the memory emulation system, control flow handling, and translation speed (in the case of DBT-based systems).…”

DiagSim

“…Recently, Wagstaff et al [35] proposed a new set of microbenchmarks to mainly evaluate the execution speed of the full-system simulators. This work is orthogonal to ours as we focus on simulator anomalies rather than its execution speed.…”