GPGPU Benchmark Suites: How Well Do They Sample the Performance Spectrum?

Abstract

Recently, GPGPUs have positioned themselves in the mainstream processor arena with their potential to perform a massive number of jobs in parallel. At the same time, many GPGPU benchmark suites have been proposed to evaluate the performance of GPGPUs. Both academia and industry have been introducing new sets of benchmarks each year while some already published benchmarks have been updated periodically. However, some benchmark suites contain benchmarks that are duplicates of each other or use the same underlying algorithm. This results in an excess of workloads in the same performance spectrum.In this paper, we provide a methodology to obtain a set of new GPGPU benchmarks that are located in the unexplored region of the performance spectrum. Our proposal uses statistical methods to understand the performance spectrum coverage and uniqueness of existing benchmark suites. Later we show techniques to identify areas that are not explored by existing benchmarks by visually showing the performance spectrum coverage. Finding unique key metrics for future benchmarks to broaden its performance spectrum coverage is also explored using hierarchical clustering and ranking by Hotelling's T 2 method. Finally, key metrics are categorized into GPGPU performance related components to show how future benchmarks can stress each of the categorized metrics to distinguish themselves in the performance spectrum. Our methodology can serve as a performance spectrum oriented guidebook for designing future GPGPU benchmarks.

Machine Learning Techniques for Understanding and Predicting Memory Interference in CPU-GPU Embedded Systems

“…We operated this initial filtering by following the naming convention explained in Section III-B; specifically, we focused on three macro categories: a first category that refers to GPU memory accesses (L2 and DRAM, starts with lts); a second category that refers to metrics related to compute instructions (starts with gr, gpu, smsp inst or cycles) and a final category that summarizes branch divergence (starts with smsp warp). The reason for choosing these categories is because it has been shown to fit into what is known as principal components for GPGPU performance prediction and/or optimization [30], [31].…”

“…all the possible slowdown values, Y ), we plot both how the slowdown changes as we vary the two parameters, keeping fixed at 7 the number of CPU interferents (Figure 4), and the total slowdown distribution among all the generated kernels on all the possible number of interferents (Figure 5). We set p 0 ∈ [1, 10, 20, 30, 40, 50, 75, 100, 125, 150, 175, 200, 250, 300, 350, 400, 500, 1000] and p 1 ∈ [1, 3,7,9,10,12,14,17,23,30]. Considering these steps and ranges and the number of interferents, we generate 180 different versions of PARKERNEL and 1440 individual experiments calculated by varying the number of interferents.…”

Visualization of profiling and tracing in CPU‐GPU programs

“…Therefore, we used the Rodinia benchmark. 29 It correctly represents a variety of workload 45 and is a proven workload to test computing and profiling techniques.…”

A Hybrid Framework for Fast and Accurate GPU Performance Estimation through Source-Level Analysis and Trace-Based Simulation

“…With regards to GPU simulation acceleration, there exist some research that either choose a portion [51] or perform a pre-characterization [52] of target workloads and then derive the execution time from the simulation results. There are also studies that focus on the generation of GPU benchmarks [53] to reveal GPU's performance spectrum, and modeling of GPU memory systems [54]. These studies are supplementary for GPU performance estimation techniques.…”