Hierarchical Roofline Performance Analysis for Deep Learning Applications

Yang, Charlene; Wang, Yunsong; Kurth, Thorsten; Farrell, S.; Williams, Samuel

doi:10.1007/978-3-030-80126-7_35

Cited by 10 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…80 × 8 × 1.312 × 4 3 × 2 = 107.479 TFLOP/s. ERT was recently updated with more precision and architecture support, and the implementation details are discussed in [35]. Note that the machine balance diagonal presented in this paper is the Tensor Core peak performance divided by the HBM bandwidth (107479 / 828.8 = 129.68).…”

Section: B Roofline Data Collectionmentioning

confidence: 99%

See 1 more Smart Citation

Time-Based Roofline for Deep Learning Performance Analysis

Wang

Yang

Farrell

et al. 2020

2020 IEEE/ACM Fourth Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

Deep learning applications are usually very compute-intensive and require a long run time for training and inference. This has been tackled by researchers from both hardware and software sides, and in this paper, we propose a Roofline-based approach to performance analysis to facilitate the optimization of these applications. This approach is an extension of the Roofline model widely used in traditional highperformance computing applications, and it incorporates both compute/bandwidth complexity and run time in its formulae to provide insights into deep learning-specific characteristics. We take two sets of representative kernels, 2D convolution and long short-term memory, to validate and demonstrate the use of this new approach, and investigate how arithmetic intensity, cache locality, auto-tuning, kernel launch overhead, and Tensor Core usage can affect performance. Compared to the common ad-hoc approach, this study helps form a more systematic way to analyze code performance and identify optimization opportunities for deep learning applications.

show abstract

Section: B Roofline Data Collectionmentioning

confidence: 99%

“…A set of metrics are collected to measure the kernel run time, computational complexity, and bandwidth complexity (please refer to [35], [24] for more details). Throughout this paper, computational complexities are treated equally thus precisionagnostic.…”

Section: B Roofline Data Collectionmentioning

confidence: 99%

Time-Based Roofline for Deep Learning Performance Analysis

Wang

Yang

Farrell

et al. 2020

2020 IEEE/ACM Fourth Workshop on Deep Learning on Supercomputers (DLS)

Self Cite

View full text Add to dashboard Cite

show abstract

“…On NVIDIA GPUs, an nvprof [9] based methodology was first proposed in [17], then an Nsight Compute [5] metrics based one developed in [22], [30]. These methodologies require a dozen of metrics to be collected for hierarchical Roofline analysis, and could incur significant profiling overhead when the number of kernels in the code is high.…”

Section: B Roofline Data Collection On Nvidia Gpusmentioning

confidence: 99%

“…To facilitate the Roofline study, a range of other tools have sprung to life as well, for example, the Empirical Roofline Toolkit (ERT) for more accurate machine characterization [12], [13], and [14], [15], [16], [17], [18] for more streamlined data collection methods. Other than tools development, there are many studies on the application of the Roofline model in traditional HPC [19], [17], [18], [20], [21] and Machine Learning [17], [18], [22], [23], and extension and refinement of the model to related topics in HPC, such as instruction Roofline [24], time-based Roofline [23], Roofline scaling trajectory [25], performance portability analysis based on Roofline [13], and power and energy Roofline [26], [27].…”

Section: Introductionmentioning

confidence: 99%

Hierarchical Roofline Analysis: How to Collect Data using Performance Tools on Intel CPUs and NVIDIA GPUs

Yang

2020

Preprint

Self Cite

View full text Add to dashboard Cite

This paper surveys a range of methods to collect necessary performance data on Intel CPUs and NVIDIA GPUs for hierarchical Roofline analysis. As of mid-2020, two vendor performance tools, Intel Advisor and NVIDIA Nsight Compute, have integrated Roofline analysis into their supported feature set. This paper fills the gap for when these tools are not available, or when users would like a more customized workflow for certain analysis. Specifically, we will discuss how to use Intel Advisor, RRZE LIKWID, Intel SDE and Intel Amplifier on Intel architectures, and nvprof, Nsight Compute metrics, and Nsight Compute section files on NVIDIA architectures. These tools will be used to collect information for as many memory/cache levels in the memory hierarchy as possible in order to provide insights into application's data reuse and cache locality characteristics.

show abstract

“…To facilitate the Roofline study, a range of tools have sprung to life, for more accurate machine characterization such as the Empirical Roofline Toolkit (ERT) [7], [8], and for more streamlined methods to collect Roofline performance data using open-source tools or workflows [3], [9]- [11]. Other than tools development, there are also many studies on the application of the Roofline model in both traditional HPC [3], [12]- [14] and the new, emerging field of Machine Learning [3], [15], [16], and the extension and refinement of the model, such as instruction Roofline [17], Roofline scaling trajectories [18], performance portability based on Roofline [8], and power and energy Roofline [19], [20].…”

Section: Introductionmentioning

confidence: 99%

8 Steps to 3.7 TFLOP/s on NVIDIA V100 GPU: Roofline Analysis and Other Tricks

Yang

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Performance optimization can be a daunting task especially as the hardware architecture becomes more and more complex. This paper takes a kernel from the Materials Science code BerkeleyGW, and demonstrates a few performance analysis and optimization techniques. Despite challenges such as high register usage, low occupancy, complex data access patterns, and the existence of several long-latency instructions, we have achieved 3.7 TFLOP/s of double-precision performance on an NVIDIA V100 GPU, with 8 optimization steps. This is 55% of the theoretical peak, 6.7 TFLOP/s, at nominal frequency 1312 MHz, and 70% of the more customized peak based on our 58% FMA ratio, 5.3 TFLOP/s. An array of techniques used to analyze this OpenACC kernel and optimize its performance are shown, including the use of hierarchical Roofline performance model and the performance tool Nsight Compute. This kernel exhibits computational characteristics that are commonly seen in many high-performance computing (HPC) applications, and are expected to be very helpful to a general audience of HPC developers and computational scientists, as they pursue more performance on NVIDIA GPUs.

show abstract

Hierarchical Roofline Performance Analysis for Deep Learning Applications

Cited by 10 publications

References 17 publications

Time-Based Roofline for Deep Learning Performance Analysis

Time-Based Roofline for Deep Learning Performance Analysis

Hierarchical Roofline Analysis: How to Collect Data using Performance Tools on Intel CPUs and NVIDIA GPUs

8 Steps to 3.7 TFLOP/s on NVIDIA V100 GPU: Roofline Analysis and Other Tricks

Contact Info

Product

Resources

About