HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

Abstract

The recent years witness a trend of applying large-scale distributed deep learning algorithms in both business and scientific computing areas, whose goal is to speed up the training time to achieve a stateof-the-art quality. The HPC community feels a great interest in building the HPC AI systems that are dedicated to running those workloads. The HPC AI benchmarks accelerate the process. Unfortunately, benchmarking HPC AI systems at scale raises serious challenges. None of previous HPC AI benchmarks achieve the goal of being equivalent, relevant, representative, affordable, and repeatable.This paper presents a comprehensive methodology, tools, Roofline performance models, and innovative metrics for benchmarking, optimizing, and ranking HPC AI systems, which we call HPC AI500 V2.0. We abstract the HPC AI system into nine independent layers, and present explicit benchmarking rules and procedures to assure equivalence of each layer, repeatability, and replicability. On the basis of AIBench-by far the most comprehensive AI benchmarks suite, we present and build two HPC AI benchmarks from both business and scientific computing: Image Classification, and Extreme Weather Analytics, achieving both representativeness and affordability. To ranking the performance and energy-efficiency of HPC AI systems, we propose Valid FLOPS, and Valid FLOPS per watt, which impose a penalty on failing to achieve the target quality. We propose using convolution and GEMM-the two most intensively-used kernel functions of AIBench to measure the upper bound performance of the HPC AI systems, and present HPC AI roofline models for guiding performance optimizations. The evaluations show our methodology, benchmarks, performance models, and metrics can measure, optimize, and rank the HPC AI systems in a scalable, simple, and affordable way. The specification, source code, datasets, and benchmarking data are publicly available from http://www.benchcouncil.org/benchhub/hpc-ai500-benchmark .

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

“…The specification, source code, and HPC AI500 ranking numbers are publicly available from http://www.benchcouncil.org/benchhub/hpc-ai 500-benchmark. A full technical report is available from [39].…”

A BenchCouncil View on Benchmarking Emerging and Future Computing

“…The algorithm and hardware implementation, like different precision, e.g., single-precision, double-precision or mixed precision, impact the learning dynamics. Even for the same system with different scales, the interactions among system size and minibatch size significantly impact the measured quantity values like time-to-quality -the training time to achieve the state-of-the-quality -and FLOPS (the computation overhead) [25,20,47,31,38].…”