AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking

Gao, Wanling; Luo, Chunjie; Wang, Lei; Xiong, Xingwang; Chen, Jianan; Hao, Tianshu; Jiang, Zihan; Fan, Fanda; Du, Mengjia; Huang, Yunyou; Zhang, Fan; Xu, Wen; Zheng, Chen; He, Xiwen; Dai, Jiahui; Ye, Hainan; Cao, Zheng; Jia, Zhen; Zhan, Kent; Tang, Haoning; Zheng, Daoyi; Xie, Bin; Li, Wei; Wang, Xiaoyu; Zhan, Jianfeng

doi:10.1007/978-3-030-32813-9_1

Cited by 31 publications

(12 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We choose AIBench Training [8,9]-the most comprehensive AI benchmark by far-as the starting point for the design and implementation of HPC AI benchmarks. The experimental results of AIBench Training [8] have demonstrated that the seventeen AI tasks are diverse in terms of model complexity, computational cost, convergent rate, and microarchitecture characteristics covering most typical AI scenarios.…”

Section: How To Achieve Representativementioning

confidence: 99%

“…For example, several state-of-the-practice HPC AI systems [3,5] are built to tackle enormous AI challenges. The benchmark accelerates the process [6,7], as it provides not only design inputs but also evaluation and optimization metrics and methodology [7][8][9]. However, there are several challenges in benchmarking HPC AI systems.…”

Section: Introductionmentioning

confidence: 99%

“…In the AI domain, there are massive AI tasks and models with different performance metrics. For example, by far, the most comprehensive and representative AI benchmark suite-AIBench [8,9] contains seventeen AI tasks. It is not affordable to implement so many massive benchmarks and further perform benchmarking at scale.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

Jiang,

Gao,

Tang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recent years witness a trend of applying large-scale distributed deep learning algorithms (HPC AI) in both business and scientific computing areas, whose goal is to speed up the training time to achieve a state-of-the-art quality. The HPC AI benchmarks accelerate the process. Unfortunately, benchmarking HPC AI systems at scale raises serious challenges. This paper presents a representative, repeatable and simple HPC AI benchmarking methodology. Among the seventeen AI workloads of AIBench Training-by far the most comprehensive AI Training benchmarks suite-we choose two representative and repeatable AI workloads. The selected HPC AI benchmarks include both business and scientific computing: Image Classification and Extreme Weather Analytics. To rank HPC AI systems, we present a new metric named Valid FLOPS, emphasizing both throughput performance and a target quality. The specification, source code, datasets, and HPC AI500 ranking numbers are publicly available from https: //www.benchcouncil.org/HPCAI500/.

show abstract

Section: How To Achieve Representativementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

Jiang,

Gao,

Tang

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The BenchCouncil AI benchmark suites (2018) present a series of AI benchmarking work, including AIBench [10,11,14,15] for datacenter AI benchmarking, AIoTBench [82] for mobile and embedded device intelligence benchmarking, Edge AIBench [83] for edge computing benchmarking, and the previous version of HPC AI500 [45]. The BenchCouncil AI benchmarks are by far the most comprehensive AI benchmark suites covering datacenter, IoT, edge, and HPC.…”

Section: Mixed Precision Trainingmentioning

confidence: 99%

“…In the AI domain, there are massive AI tasks and models with different performance metrics. For example, by far the most comprehensive and representative AI benchmark suite-AIBench [10,11,14,15] contains seventeen AI tasks. It is not affordable to implement so many massive benchmarks and further perform benchmarking at scale.…”

Section: Introductionmentioning

confidence: 99%

HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

Jiang,

Wang,

Xiong

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

The recent years witness a trend of applying large-scale distributed deep learning algorithms in both business and scientific computing areas, whose goal is to speed up the training time to achieve a stateof-the-art quality. The HPC community feels a great interest in building the HPC AI systems that are dedicated to running those workloads. The HPC AI benchmarks accelerate the process. Unfortunately, benchmarking HPC AI systems at scale raises serious challenges. None of previous HPC AI benchmarks achieve the goal of being equivalent, relevant, representative, affordable, and repeatable.This paper presents a comprehensive methodology, tools, Roofline performance models, and innovative metrics for benchmarking, optimizing, and ranking HPC AI systems, which we call HPC AI500 V2.0. We abstract the HPC AI system into nine independent layers, and present explicit benchmarking rules and procedures to assure equivalence of each layer, repeatability, and replicability. On the basis of AIBench-by far the most comprehensive AI benchmarks suite, we present and build two HPC AI benchmarks from both business and scientific computing: Image Classification, and Extreme Weather Analytics, achieving both representativeness and affordability. To ranking the performance and energy-efficiency of HPC AI systems, we propose Valid FLOPS, and Valid FLOPS per watt, which impose a penalty on failing to achieve the target quality. We propose using convolution and GEMM-the two most intensively-used kernel functions of AIBench to measure the upper bound performance of the HPC AI systems, and present HPC AI roofline models for guiding performance optimizations. The evaluations show our methodology, benchmarks, performance models, and metrics can measure, optimize, and rank the HPC AI systems in a scalable, simple, and affordable way. The specification, source code, datasets, and benchmarking data are publicly available from http://www.benchcouncil.org/benchhub/hpc-ai500-benchmark .

show abstract

HPC AI500: A Benchmark Suite for HPC AI Systems

Jiang

Gao

Wang

et al. 2019

Benchmarking, Measuring, and Optimizing

Self Cite

View full text Add to dashboard Cite

In recent years, with the trend of applying deep learning (DL) in high performance scientific computing, the unique characteristics of emerging DL workloads in HPC raise great challenges in designing, implementing HPC AI systems. The community needs a new yard stick for evaluating the future HPC systems. In this paper, we propose HPC AI500 -a benchmark suite for evaluating HPC systems that running scientific DL workloads. Covering the most representative scientific fields, each workload from HPC AI500 is based on real-world scientific DL applications. Currently, we choose 14 scientific DL benchmarks from perspectives of application scenarios, data sets, and software stack. We propose a set of metrics for comprehensively evaluating the HPC AI systems, considering both accuracy, performance as well as power and cost. We provide a scalable reference implementation of HPC AI500. The specification and source code are publicly available from http://www.benchcouncil.org/HPCAI500/index.html. Meanwhile, the AI benchmark suites for datacenter, IoT, Edge are also released on the BenchCouncil web site.

show abstract

AIBench: Towards Scalable and Comprehensive Datacenter AI Benchmarking

Cited by 31 publications

References 22 publications

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

HPC AI500: A Benchmark Suite for HPC AI Systems

Contact Info

Product

Resources

About