AIBench Scenario: Scenario-distilling AI Benchmarking

Abstract

Real-world application scenarios like modern Internet services consist of diversity of AI and non-AI modules with very long and complex execution paths. Using component or micro AI benchmarks alone can lead to error-prone conclusions. This paper proposes a scenario-distilling AI benchmarking methodology. Instead of using real-world applications, we propose the permutations of essential AI and non-AI tasks as a scenario-distilling benchmark. We consider scenario-distilling benchmarks, component and micro benchmarks as three indispensable parts of a benchmark suite.Together with seventeen industry partners, we identify nine important real-world application scenarios. We design and implement a highly extensible, configurable, and flexible benchmark framework. On the basis of the framework, we propose the guideline for building scenario-distilling benchmarks, and present two Internet service AI ones.The preliminary evaluation shows the advantage of scenario-distilling AI benchmarking against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from the web site http://www.benchcouncil.org/AIBench/index.html.

HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking

“…Dataset Against other domain AI benchmarks, there are two unique differences in HPC AI benchmarking. First, the challenges of HPC AI benchmarking inherit from the complexity of benchmarking scalable hardware and software systems at scale, i.e., tens of thousands of nodes, significantly different from that of IoT [23] or datacenter [24]. On this point, we need to make the benchmark as simple as possible, which we have discussed in detail before.…”

HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems

“…With respect to other AI benchmarks, there are two unique differences of HPC AI benchmarking. First, the challenges of HPC AI benchmarking inherit from the complexity of benchmarking scalable hardware and software systems at scale, i.e., tens of thousands of nodes, significantly different from that of IoT [43] or datacenter [11]. On this point, we need consider the cost of benchmarking at scale.…”

“…The BenchCouncil AI benchmark suites (2018) present a series of AI benchmarking work, including AIBench [10,11,14,15] for datacenter AI benchmarking, AIoTBench [82] for mobile and embedded device intelligence benchmarking, Edge AIBench [83] for edge computing benchmarking, and the previous version of HPC AI500 [45]. The BenchCouncil AI benchmarks are by far the most comprehensive AI benchmark suites covering datacenter, IoT, edge, and HPC.…”

“…For example, the Summit system [7] is built to tackle huge AI challenges. The benchmark accelerates the process [8,9], as it provides not only design inputs, but also evaluation and optimization metric and methodology [10,11]. However, there are several challenges in benchmarking HPC AI systems.…”

“…In the AI domain, there are massive AI tasks and models with different performance metrics. For example, by far the most comprehensive and representative AI benchmark suite-AIBench [10,11,14,15] contains seventeen AI tasks. It is not affordable to implement so many massive benchmarks and further perform benchmarking at scale.…”