MLPerf™ HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems

Farrell, S.; Emani, Murali; Balma, Jacob; Drescher, Lukas; Drozd, Aleksandr; Fink, Andreas; Fox, Geoffrey; Kanter, David; Kurth, Thorsten; Mattson, Peter; Mu, Dawei; Ruhela, Amit; Sato, Kento; Sairyo, Koichi; Tabaru, Tsuguchika; Tsaris, Aristeidis; Balewski, J.; Cumming, Ben; Danjo, Takumi; Domke, Jens; Fukai, Takaaki; Fukumoto, Naoto; Fukushi, Tatsuya; Gerofi, Balazs; Honda, Tomohiro; Imamura, Toshiyuki; Kasagi, Akihiko; Kawakami, Kentaro; Kudo, Shuhei; Kuroda, Akiyoshi; Martinasso, Maxime; Matsuoka, Satoshi; Mendonça, Henrique; Minami, Kazuki; Ram, Prabhat; Sawada, Takashi; Shankar, Mallikarjun; John, Tom St.; Tabuchi, Akihiro; Vishwanath, Venkatram; Wahib, Mohamed; Yamazaki, Masami; Yin, Junqi

doi:10.1109/mlhpc54614.2021.00009

Cited by 15 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data parallelism techniques are widely utilized for training on large datasets [9,13,21,26]. When applying data parallelism on HPC clusters, the training typically includes three stages: (1) I/O: loading the data from a remote parallel file system (i.e., GPFS, Lustre) to host memory; (2) Computation: performing forward and backward phases to calculate the local gradient on each device; (3) Communication: synchronizing averaged gradients across multiple devices to update model weights.…”

Section: Distributed Training With Data Parallelismmentioning

confidence: 99%

“…We conduct the benchmarking on ThetaGPU [2] (each node has eight A100 GPUs) at the Argonne Leadership Computing Facility, following the guidelines from MLPerf HPC [13].…”

Section: Research Motivationmentioning

confidence: 99%

“…Furthermore, we see the potential of this optimization to be applied to distributed training tasks that do not require full randomization. For example, MLPerfHPC performs intra-node shuffle for DeepCAM [13].…”

Section: Evaluation Of Each Optimizationmentioning

confidence: 99%

See 2 more Smart Citations

SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates

Sun¹,

Yu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

CNN-based surrogates have become prevalent in scientific applications to replace conventional time-consuming physical approaches. Although these surrogates can yield satisfactory results with significantly lower computation costs over small training datasets, our benchmarking results show that data-loading overhead becomes the major performance bottleneck when training surrogates with large datasets. In practice, surrogates are usually trained with highresolution scientific data, which can easily reach the terabyte scale. Several state-of-the-art data loaders are proposed to improve the loading throughput in general CNN training; however, they are sub-optimal when applied to the surrogate training. In this work, we propose SOLAR, a surrogate data loader, that can ultimately increase loading throughput during the training. It leverages our three key observations during the benchmarking and contains three novel designs. Specifically, SOLAR first generates a pre-determined shuffled index list and accordingly optimizes the global access order and the buffer eviction scheme to maximize the data reuse and the buffer hit rate. It then proposes a tradeoff between lightweight computational imbalance and heavyweight loading workload imbalance to speed up the overall training. It finally optimizes its data access pattern with HDF5 to achieve a better parallel I/O throughput. Our evaluation with three scientific surrogates and 32 GPUs illustrates that SOLAR can achieve up to 24.4× speedup over PyTorch Data Loader and 3.52× speedup over state-of-the-art data loaders.

show abstract

Section: Distributed Training With Data Parallelismmentioning

confidence: 99%

“…We conduct the benchmarking on ThetaGPU [2] (each node has eight A100 GPUs) at the Argonne Leadership Computing Facility, following the guidelines from MLPerf HPC [13].…”

Section: Research Motivationmentioning

confidence: 99%

See 1 more Smart Citation

SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates

Sun¹,

Yu²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…General ML Workload Benchmarks. MLPerf Inference (Reddi et al, 2020) is a set of industry standard, singlekernel ML benchmarks that span the ML landscape, from high performance computers (Farrell et al, 2021) to tiny embedded systems (Banbury et al, 2021). It also provides a rich set of inference scenarios based on realistic use cases from industry: single-stream (single inference), multistream (repeated inference with a time interval), server (random inference request modeled via Poisson distribution), and offline (batch processing).…”

Section: Detailed Related Work Comparisonmentioning

confidence: 99%

“…ML and XR Benchmarks. Compared to the abovementioned benchmarks (Dee, 2016;EEM, 2020;VRM, 2020;Banbury et al, 2021;Farrell et al, 2021;Gao et al, 2019;Huzaifa et al, 2021;Ignatov et al, 2018;Luo et al, 2018;Reddi et al, 2020), XRBENCH covers all requirement of an ML-based XR workloads. To be specific, XRBENCH provides diverse cascon-MMMT scenarios with real-time requirement and complex dependencies, which majority ML benchmarks are missing.…”

Section: Detailed Related Work Comparisonmentioning

confidence: 99%

XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse

Kwon¹,

Nair²,

Seo³

et al. 2022

Preprint

View full text Add to dashboard Cite

Real-time multi-model multi-task (MMMT) workloads, a new form of deep learning inference workloads, are emerging for applications areas like extended reality (XR) to support metaverse use cases. These workloads combine user interactivity with computationally complex machine learning (ML) activities. Compared to standard ML applications, these ML workloads present unique difficulties and constraints. Real-time MMMT workloads impose heterogeneity and concurrency requirements on future ML systems and devices, necessitating the development of new capabilities. This paper begins with a discussion of the various characteristics of these real-time MMMT ML workloads and presents an ontology for evaluating the performance of future ML hardware for XR systems. Next, we present XRBENCH, a collection of MMMT ML tasks, models, and usage scenarios that execute these models in three representative ways: cascaded, concurrent, and cascaded-concurrency for XR use cases. Finally, we emphasize the need for new metrics that capture the requirements properly. We hope that our work will stimulate research and lead to the development of a new generation of ML systems for XR use cases.

show abstract

AI Benchmarking for Science: Efforts from the MLCommons Science Working Group

Thiyagalingam

Laszewski

Yin

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

MLPerf™ HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems

Cited by 15 publications

References 15 publications

SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates

SOLAR: A Highly Optimized Data Loading Framework for Distributed Training of CNN-based Scientific Surrogates

XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse

AI Benchmarking for Science: Efforts from the MLCommons Science Working Group

Contact Info

Product

Resources

About