ECP Software Technology Capability Assessment Report

Heroux, Michael A.; Computing, Exascale; Carter, Jonathan; Thakur, Rajeev; Vetter, Jeffrey S.; McInnes, Lois Curfman; Ahrens, James; Neely, J R

doi:10.2172/1463232

Cited by 7 publications

(3 citation statements)

References 100 publications

(119 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The h5bench suite is available at https://github.com/hpc‐io/h5bench with a BSD‐like license that allows external contributions as well as using the code publicly. It can also be easily installed with Spack (spack install h5bench), and it is available as part of the Extreme‐scale Scientific Software Stack (E4S) 21 deployed at multiple supercomputing facilities (e.g., NERSC, JLSE, OLCF). In future, we plan to support additional features (e.g., compression, unified performance report, breakdown of data versus metadata) and provide a unified reporting API and score to follow results and track changes in exascale system's performance.…”

Section: Discussionmentioning

confidence: 99%

“…To facilitate its usage on diverse systems and software stacks, h5bench is also available as a Spack (i.e., spack install h5bench) recipe with multiple variants to allow a fine grain control of which parts of the benchmark suite are compiled. h5bench is also part of the Extreme‐scale Scientific Software Stack (E4S) 21 community effort to provide open‐source software packages for developing, deploying, and running scientific applications on high‐performance computing platforms.…”

Section: The H5bench Benchmarking Suitementioning

confidence: 99%

See 1 more Smart Citation

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Bez,

Tang,

Breitenfeld

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryParallel I/O is a critical technique for moving data between compute and storage subsystems of supercomputers. With massive amounts of data produced or consumed by compute nodes, high‐performant parallel I/O is essential. I/O benchmarks play an important role in this process; however, there is a scarcity of I/O benchmarks representative of current workloads on HPC systems. Toward creating representative I/O kernels from real‐world applications, we have created h5bench , a set of I/O kernels that exercise hierarchical data format version 5 (HDF5) I/O on parallel file systems in numerous dimensions. Our focus on HDF5 is due to the parallel I/O library's heavy usage in various scientific applications running on supercomputing systems. The various tests benchmarked in the h5bench suite include I/O operations (read and write), data locality (arrays of basic data types and arrays of structures), array dimensionality (one‐dimensional arrays, two‐dimensional meshes, three‐dimensional cubes), I/O modes (synchronous and asynchronous). In this paper, we present the observed performance of h5bench executed along several of these dimensions on existing supercomputers (Cori and Summit) and pre‐exascale platforms (Perlmutter, Theta, and Polaris). h5bench measurements can be used to identify performance bottlenecks and their root causes and evaluate I/O optimizations. As the I/O patterns of h5bench are diverse and capture the I/O behaviors of various HPC applications, this study will be helpful to the broader supercomputing and I/O community.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: The H5bench Benchmarking Suitementioning

confidence: 99%

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Bez,

Tang,

Breitenfeld

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…This has been a sustainable model owing to vendor and community support, especially for C++ and Python as rapidly evolving general-purpose languages. The HPC software stacks funded by the US Department of Energy's (DOE's) Exascale Computing Project (ECP) (Heroux et al 2018;Heroux 2019;Dongarra et al 2011) have continued to build upon the legacy of Fortran, C, and C++, and Python's highproductivity ecosystem has been widely adopted for data analysis, AI, and workflow composition (Straßel et al 2020). Ousterhout (1998) already observed the split of programming languages into two distinct groups: implementation and scripting.…”

Section: Introductionmentioning

confidence: 99%

Bridging HPC Communities through the Julia Programming Language

Churavy¹,

Godoy²,

Bauer³

et al. 2022

Preprint

View full text Add to dashboard Cite

The Julia programming language has evolved into a modern alternative to fill existing gaps in scientific computing and data science applications. Julia leverages a unified and coordinated single-language and ecosystem paradigm and has a proven track record of achieving high performance without sacrificing user productivity. These aspects make Julia a viable alternative to high-performance computing's (HPC's) existing and increasingly costly many-body workflow composition strategy in which traditional HPC languages (e.g., Fortran, C, C++) are used for simulations, and higher-level languages (e.g., Python, R, MATLAB) are used for data analysis and interactive computing. Julia's rapid growth in language capabilities, package ecosystem, and community make it a promising universal language for HPC. This paper presents the views of a multidisciplinary group of researchers from academia, government, and industry that advocate for an HPC software development paradigm that emphasizes developer productivity, workflow portability, and low barriers for entry. We believe that the Julia programming language, its ecosystem, and its community provide modern and powerful capabilities that enable this group's objectives. Crucially, we believe that Julia can provide a feasible and less costly approach to programming scientific applications and workflows that target HPC facilities. In this work, we examine the current practice and role of Julia as a common, end-to-end programming model to address major challenges in scientific reproducibility, data-driven AI/machine learning, co-design and workflows, scalability and performance portability in heterogeneous computing, network communication, data management, and community education. As a result, the diversification of current investments to fulfill the needs of the upcoming decade is crucial as more supercomputing centers prepare for the exascale era.

show abstract

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Zhuang

Jacob

Lin

et al. 2020

J Adv Model Earth Syst

View full text Add to dashboard Cite

Cloud computing platforms can facilitate the use of Earth science models by providing immediate access to fully configured software, massive computing power, and large input data sets. However, slow internode communication performance has previously discouraged the use of cloud platforms for massively parallel simulations. Here we show that recent advances in the network performance on the Amazon Web Services cloud enable efficient model simulations with over a thousand cores. The choices of Message Passing Interface library configuration and internode communication protocol are critical to this success. Application to the Goddard Earth Observing System (GEOS)-Chem global 3-D chemical transport model at 50-km horizontal resolution shows efficient scaling up to at least 1,152 cores, with performance and cost comparable to the National Aeronautics and Space Administration Pleiades supercomputing cluster.Plain Language Summary Earth science model simulations are computationally expensive, typically requiring the use of high-end supercomputing clusters that are managed by universities or national laboratories. Commercial cloud computing offers an alternative. However, past work found that cloud computing platforms were not efficient for large-scale simulations on over 100 CPU cores, because the network communication performance on the cloud was slow compared to local clusters. Here we show that recent advances in the cloud network performance enable efficient model simulations with over a thousand cores, and cloud platforms can now serve as a viable alternative to local clusters for simulations at large scale. Computing on the cloud has extensive advantages, such as providing immediate access to fully configured model code and large data sets for any users, allowing full reproducibility of model simulation results, offering quick access to novel hardware that might not be available on local clusters, and being able to scale to virtually unlimited amounts of compute and storage resources. Those benefits will help advance Earth science modeling research.

show abstract

ECP Software Technology Capability Assessment Report

Cited by 7 publications

References 100 publications

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

h5bench: A unified benchmark suite for evaluating HDF5 I/O performance on pre‐exascale platforms

Bridging HPC Communities through the Julia Programming Language

Enabling High‐Performance Cloud Computing for Earth Science Modeling on Over a Thousand Cores: Application to the GEOS‐Chem Atmospheric Chemistry Model

Contact Info

Product

Resources

About