A Workload Analysis of NSF's Innovative HPC Resources Using XDMoD

Abstract

Workload characterization is an integral part of performance analysis of high performance computing (HPC) systems. An understanding of workload properties sheds light on resource utilization and can be used to inform performance optimization both at the software and system configuration levels. It can provide information on how computational science usage modalities are changing that could potentially aid holistic capacity planning for the wider HPC ecosystem. Here, we report on the results of a detailed workload analysis of the portfolio of supercomputers comprising the NSF Innovative HPC program in order to characterize its past and current workload and look for trends to understand the nature of how the broad portfolio of computational science research is being supported and how it is changing over time. The workload analysis also sought to illustrate a wide variety of usage patterns and performance requirements for jobs running on these systems. File system performance, memory utilization and the types of parallelism employed by users (MPI, threads, etc) were also studied for all systems for which job level performance data was available.Unless stated otherwise, the analysis covered the date range 2011-07-01 to 2017-09-30, the start date of which coincides with that of the XSEDE program. In addition to job accounting data for almost all of the production systems during this time period, job level performance data was available for TACC RANGER, TACC LONESTAR4, TACC STAMPEDE, TACC STAMPEDE2, CCT LSU SUPERMIC, NICS DARTER, SDSC COMET, and SDSC GORDON, as described in Appendix A.2. Highlights of the analysis are as follows. Trends in Utilization• Overall, the allocation utilization of NSF Innovative HPC resources is high, with only 10% of allocations going unused by researchers.• Since 2011, utilization over most NSF directorates in terms of XD SUs has increased by two orders of magnitude.• The Mathematical and Physical Sciences (MPS) and Biological Sciences (BIO) Directorates account for about 70% of XD SUs consumed and this percentage has remained constant over time.• Molecular Biosciences, Physics, and Materials Research account for half of all XD SUs consumed by Parent Science.• Behavioral and Neural Sciences and Integrative Biology and Neuroscience account for over 50% of all jobs run, with the bulk of those on Open Science Grid.• 27% of XSEDE projects are responsible for 73% of the utilization. Job Characteristics• Average job size decreased from a high of about 9000 cores in 2011 to about 1000 cores in 2017 (prior to the introduction of TACC STAMPEDE2). Several factors contributed to this decrease including, the retirement of NICS KRAKEN, the availability of Blue Waters for capability class computing, improved core performance, and resource policies limiting the maximum core count.

Catch Me If You Can: Using Power Analysis to Identify HPC Activity

“…Specifically, while their approach requires a single power meter per node, our approach can identify programs running on an entire rack using only one power sensor. Previous studies indicate that single node jobs compose somewhere between 2% [17] and 20% of the workloads [39].…”

“…We note that while the corpus appears small, in many HPC environments, it is usually the case that a small number of applications (between 10-50) compose the majority of the workload (between 60-80%) [17,39,3]. While our corpus is composed of benchmark codes, these codes implement common algorithms as characterized by Colella's seven dwarfs and are highly representative of HPC workloads [17].…”

“…Specifically, our approach can only identify the programs the algorithm was trained on. As discussed earlier, we do not believe this to be a critical limitation of our approach considering the homogeneous nature of most HPC workloads (i.e., few applications consume majority of cycles [17,39,3]).…”