A Case For Intra-rack Resource Disaggregation in HPC

Klenk, Benjamin; Cook, Brandon; Teh, Min Yee; Glick, Madeleine; Dennison, Larry; Bergman, Keren; Shalf, John

doi:10.1145/3514245

Cited by 22 publications

(22 citation statements)

References 64 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other works utilize performance monitoring infrastructure to characterize application and system performance in HPC [6,9,19,20,24,25]. In particular, the paper presented by Ji et al analyzed various application memory usage in terms of object access patterns [6].…”

Section: Related Workmentioning

confidence: 99%

“…Peng et al focused on the memory subsystem and studied the temporal and spatial memory usage in two production HPC systems at LLNL [20]. Michelogiannakis et al [9] performed a detailed analysis of key metrics sampled in NERSC's Cori as a data-driven study of the potential of resource disaggregation in HPC.…”

Section: Related Workmentioning

confidence: 99%

“…Heterogeneous hardware combined with diverse workloads that have a wide range of resource requirements, have made it difficult to achieve efficient resource management; inefficient resource management threatens to underutilize expensive resources that can rapidly increase capital and operating costs. Previous studies have shown that the resources of HPC systems are often under-utilized, especially memory [9,16,21].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Analyzing Resource Utilization in an HPC System: A Case Study of NERSC Perlmutter

Li¹,

Cook²,

Dulanya³

et al. 2023

Preprint

View full text Add to dashboard Cite

The resource demands of HPC applications vary significantly. However, it is common for HPC systems to assign resources on a pernode basis to prevent interference from co-located workloads. This gap between the coarse-grained resource allocation and the varying resource demands can lead to underutilization of HPC resources. In this study, we comprehensively analyzed the resource usage and characteristics of NERSC's Perlmutter, a state-of-the-art HPC system with both CPUonly and GPU-accelerated nodes. Our three-week usage analysis revealed that the majority of jobs had low CPU utilization and that around 86% of both CPU and GPU-enabled jobs used 50% or less of the available host memory. Additionally, 52.1% of GPU-enabled jobs used up to 25% of the GPU memory, and the memory capacity was over-provisioned in some ways for all jobs. The study also found that 60% of GPU-enabled jobs had idle GPUs, which could indicate that resource underutilization may occur as users adapt workflows to a system with new resources. Our research provides valuable insights into performance characterization and offers new perspectives for system operators to understand and track the migration of workloads. Furthermore, it can be extremely useful for designing, optimizing, and procuring HPC systems.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing Resource Utilization in an HPC System: A Case Study of NERSC Perlmutter

Li¹,

Cook²,

Dulanya³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…We evaluate three metrics of memory usage in this section capacity, page access, and bandwidth. While capacity usage has been extensively studied in multiple large-scale studies on leadership clusters [1], [2], [24], the other two metrics on page and bandwidth usage are important for understanding the need for memory resources and optimization opportunities.…”

Section: A Characterization Of Memory Usagementioning

confidence: 99%

“…Such static coarse-grained provisioning simplifies resource management. However, recent studies on large-scale supercomputers indicate that node-level memory utilization can be as low as 15% [1], [2]. One reason is that memory bandwidth and capacity are tightly coupled in current HPC systems.…”

Section: Introductionmentioning

confidence: 99%

Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems

Wahlgren¹,

Gokhale²

2022

Preprint

View full text Add to dashboard Cite

Current HPC systems provide memory resources that are statically configured and tightly coupled with compute nodes. However, workloads on HPC systems are evolving. Diverse workloads lead to a need for configurable memory resources to achieve high performance and utilization. In this study, we evaluate a memory subsystem design leveraging CXL-enabled memory pooling. Two promising use cases of composable memory subsystems are studied -fine-grained capacity provisioning and scalable bandwidth provisioning. We developed an emulator to explore the performance impact of various memory compositions. We also provide a profiler to identify the memory usage patterns in applications and their optimization opportunities. Seven scientific and six graph applications are evaluated on various emulated memory configurations. Three out of seven scientific applications had less than 10% performance impact when the pooled memory backed 75% of their memory footprint. The results also show that a dynamically configured highbandwidth system can effectively support bandwidth-intensive unstructured mesh-based applications like OpenFOAM. Finally, we identify interference through shared memory pools as a practical challenge for adoption on HPC systems.

show abstract

Evaluating the potential of disaggregated memory systems for HPC applications

Ding,

Maris,

Nam

et al. 2024

Concurrency and Computation

View full text Add to dashboard Cite

SummaryDisaggregated memory is a promising approach that addresses the limitations of traditional memory architectures by enabling memory to be decoupled from compute nodes and shared across a data center. Cloud platforms have deployed such systems to improve overall system memory utilization, but performance can vary across workloads. High‐performance computing (HPC) is crucial in scientific and engineering applications, where HPC machines also face the issue of underutilized memory. As a result, improving system memory utilization while understanding workload performance is essential for HPC operators. Therefore, learning the potential of a disaggregated memory system before deployment is a critical step. This paper proposes a methodology for exploring the design space of a disaggregated memory system. It incorporates key metrics that affect performance on disaggregated memory systems: memory capacity, local and remote memory access ratio, injection bandwidth, and bisection bandwidth, providing an intuitive approach to guide machine configurations based on technology trends and workload characteristics. We apply our methodology to analyze thirteen diverse workloads, including AI training, data analysis, genomics, protein, fusion, atomic nuclei, and traditional HPC bookends. Our methodology demonstrates the ability to comprehend the potential and pitfalls of a disaggregated memory system and provides motivation for machine configurations. Our results show that eleven of our thirteen applications can leverage injection bandwidth disaggregated memory without affecting performance, while one pays a rack bisection bandwidth penalty and two pay the system‐wide bisection bandwidth penalty. In addition, we also show that intra‐rack memory disaggregation would meet the application's memory requirement and provide enough remote memory bandwidth.

show abstract

A Case For Intra-rack Resource Disaggregation in HPC

Cited by 22 publications

References 64 publications

Analyzing Resource Utilization in an HPC System: A Case Study of NERSC Perlmutter

Analyzing Resource Utilization in an HPC System: A Case Study of NERSC Perlmutter

Evaluating Emerging CXL-enabled Memory Pooling for HPC Systems

Evaluating the potential of disaggregated memory systems for HPC applications

Contact Info

Product

Resources

About