Protecting Metadata Servers From Harm Through Application-level I/O Control

Macedo, Ricardo; Miranda, Mariana; Tanimura, Yusuke; Haga, Jason; Ruhela, Amit; Harrell, Stephen Lien; Evans, R. T.; Paulo, João

doi:10.1109/cluster51413.2022.00075

Cited by 3 publications

(2 citation statements)

References 41 publications

(103 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The scale and intricacy of applications have simultaneously elevated both the volume and complexity of metadata. Recent research indicates that metadata can profoundly influence the entire HPC I/O efficiency [4]. Importantly, merely scaling up hardware resources is not a panacea for the metadata conundrum.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Venkatesh,

Eisenhauer,

Podhorszki

et al. 2024

ISC High Performance 2024 Research Paper Proceedings (39th International Conference)

View full text Add to dashboard Cite

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.

show abstract

Section: Introductionmentioning

confidence: 99%

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Venkatesh,

Eisenhauer,

Podhorszki

et al. 2024

ISC High Performance 2024 Research Paper Proceedings (39th International Conference)

View full text Add to dashboard Cite

show abstract

“…The sharp increase in data volume and the increasingly complex and diverse workload patterns of HPC applications pose numerous challenges to the I/O processing capabilities of HPC systems [25,26]. The traditional disk-based parallel file systems being unable to meet the IO demands of applications [28]. Some HPC applications also have characteristics of alternating computation and IO cycles.…”

Section: Introductionmentioning

confidence: 99%

GABB: the plan-based job scheduling optimized by genetic algorithm for HPC systems with shared burst buffers

Lv,

Xie,

Song

2024

Third International Symposium on Computer Applications and Information Systems (ISCAIS 2024)

View full text Add to dashboard Cite

Burst Buffer was proposed to address the growing performance gap between computation and I/O in high-performance computing systems (HPC). However, the introduction of Burst Buffer has brought new issues in system resource management and job scheduling. Resource management systems on HPC platforms lack efficient management of multiple types of resources, and the allocation of computing resources and Burst Buffer resources is independent of each other. This can lead to underutilization of system resources and job blocking. To address these issues, this paper proposes GABB, a plan-based job scheduling strategy for shared Burst Buffer, and utilizes genetic algorithms for optimization. GABB employs a plan-based job scheduling strategy to uniformly schedule all jobs in the waiting queue to generate an execution plan. Furthermore, GABB comprehensively considers changes in system resources and the job's demand for computing and Burst Buffer resources during the job scheduling process. Finally, GABB utilizes an improved genetic algorithm to optimize the job scheduling scheme. We conducted experimental simulations of a shared Burst Buffer system and implemented a plan-based job scheduling algorithm. The experimental results indicate that BBGA significantly reduces the mean waiting time of jobs by over 20% compared to the Shortest Job First (SJF) algorithm, and reduces the mean bounded slowdown of jobs by over 25%.

show abstract

I/O Access Patterns in HPC Applications: A 360-Degree Survey

2023

View full text Add to dashboard Cite

The high-performance computing (HPC) I/O stack has been complex due to multiple software layers, the inter-dependencies among these layers, and the different performance tuning options for each layer. In this complex stack, the definition of an “I/O access pattern” has been re-appropriated to describe what an application is doing to write or read data from the perspective of different layers of the stack, often comprising a different set of features. It has become common having to redefine what is meant when discussing a pattern in every new study as no assumption can be made. This survey aims to propose a baseline taxonomy, harnessing the I/O community’s knowledge over the last 20 years. This definition can serve as a common ground for HPC I/O researchers and developers to apply known I/O tuning strategies and design new strategies for improving I/O performance. We seek to summarize and bring a consensus with the multiple ways to describe a pattern based on common features already used by the community over the years.

show abstract

Protecting Metadata Servers From Harm Through Application-level I/O Control

Cited by 3 publications

References 41 publications

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

GABB: the plan-based job scheduling optimized by genetic algorithm for HPC systems with shared burst buffers

I/O Access Patterns in HPC Applications: A 360-Degree Survey

Contact Info

Product

Resources

About