Making the case for reforming the I/O software stack of extreme-scale systems

Isaila, Florin; García, Javier Avilés; Carretero, Jesús; Ross, Robert; Kimpe, Dries

doi:10.1016/j.advengsoft.2016.07.003

Cited by 11 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The current uncoordinated development model of independently applying optimizations at each layer of the HPC I/O software stack will not scale to the new levels of concurrency and deepening storage hierarchy . Making The Fast Forward Storage and I/O (FFSIO) project redesigns the HPC I/O stack to address the requirements for extreme scale data storage, by preferring high level I/O library like HDF5 and translating data model to the underlying storage options.…”

Section: Related Workmentioning

confidence: 99%

Cross‐layer coordination in the I/O software stack of extreme‐scale systems

Liu

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary I/O forwarding layer has now become a standard storage layer in today's HPC systems in order to scale current storage systems to new levels of concurrency. With the deepening of storage hierarchy, I/O requests must traverse through several types of nodes to access required data, including compute nodes, I/O nodes, and storage nodes. It becomes difficult to control the data path and apply cross‐layer I/O optimization. In this paper, we propose a well coordinated I/O stack, which coordinates the data path between compute nodes and I/O nodes for better load balancing and data locality with a job‐level I/O node mapping, and coordinates data path between I/O nodes and storage nodes for lighter I/O interference. We implement and evaluate our ideas on Tianhe‐1A by leveraging an open‐source I/O forwarding layer named IOFSL. The experimental results show that our proposals can significantly accelerate I/O performance of multiple I/O kernels and real applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Cross‐layer coordination in the I/O software stack of extreme‐scale systems

Liu

et al. 2017

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Another work has focused on a scheduling model reducing I/O interferences between HPC applications [15] leading to more stable performance. Some work has also been done to re-think the software stack managing I/O in a unified way [16].…”

Section: Related Workmentioning

confidence: 99%

Reproducibility and Variability of I/O Performance on BG/Q: Lessons Learned from a Data Aggregation Algorithm

Tessier

Vishwanath

2017

View full text Add to dashboard Cite

Reading and writing data efficiently from different tiers of storage is necessary for most scientific simulations to achieve good performance at scale. Many software solutions have been developed to decrease the I/O bottleneck. One wellknown strategy, in the context of collective I/O operations, is the two-phase I/O scheme. This strategy consists of selecting a subset of processes to aggregate contiguous pieces of data before performing reads/writes. In our previous work, we implemented the two-phase I/O scheme with a MPI-based topology-aware algorithm. Our algorithm showed very good performance at scale compared to the standard I/O libraries such as POSIX I/O and MPI I/O. However, the algorithm had several limitations hindering a satisfying reproducibility of our experiments. In this paper, we extend our work by 1) identifying the obstacles we face to reproduce our experiments and 2) discovering solutions that reduce the unpredictability of our results.

show abstract

“…Economic realities drive the architecture, performance, and reliability of the hardware that will comprise an exascale I/O system [1]. Moreover, I/O researchers [2] have highlighted significant weaknesses in the current I/O stacks that will need to be addressed in order to enable the development of systems that measurably demonstrate all of the properties required of an exascale I/O system. Possible poor filesystem performance at exascale has lead to the introduction of new or augmented file systems [3].…”

Section: Introductionmentioning

confidence: 99%

DAOS for Extreme-scale Systems in Scientific Applications

Breitenfeld,

Fortner,

Henderson

et al. 2017

Preprint

View full text Add to dashboard Cite

Exascale I/O initiatives will require new and fully integrated I/O models which are capable of providing straightforward functionality, fault tolerance and efficiency. One solution is the Distributed Asynchronous Object Storage (DAOS) technology, which is primarily designed to handle the next generation NVRAM and NVMe technologies envisioned for providing a high bandwidth/IOPS storage tier close to the compute nodes in an HPC system. In conjunction with DAOS, the HDF5 library, an I/O library for scientific applications, will support end-to-end data integrity, fault tolerance, object mapping, index building and querying. This paper details the implementation and performance of the HDF5 library built over DAOS by using three representative scientific application codes.

show abstract

Making the case for reforming the I/O software stack of extreme-scale systems

Cited by 11 publications

References 22 publications

Cross‐layer coordination in the I/O software stack of extreme‐scale systems

Cross‐layer coordination in the I/O software stack of extreme‐scale systems

Reproducibility and Variability of I/O Performance on BG/Q: Lessons Learned from a Data Aggregation Algorithm

DAOS for Extreme-scale Systems in Scientific Applications

Contact Info

Product

Resources

About