Storage challenges at Los Alamos National Lab

Bent, John; Grider, Gary; Kettering, Brett; Manzanares, Adam; McClelland, Meghan; Torres, Aaron; Torrez, Alfred

doi:10.1109/msst.2012.6232376

Cited by 39 publications

(14 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The optimization proposed by Manzanares et al [68] consists of distributing the library files in different points of this tree to avoid a situation where only one of the servers is keeping all of them. Bent et al [5] point PLFS as one of the key solutions to support storage demands at the exascale era. Figure 7(a) presents the number of publications on collective I/O by year.…”

Section: Requests Aggregation and Reorderingmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

Section: Requests Aggregation and Reorderingmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Examples including Bent [10], Brown [13], Liu [30], and other works [35,33,27] have explored utilizing burst buffers to improve the resilience.…”

Section: Burst Buffersmentioning

confidence: 99%

“…Non-volatile memories in the form of SSD's (Solid State Disks) or in-array storage and used as burst buffers are a promising approach to meet these bandwidth demands, absorbing checkpoint data at much higher bandwidth than possible with the disk-based file systems [30,10,13]. Emerging CR systems (SCR [31] and FTI [9]) exploit these burst buffers to increase checkpoint frequency, and reduce checkpoint costs.…”

Section: Introductionmentioning

confidence: 99%

How Much SSD Is Useful for Resilience in Supercomputers

Fang

Chien

2015

Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale

View full text Add to dashboard Cite

We consider the use of non-volatile memories in the form of burst buffers for resilience in supercomputers. Their cost and limited lifetime demand effective use and appropriate provisioning. We develop an analytic model for the behavior of workloads on systems with burst buffers, and use it to explore questions of cost-effective provisioning, and missiondirected allocation of burst-buffer (SSD) lifetime.First, our results show that system efficiency can be increased by as much as 14% by considering a global perspective (workload mix, job size) for SSD lifetime allocation. Second, with size-based and system-efficiency based lifetime allocation, large jobs suffer as much as 40% job efficiency loss; job-efficiency based allocation must increase their allocations by 50% to eliminate this disparity. Finally, further results suggest that underprovisioning SSD lifetime (only 10-20% of the "optimum" as defined by per-job requirements without resource constraint) is sufficient to produce 90% system efficiency at failure rates three times that of current systems.

show abstract

“…PLFS [5,23,7,6] is an interposing filesystem that sits between HPC applications and one or more backing filesystems. PLFS has no persistent storage itself.…”

Section: Plfsmentioning

confidence: 99%

“…In addition to these three extensions, the flexible PLFS framework has also been used to prototype exascale-level I/O subsystems such as burst buffers [4,7]. A burst buffer is a local solid state storage device (SSD) that quickly absorbs a burst of I/O (e.g.…”

Section: Plfs Architecture Extensibilitymentioning

confidence: 99%

Structuring PLFS for extensibility

Cranor

Polte²,

Gibson

2013

Proceedings of the 8th Parallel Data Storage Workshop

View full text Add to dashboard Cite

The Parallel Log Structured Filesystem (PLFS) [5] was designed to transparently transform highly concurrent, massive high-performance computing (HPC) N-to-1 checkpoint workloads into N-to-N workloads to avoid single-file performance bottlenecks in typical HPC distributed filesystems. PLFS has produced speedups of 2-150X for N-1 workloads at Los Alamos National Lab. Having successfully improved N-1 performance, we have restructured PLFS for extensibility so that it can be applied to more workloads and storage systems. In this paper we describe PLFS' evolution from a single-purpose log-structured middleware filesystem into a more general platform for transparently translating application I/O patterns. As an example of this extensibility, we show how PLFS can now be used to enable HPC applications to perform N-1 checkpoints on an HDFS-based cloud storage system.

show abstract

Storage challenges at Los Alamos National Lab

Cited by 39 publications

References 9 publications

A Checkpoint of Research on Parallel I/O for High-Performance Computing

A Checkpoint of Research on Parallel I/O for High-Performance Computing

How Much SSD Is Useful for Resilience in Supercomputers

Structuring PLFS for extensibility

Contact Info

Product

Resources

About