Improving collective I/O performance using threads

Dickens, Phillip M.; Thakur, Rajeev

doi:10.1109/ipps.1999.760432

Cited by 27 publications

(20 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dickens and Thakur (1999) implemented split-collective I/O in MPI-2 standard to provide a collective I/O that overlaps with computing by threads. The research found that simply spawning a thread to perform the collective I/O operation in the background is worse than the sequential approach.…”

Section: Related Workmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

Abstract. We describe the design and implementation of climate fast input/output (CFIO), a fast input/output (I/O) library for high-resolution climate models. CFIO provides a simple method for modelers to overlap the I/O phase with the computing phase automatically, so as to shorten the running time of numerical simulations. To minimize the code modifications required for porting, CFIO provides similar interfaces and features to parallel Network Common Data Form (PnetCDF), which is one of the most widely used I/O libraries in climate models. We deployed CFIO in three highresolution climate models, including two ocean models (POP and LICOM) and one sea ice model (CICE). The experimental results show that CFIO improves the performance of climate models significantly versus the original serial I/O approach. When running with CFIO at 0.1 • resolution with about 1000 CPU cores, we managed to reduce the running time by factors of 7.9, 4.6 and 2.0 for POP, CICE, and LI-COM, respectively. We also compared the performance of CFIO against two existing libraries, PnetCDF and parallel I/O (PIO), in different scenarios. For scenarios with both data output and computations, CFIO decreases the I/O overhead compared to PnetCDF and PIO.

show abstract

Section: Related Workmentioning

confidence: 99%

A fast input/output library for high-resolution climate models

Wang

Yang

et al. 2014

Geosci. Model Dev.

View full text Add to dashboard Cite

show abstract

“…Many strategies have been proposed to improve collective I/O performance [6,20,1]. None of the existing approaches, however, try to overlap the data aggregation and I/O phases.…”

Section: Background and Related Workmentioning

confidence: 99%

Improving collective I/O performance by pipelining request aggregation and file access

Sehrish¹,

Son²,

Liao³

et al. 2013

Proceedings of the 20th European MPI Users' Group Meeting

View full text Add to dashboard Cite

In this paper, we propose a multi-bu↵er pipelining approach to improve collective I/O performance by overlapping the dominant request aggregation phases with the I/O phase in the two-phase I/O implementation. Our pipelining method first divides the collective bu↵er into a group of small size bu↵ers for an individual collective I/O call and then pipelines the asynchronous communication to exchange the I/O requests with the I/O requests sent to the file system. Our performance evaluation of a representative I/O benchmark and a production application shows 20% improvement in the I/O time, given theoretical upper bound of 50% when both phases completely overlap.

show abstract

“…Compared to previous work on overlapping I/O with computation automatically [2], [5], [21], active buffering is more platform-independent and has neither hard requirements for, nor performance sensitivity to, extra memory availability. Active buffering can also be easily extended to support efficient wide-area remote I/O, and can help users to tune the frequency of periodic output.…”

Section: Other Related Workmentioning

confidence: 99%

High-level buffering for hiding periodic output cost in scientific simulations

Ma¹,

Lee²,

Winslett³

2006

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Abstract-Scientific applications often need to write out large arrays and associated metadata periodically for visualization or restart purposes. In this paper, we present active buffering, a high-level transparent buffering scheme for collective I/O, in which processors actively organize their idle memory into a hierarchy of buffers for periodic output data. It utilizes idle memory on the processors, yet makes no assumption regarding runtime memory availability. Active buffering can perform background I/O while the computation is going on, is extensible to remote I/O for more efficient data migration, and can be implemented in a portable style in today's parallel I/O libraries. It can also mask performance problems of scientific data formats used by many scientists. Performance experiments with both synthetic benchmarks and real simulation codes on multiple platforms show that active buffering can greatly reduce the visible I/O cost from the application's point of view.

show abstract

Improving collective I/O performance using threads

Cited by 27 publications

References 19 publications

A fast input/output library for high-resolution climate models

A fast input/output library for high-resolution climate models

Improving collective I/O performance by pipelining request aggregation and file access

High-level buffering for hiding periodic output cost in scientific simulations

Contact Info

Product

Resources

About