An Implementation and Evaluation of Client-Side File Caching for MPI-IO

Liao, Wei-keng; Ching, Avery; Coloma, Kenin; Choudhary, Alok; Ward, Lee

doi:10.1109/ipdps.2007.370239

Cited by 38 publications

(33 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data prefetching improves performance by overlapping application's data access phases and computation phases. The prefetching thread needs to predict what data the application will request and fetches the data from storage devices to local buffer (or a collective cache in parallel I/O system [10]) before the application issuing the actual requests.…”

Section: A Accurate and Timely Data Prefetchingmentioning

confidence: 99%

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

Yin

Byna

Song

et al. 2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

View full text Add to dashboard Cite

Section: A Accurate and Timely Data Prefetchingmentioning

confidence: 99%

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

Yin

Byna

Song

et al. 2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

View full text Add to dashboard Cite

“…Such synchronization provides a collective I/O implementation an abundant opportunity for process collaboration, for instance, by exchanging access information and reorganizing the I/O requests. Many collaboration strategies besides the two-phase I/O and disk directed I/O have been proposed and demonstrated their successes, including server-directed I/O [18], persistent file domain [3,2], active bu↵ering [13], and collaborative caching [4,11]. Independent I/O, in contrast, requiring no synchronization makes any collaborative optimization di cult.…”

Section: Background and Related Workmentioning

confidence: 99%

Improving collective I/O performance by pipelining request aggregation and file access

Sehrish¹,

Son²,

Liao³

et al. 2013

Proceedings of the 20th European MPI Users' Group Meeting

Self Cite

View full text Add to dashboard Cite

In this paper, we propose a multi-bu↵er pipelining approach to improve collective I/O performance by overlapping the dominant request aggregation phases with the I/O phase in the two-phase I/O implementation. Our pipelining method first divides the collective bu↵er into a group of small size bu↵ers for an individual collective I/O call and then pipelines the asynchronous communication to exchange the I/O requests with the I/O requests sent to the file system. Our performance evaluation of a representative I/O benchmark and a production application shows 20% improvement in the I/O time, given theoretical upper bound of 50% when both phases completely overlap.

show abstract

“…Previous research efforts with parallel I/O on the Lustre file system have shed some light on factors contributing to the poor performance of MPI-IO, including the problems caused by I/O accesses that are not aligned on stripe boundaries [17,18]. Figure 3 helps to illustrate the problem that arises when I/O accesses cross stripe boundaries.…”

Section: A Known Issues With Parallel I/o On Lustrementioning

confidence: 99%

“…In [18], it was shown that aligning the data to be written with the basic striping pattern improves performance. They also showed that it was important to align on lock boundaries.…”

Section: Related Workmentioning

confidence: 99%

Towards a High Performance Implementation of MPI-IO on the Lustre File System

Dickens

Logan

2008

On the Move to Meaningful Internet Systems: OTM 2008

View full text Add to dashboard Cite

Abs tra ct -Lustre is becoming an increasingly important file system for large-scale computing clusters. The problem is that many dataintensive applications use MPI-IO for their I/O requirements, and it has been well documented that MPI-IO performs poorly in a Lustre file system environment. However, the reasons for such poor performance are not currently well understood. We believe that the primary reason for poor performance is that the assumptions underpinning most of the parallel I/O optimizations implemented in MPI-IO do not hold in a Lustre environment. Perhaps the most important assumption that appears to be incorrect is that optimal performance is obtained by performing large, contiguous I/O operations. Our research suggests that this is often the worst approach to take in a Lustre file system. In fact, we found that the best performance is sometimes achieved when each process performs a series of smaller, non-contiguous I/O requests. In this paper, we provide experimental results showing that such assumptions do not apply in Lustre, and explore new approaches that appear to provide significantly better performance.

show abstract

An Implementation and Evaluation of Client-Side File Caching for MPI-IO

Abstract: Abstract

Cited by 38 publications

References 16 publications

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

Boosting Application-Specific Parallel I/O Optimization Using IOSIG

Improving collective I/O performance by pipelining request aggregation and file access

Towards a High Performance Implementation of MPI-IO on the Lustre File System

Contact Info

Product

Resources

About