Characterizing parallel file-access patterns on a large-scale multiprocessor

Purakayastha, Apratim; Ellis, Carla Schlatter; Kotz, David; Nieuwejaar, Nils; Best, Michael L.

doi:10.1109/ipps.1995.395928

Cited by 39 publications

(26 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Therefore different accesses interleave with each other; in many cases, this is an interleav-ing of streams of strided access used to read or write multidimensional data structures [149,515,415,516,554]. This interleaving leads to a phenomenon known as interprocess locality, because accesses by one process allow the system to predict accesses by other processes [412].…”

Section: Parallel File Systemsmentioning

confidence: 99%

Workload Modeling for Computer Systems Performance Evaluation

Feitelson

2015

191

131

View full text Add to dashboard Cite

Reliable performance evaluations require the use of representative workloads. This is no easy task since modern computer systems and their workloads are complex, with many interrelated attributes and complicated structures. Experts often use sophisticated mathematics to analyze and describe workload models, making these models difficult for practitioners to grasp. This book aims to close this gap by emphasizing the intuition and the reasoning behind the definitions and derivations related to the workload models. It provides numerous examples from real production systems, with hundreds of graphs. Using this book, readers will be able to analyze collected workload data and clean it if necessary, derive statistical models that include skewed marginal distributions and correlations, and consider the need for generative models and feedback from the system. The descriptive statistics techniques covered are also useful for other domains.

show abstract

Section: Parallel File Systemsmentioning

confidence: 99%

Workload Modeling for Computer Systems Performance Evaluation

Feitelson

2015

191

131

View full text Add to dashboard Cite

show abstract

“…The CHARISMA study [5][6][7] finds that file sizes in scientific workloads are much larger than those typically found in UNIX workstation environments and that most applications access only a few files. Approximately 90% of file accesses are small-less than 4 KB-and represent a considerable portion of application execution time, even though approximately 90% of the data is transferred in large accesses.…”

Section: Small I/o Requestsmentioning

confidence: 99%

“…Numerous workload characterization studies have highlighted the prevalence of small and sequential data requests in modern scientific applications [5][6][7][8][9][10][11]. This trend will likely continue since many HPC applications take years to develop, have a productive lifespan of ten years or more, and are not easily re-architected for the latest file access paradigm [12].…”

Section: Introductionmentioning

confidence: 99%

Large files, small writes, and pNFS

Hildebrand

Ward

Honeyman

2006

Proceedings of the 20th Annual International Conference on Supercomputing

View full text Add to dashboard Cite

Workload characterization studies highlight the prevalence of small and sequential data requests in scientific applications. Parallel file systems excel at large data transfers but sometimes at the expense of small I/O performance. pNFS is an NFSv4.1 high-performance enhancement that provides direct storage access to parallel file systems while preserving NFSv4 operating system and hardware platform independence. This paper demonstrates that distributed file systems can increase write throughput to parallel data stores-regardless of file size-by overcoming parallel file system inefficiencies. We also show how pNFS can improve the overall write performance of parallel file systems by using direct, parallel I/O for large write requests and a distributed file system for small write requests. We describe our pNFS prototype and present experiments demonstrating the performance improvements. ABSTRACTWorkload characterization studies highlight the prevalence of small and sequential data requests in scientific applications. Parallel file systems excel at large data transfers but sometimes at the expense of small I/O performance. pNFS is an NFSv4.1 highperformance enhancement that provides direct storage access to parallel file systems while preserving NFSv4 operating system and hardware platform independence. This paper demonstrates that distributed file systems can increase write throughput to parallel data stores-regardless of file size-by overcoming parallel file system inefficiencies. We also show how pNFS can improve the overall write performance of parallel file systems by using direct, parallel I/O for large write requests and a distributed file system for small write requests. We describe our pNFS prototype and present experiments demonstrating the performance improvements.

show abstract

“…There has also been a large body of recent work on characterizing the I/O behavior of scientific applications in parallel and supercomputing environments. Among them are [6,15,41,47,50,51,53]. For a more detailed discussion of related work, the reader is referred to Section A1 in the Appendix.…”

Section: Related Workmentioning

confidence: 99%

“…See for instance [6,15,41,47,50,51,53]. In general, the I/O reference patterns of these applications are more regular and predictable than those of commercial database workloads.…”

Section: A15 Parallel and Supercomputing I/omentioning

confidence: 99%

I/O reference behavior of production database workloads and the TPC benchmarks—an analysis at the logical level

Hsu

Smith

Young

2001

ACM Trans. Database Syst.

View full text Add to dashboard Cite

As improvements in processor performance continue to far outpace improvements in storage performance, I/O is increasingly the bottleneck in computer systems, especially in large database systems that manage huge amounts of data. The key to achieving good I/O performance is to thoroughly understand its characteristics. In this paper, we present a comprehensive analysis of the logical I/O reference behavior of the peak production database workloads from ten of the world's largest corporations by focusing on how these workloads respond to different techniques for caching, prefetching and write buffering. Our findings include several broadly applicable rules of thumb that describe how effective the various I/O optimization techniques are for the production workloads. For instance, our results indicate that the buffer pool miss ratio tends to be related to the ratio of buffer pool size to data size by an inverse square root rule. A similar fourth root rule relates the write miss ratio and the ratio of buffer pool size to data size.

show abstract

Characterizing parallel file-access patterns on a large-scale multiprocessor

Cited by 39 publications

References 20 publications

Workload Modeling for Computer Systems Performance Evaluation

Workload Modeling for Computer Systems Performance Evaluation

Large files, small writes, and pNFS

I/O reference behavior of production database workloads and the TPC benchmarks—an analysis at the logical level

Contact Info

Product

Resources

About