Dynamic file-access characteristics of a production parallel scientific workload

Abstract

Multiprocessors have permitted astounding increases in computational performance, but many cannot meet the intense I/O requirements of some scienti c applications. An important component of any solution to this I/O bottleneck is a parallel le system that can provide high-bandwidth access to tremendous amounts of data in parallel to hundreds or thousands of processors.Most successful systems are b ased on a solid understanding of the expected workload, but thus far there have been no comprehensive workload characterizations of multiprocessor le systems. This paper presents the results of a three w e ek tracing study in which all le-related activity on a massively parallel computer was recorded. Our instrumentation di ers from previous e orts in that it collects information about every I/O request and about the mix of jobs running in a production environment. We also present the results of a trace-driven caching simulation and recommendations for designers of multiprocessor le systems.

CEFT: A cost-effective, fault-tolerant parallel virtual file system

“…The I/O traces of scientific applications show a frequent pattern in which multiple clients concurrently access the same files [30]. In CEFT, we employ a centralized byte-range lease-based mechanism to allow simultaneous accesses to different portions of a single file while maintaining the multiplereader single-writer semantics to each requested data portion.…”

A New Flexible MPI Collective I/O Implementation

“…These datatypes are used to represent memory and/or file. The most common I/O patterns found in scientific computing involve non-contiguous access to files [10]. For Figure 4, data in both memory and file are noncontiguous.…”

An evaluation of Java's I/O capabilities for high-performance computing

“…Related work is discussed in Section 7, followed by conclusions in Section 8. Many computationally intensive s c i e n ti c applications also need to access large amounts of data, and I/O is often the bottleneck in such applications 3,13,22]. A common I/O requirement i s a s f o l l o ws.…”