Language, compiler and parallel database support for I/O intensive applications

Discretionary Caching for I/O on Clusters

“…Compilation of I/O-intensive codes using explicit I/O has also been the focus of some research (see [3,1,28] for example techniques that target out-of-core datasets). Brezany et al [3] have developed a parallel I/O system called VIPIOS that can be used by an optimizing compiler.…”

“…Brezany et al [3] have developed a parallel I/O system called VIPIOS that can be used by an optimizing compiler. Bordawekar et al [1,2] have focussed on stencil computations that can be re-ordered freely due to lack of flow-dependences.…”

Compiler-directed I/O optimization

“…This second point manifests itself in a number of ways. First, at the compiler level, very few techniques (e.g., [20,4,3,12]) target specifically I/O statements; rather, many data dependence and data reuse analysis techniques employed by state-of-the-art optimizing compilers fail when, during compilation, an I/O statement (e.g., within a loop) is encountered. Second, at the runtime system, libraries, and operating system (OS) levels, no single, standard API for I/O exists.…”

“…While many scientific environments process vast amounts of data, in most cases, the regularity in data access patterns enable code and data structuring for better I/O behavior. Therefore, previous work in I/O compilation [3,20,15,4] targeted scientific applications that manipulate out-of-core data sets. Note that a compiler is in a good position for optimizing I/O in a program-wide fashion as it can capture the global (program-wide) data access pattern.…”

“…In the first-phase, the processors read the file using the access pattern 4 . In the second phase, they perform a collective communication to re-distribute the data between them so as to obtain the final (desired) access pattern 4 . While collective I/O incurs an extra communication cost (whose magnitude depends on the number of processors involved, on the amount of data that needs to be communicated, and on the existence of collective communication primitives in the underlying file system), since the per item I/O cost is, in general, much higher than the per item communication cost in today's parallel machines, we can expect large performance gains.…”