Clusterfile: a flexible physical layout parallel file system

Abstract

This paper presents Clusterfile, a parallel file system that provides parallel file access on a cluster of computers. We introduce a file partitioning model that has been used in the design of Clusterfile. The model uses a data representation that is optimized for multidimensional array partitioning while allowing arbitrary partitions. The paper shows how the file model can be employed for file partitioning into both physical subfiles and logical views. We also present how the conversion between two partitions of the same file is implemented using a general memory redistribution algorithm. We show how we use the algorithm to optimize non‐contiguous read and write operations. The experimental results include performance comparisons with the Parallel Virtual File System (PVFS) and an MPI‐IO implementation for PVFS. Copyright © 2003 John Wiley & Sons, Ltd.

Making the case for reforming the I/O software stack of extreme-scale systems

“…Exploiting data locality is directly linked to data layout awareness and control. Existing solutions either offer the applications mechanisms for controlling data layout of the file system (e.g., PVFS distributions [3], Clusterfile logical and physical layout [23]) or propose intermediary layers for optimally exploiting given data layouts through collective I/O [38], block alignment [26], or mapping of application data models on a file system optimized layout [11].…”

A New Framework to Build Science Gateways based on EnginFrame and Liferay

“…One approach to address the bandwidth limitation of scattered I/O is to describe the required I/O operations at a more abstract level. Regularly repeating patterns of binary data can be described by the so-called 'Family of Line Segments' (FALLS)[14]. The pattern-based I/O routines in SAGA use such descriptions to reduce the bandwidth limitation of scattered I/O.…”