Efficient I/O for computational grid applications.

Abstract

High-performance computing increasingly occurs on "computational grids" composed of heterogeneous and geographically distributed systems of computers, networks, and storage devices that collectively act as a single "virtual" computer. A key challenge in this environment is to provide efficient access to data distributed across remote data servers. This dissertation explores some of the issues associated with I/O for wide-area distributed computing and describes an I/O system, called Armada, with the following features: a framework to allow application and dataset providers to flexibly compose graphs of processing modules that describe the distribution, application interfaces, and processing required of the dataset before or after computation; an algorithm to restructure application graphs to increase parallelism and to improve network performance in a wide-area network; and a hierarchical graph-partitioning scheme that deploys components of the application graph in a way that is both beneficial to the application and sensitive to the administrative policies of the different administrative domains. Experiments show that applications using Armada perform well in both low-and high-bandwidth environments, and that our approach does an exceptional job of hiding the network latency inherent in grid computing. ii Thanks to my parents for their love and encouragement over the years. I can only hope to be as much of an inspiration to my children as they were to me. Thanks to David Kotz for his guidance and friendship. He has a great instinct for knowing when to provide direction, when to provide encouragement, and when to get out of the way-all are qualities of a great advisor. Thanks to Tom Cormen for his car, two pairs of tennis shoes, his pit smoker, and for teaching me that nothing is more important than good barbecue. Thanks to the many graduate students that made my time at Dartmouth an enjoyable experience. Special thanks to Clint Hepner (my fishing companion), B.J. Premore (my hiking companion), and Senthil Periaswamy, a great friend and colleague. Thanks to the many people at Sandia National Laboratories that provided guidance (and funding) for my research. Special thanks to Jeff Nelson, Bill Camp, and David Womble. They are each great mentors and deserve much of the credit for my success. Thanks to Jay Lepreau, and the students and staff who run the Emulab at the University of Utah, for allowing us to use their facility to run our experiments. Most of all, thanks to my wife Susan for her loving support over the past six years.

Improving Data Access for Computational Grid Applications

“…We include a detailed analysis of graph restructuring in [24]. To summarize, our analysis has two parts: first, we calculate the size of the largest possible tree (based on the number of leaves in the initial tree) and the number of operations used by PARALLELIZELEFT and PARALLELIZERIGHT to generate the tree, then we calculate the number of operations (based on the number of nodes in the largest possible tree) used by RESTRUCTURE, SLIDELEFT, and SLIDERIGHT to decide whether or not to swap series-connected nodes.…”

“…For each application, we present a brief description followed by performance results. For details on the design and implementation of the various applications, see [24].…”

“…Intuitively, COMMUTATIVE returns true when the requestmapping function for at least one ship is the request-identity, and the data-mapping function for at least one ship is the dataidentity. We provide a more formal proof of commutativity in [24].…”

“…Our implementation makes use of a graph-partitioning software library called Chaco [17]. Further details of the placement algorithm are beyond the scope of this document, but can be found in [24].…”