GEMS: User Control for Cooperative Scientific Repositories

Abstract

Data repositories are an integral part of modern scientific computing systems. While a variety of grid-enabled storage systems have been developed to improve scalability, administrative control, and interoperability, users have several outstanding needs: to seamlessly and efficiently work with replicated data sets, to customize system behavior within a grid, and to quickly tie together remotely administered grids or independently operated resources. This is particularly true in the small virtual organization, in which a subset of possible users seek to coordinate subcomponents of existing grids into a workable collaborative system. Our approach to this problem starts with the storage system and seeks to enable this functionality by creating ad hoc storage grids. Modern commodity hardware in use at research labs and university networks ships with an abundance of storage space that is often underutilized, and even consumer gadgets provide extensive storage resources that will not immediately be filled. The installation of simple software enables these systems to be pooled and cataloged into a spacious, parallel ad hoc storage network. While traditional storage networks or tertiary storage systems are isolated behind file servers and firewalls, constricting data movement, we layer the storage service network atop the client consumer network, improving the available network parallelism and boosting I/O performance for data-intensive scientific tasks.