The Galley parallel file system

Nieuwejaar, Nils; Kotz, David

doi:10.1016/s0167-8191(97)00009-4

Cited by 73 publications

(40 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our collective download approach resembles the collective I/O technique extensively studied in the parallel I/O field and widely used in parallel simulations [8,28,35,48,54]. Collective I/O attacks the I/O performance problem caused by a mismatch of data distribution in memory and in files by consolidating small, scattered I/O requests into large, sequential ones.…”

Section: Related Workmentioning

confidence: 99%

“…A more efficient alternative is to let each node download a large, contiguous data segment and perform local data shuffling to exchange data stripes. We refer to this as collective download, following similar concepts of collective I/O [8,28,35,48,54]. The downloaded data is simultaneously shuffled locally for a rearranged layout, conforming to the smaller stripe size used in the distributed cache.…”

Section: Coupling Rpdr With Prefix Caching and Collective Downloadmentioning

confidence: 99%

See 1 more Smart Citation

Improving Data Availability for Better Access Performance: A Study on Caching Scientific Data on Distributed Desktop Workstations

Vazhkudai

Zhang

2009

J Grid Computing

View full text Add to dashboard Cite

Client-side data caching serves as an excellent mechanism to store and analyze the rapidly growing scientific data, motivating distributed, client-side caches built from unreliable desktop storage contributions to store and access large scientific data. They offer several desirable properties, such as performance impedance matching, improved space utilization, and high parallel I/O bandwidth. In this context, we are faced with two key challenges: (1) the finite amount of contributed cache space is stretched This work builds on several of our previously published studies. While we included extended introduction of the FreeLoader framework [57,58] as well as our combination of prefix caching and collective download techniques [33], this paper proposes the novel RDPR technique as well as a new striping-enabled cache management algorithm. We also discussed methods to combine RDPR with prefix caching and collective download techniques. Overall, more than 60% of the manuscript's content has not been previously published.X. Ma · Z. Zhang (B) North Carolina State University, Raleigh, NC, USA e-mail: zzhang3@ncsu.edu X. Ma · S. S. Vazhkudai Oak Ridge National Laboratory, Oak Ridge, TN, USA by the ever increasing scientific dataset sizes and (2) the transient nature of volunteered storage nodes impacts data availability. In this article, we address these challenges by exploiting the existence of external, primary copies of datasets. We propose a novel combination of prefix caching, collective download, and remote partial data recovery (RPDR), to deal with optimal cache space consumption and storage node volatility. Our evaluation, performed on our FreeLoader prototype, indicates that prefix caching can significantly improve the cache hit rate and partial data recovery is better than (or comparable to) many persistent-data availability techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Coupling Rpdr With Prefix Caching and Collective Downloadmentioning

confidence: 99%

Improving Data Availability for Better Access Performance: A Study on Caching Scientific Data on Distributed Desktop Workstations

Vazhkudai

Zhang

2009

J Grid Computing

View full text Add to dashboard Cite

show abstract

“…This workload is constructed based on data extracted from [1,23,24,25], and has the following attributes: a sequential read access stream from a single large file followed by a sequential write access stream to the same file. This read/write access stream represents a multi-phase application with an initial read phase to load input data and a final write phase that saves the computed results.…”

Section: Validating the Mathematical Machinerymentioning

confidence: 99%

A Cost-Effective Distributed File Service with QoS Guarantees

Bianchini

Nguyen

2007

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. Large-scale, value-added Internet services composed of independent cooperating or competing services will soon become common place. Several groups have addressed the performance, communication, discovery, and description aspects of these services. However, little work has been done on effectively composing paid services and the quality-of-service (QoS) guarantees that they provide. We address these issues in the context of distributed file storage in this paper. In particular, we propose, implement, and evaluate a cost-effective, QoSaware distributed file service comprising a front-end file service and back-end (third-party) storage services. Our front-end service uses mathematical modeling and optimization to provide performance and availability guarantees at low cost by carefully orchestrating the accesses to the back-end services. Experimental results from our prototype implementation validate our modeling and optimization. We conclude that our approach for providing QoS at low cost should be useful to future composite Internet services.

show abstract

“…However, many if not all these extensions were ad-hoc and specific to a vendor or academic prototype. Designers of the Galley parallel file system showed that many applications in scientific computing environments exhibit highly regular, but nonconsecutive I/O access patterns in [15,14]. Further, since the conventional interfaces do not provide an efficient method of describing these patterns, they present three extensions to the I/O interface to support such applications with regular access patterns.…”

mentioning

confidence: 99%