Large files, small writes, and pNFS

Hildebrand, Dean; Ward, Lee; Honeyman, Peter

doi:10.1145/1183401.1183419

Cited by 21 publications

(12 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the future, we plan to apply our response time-based replica management strategy in HDFS [3], PVFS [11], pNFS [12], Gpfs [13], and LusterFS [14].…”

Section: Discussionmentioning

confidence: 99%

RTRM: A Response Time-Based Replica Management Strategy for Cloud Storage System

Bai

Jin

Xiaofei

et al. 2013

Grid and Pervasive Computing

View full text Add to dashboard Cite

Abstract.Replica management has become a hot research topic in storage systems. This paper presents a dynamic replica management strategy based on response time, named RTRM. RTRM strategy consists of replica creation, replica selection, and replica placement mechanisms. RTRM sets a threshold for response time, if the response time is longer than the threshold, RTRM will increase the number of replicas and create new replica. When a new request comes, RTRM will predict the bandwidth among the replica servers, and make the replica selection accordingly. The replica placement refers to search new replica placement location, and it is a NP-hard problem. Based on graph theory, this paper proposes a reduction algorithm to solve this problem. The simulation results show that RTRM strategy performs better than the five built-in replica management strategies in terms of network utilization and service response time.

show abstract

“…In the future, we plan to apply our response time-based replica management strategy in HDFS [3], PVFS [11], pNFS [12], Gpfs [13], and LusterFS [14].…”

Section: Discussionmentioning

confidence: 99%

RTRM: A Response Time-Based Replica Management Strategy for Cloud Storage System

Bai

Jin

Xiaofei

et al. 2013

Grid and Pervasive Computing

View full text Add to dashboard Cite

show abstract

“…Furthermore, we implement I/O buffering in BMF to batch small read and write operations into large ones before accessing the file system. This buffering strategy avoids flurries of small reads and writes that can degrade the I/O performance of large parallel file systems [20], [21].…”

Section: Parallelizing the Fastbit-based Mass Querymentioning

confidence: 99%

BMF: Bitmapped Mass Fingerprinting for Fast Protein Identification

Wu²,

et al. 2011

2011 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Abstract-Protein identification is an important objective for proteomic and medical sciences as well as for pharmaceutical industry. With recent large-scale automation of genome sequencing and the explosion of protein databases, it is important to exploit latest data processing technologies and design highly scalable algorithms to speed up protein identification. In this study, we design, implement, and evaluate a new software tool, Bitmapped Mass Fingerprinting (BMF), that can efficiently construct a bitmap index for short peptides, and quickly identify candidate proteins from leading protein databases. BMF is developed by integrating the FastBit indexing technology and the popular Message Passing Interface (MPI) for parallelization. By exploiting FastBit for peptide mass fingerprinting across protein boundaries, we are able to accomplish parallel computation and I/O for a scalable implementation of protein identification. Our experimental results show that BMF brings dramatic performance improvement for protein identification from various protein databases. In particular, we demonstrate that BMF can effectively scale up to 8,192 cores on the Jaguar Supercomputer at Oak Ridge National Laboratory, achieving superb performance in identifying proteins from the National Center for Biotechnology Information (NCBI) non-redundant (NR) protein database.

show abstract

“…The traditional dynamic power management strategy is an efficient energy conservation technique for large idle time periods, which make it worthwhile to spin down disks when they are sitting idle. However, small and sequential data requests in modern scientific applications are very prevalent [7], making it less likely to observe large idle time intervals among requests. Moreover, small writes cause not only an energy consumption problem but also an I/O performance problem [1].…”

Section: Related Workmentioning

confidence: 99%

“…In the past decade, large-scale storage systems have been developed to achieve high I/O performance and large storage capacity for a wide variety of data-intensive applications [13] [6][10] [7]. Much attention has been paid to the issues of performance and security in storage systems [21][19] [2].…”

Section: Introductionmentioning

confidence: 99%

Performance Evaluation of Energy-Efficient Parallel I/O Systems with Write Buffer Disks

Ruan

Manzanares

Yin

et al. 2009

2009 International Conference on Parallel Processing

View full text Add to dashboard Cite

Abstract-In the past decade, parallel disk systems have been developed to address the problem of I/O performance. A critical challenge with modern parallel I/O systems is that parallel disks consume a significant amount of energy in servers and highperformance computers. To conserve energy consumption in parallel I/O systems, one can immediately spin down disks when disk are idle; however, spinning down disks might not be able to produce energy savings due to penalties of spinning operations. Unlike powering up CPUs, spinning down and up disks need physical movements. Therefore, energy savings provided by spinning down operations must offset energy penalties of the disk spinning operations. To substantially reduce the penalties incurred by disk spinning operations, we developed a novel approach to conserving energy of parallel I/O systems with write buffer disks, which are used to accumulate small writes using a log file system. Data sets buffered in the log file system can be transferred to target data disks in a batch way. Thus, buffer disks aim to serve a majority of incoming write requests, attempting to reduce the large number of disk spinning operations by keeping data disks in standby for long period times. Interestingly, the write buffer disks not only can achieve high energy efficiency in parallel I/O systems, but also can shorten response times of write requests. To evaluate the performance and energy efficiency of our parallel I/O systems with buffer disks, we implemented a prototype using a cluster storage system as a testbed. Experimental results show that under light and moderate I/O load, buffer disks can be employed to significantly reduce energy dissipation in parallel I/O systems without adverse impacts on I/O performance.

show abstract

Large files, small writes, and pNFS

Cited by 21 publications

References 21 publications

RTRM: A Response Time-Based Replica Management Strategy for Cloud Storage System

RTRM: A Response Time-Based Replica Management Strategy for Cloud Storage System

BMF: Bitmapped Mass Fingerprinting for Fast Protein Identification

Performance Evaluation of Energy-Efficient Parallel I/O Systems with Write Buffer Disks

Contact Info

Product

Resources

About