Design of Object Storage Using OpenNVM for High-performance Distributed File System

Takatsu, Fuyumasa; Hiraga, Kenji; Tatebe, Osamu

doi:10.2197/ipsjjip.24.824

Cited by 5 publications

(5 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To cope with the issue, this paper proposed a scalable metadata server system, PPMDS. PPMDS is motivated from the Gfarm [Tatebe et al 2010] distributed file system and is already integrated with an object storage [Takatsu et al 2016] and forms a file system called PPFS [Takatsu et al 2017].…”

Section: Discussionmentioning

confidence: 99%

“…PPFS [Takatsu et al 2017] is a distributed file system that utilizes the PP-MDS [Hiraga et al 2018] for file system metadata management. It introduces the design of the PPFS file system using the PPMDS and PPOST object storagebased file servers [Takatsu et al 2016]. It includes the performance evaluation of not only the PPFS but also the PPMDS comparing with the IndexFS.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Scalable Distributed Metadata Server Based on Nonblocking Transactions

Hiraga¹,

Tatebe²,

Kawashima³

2020

jucs

Self Cite

View full text Add to dashboard Cite

Metadata performance scalability is critically important in high-performance computing when accessing many small files from millions of clients. This paper proposes a design of a scalable distributed metadata server, PPMDS, for parallel file systems using multiple key-value servers. In PPMDS, hierarchical namespace of a file system is efficiently managed by multiple servers. Multiple entries can be atomically updated using a nonblocking distributed transaction based on an algorithm of dynamic software transactional memory. This paper also proposes optimizations to further improve the metadata performance by introducing a server-side transaction processing, multiple readers, and a shared lock mode, which reduce the number of remote procedure calls and prevent unnecessary blocking. Performance evaluation shows the scalable performance up to 3 servers, and achieves 62,000 operations per second, which is 2.58x performance improvement compared to a single metadata performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Scalable Distributed Metadata Server Based on Nonblocking Transactions

Hiraga¹,

Tatebe²,

Kawashima³

2020

jucs

Self Cite

View full text Add to dashboard Cite

show abstract

“…Post-Petascale File System (PPFS) based on object storage using OpenNVM and targets to converge both HPC and cloud computing [90,91]. This system uses a key-value store for metadata storage and non-blocking distributed transactions to update multiple entries at the same time.…”

Section: Storage and File Systemsmentioning

confidence: 99%

Survey of Storage Systems for High-Performance Computing

Lüttgau

Kuhn

Duwe

et al. 2018

JSFI

View full text Add to dashboard Cite

In current supercomputers, storage is typically provided by parallel distributed file systems for hot data and tape archives for cold data. These file systems are often compatible with local file systems due to their use of the POSIX interface and semantics, which eases development and debugging because applications can easily run both on workstations and supercomputers. There is a wide variety of file systems to choose from, each tuned for different use cases and implementing different optimizations. However, the overall application performance is often held back by I/O bottlenecks due to insufficient performance of file systems or I/O libraries for highly parallel workloads. Performance problems are dealt with using novel storage hardware technologies as well as alternative I/O semantics and interfaces. These approaches have to be integrated into the storage stack seamlessly to make them convenient to use. Upcoming storage systems abandon the traditional POSIX interface and semantics in favor of alternative concepts such as object and key-value storage; moreover, they heavily rely on technologies such as NVM and burst buffers to improve performance. Additional tiers of storage hardware will increase the importance of hierarchical storage management. Many of these changes will be disruptive and require application developers to rethink their approaches to data management and I/O. A thorough understanding of today's storage infrastructures, including their strengths and weaknesses, is crucially important for designing and implementing scalable storage systems suitable for demands of exascale computing.

show abstract

“…High-performance Distributed File System We previously proposed a local object storage that manages file data as an object on a local storage device using OpenNVM flash primitives for high-performance distributed file systems [5]. The sparse address space allows the design of an array of fixed-size regions that contain a single object, where the object can be addressed by a region number, i.e., object ID.…”

Section: Ppost-an Object Storage Using Opennvm Formentioning

confidence: 99%

“…We proposed the design of object storage using OpenNVM [4] for HPC [5] (hereafter termed PPOST). This is used as the backend of a storage node that manages file data on a local storage device.…”

Section: Introductionmentioning

confidence: 99%

PPFS: A Scale-out Distributed File System for Post-petascale Systems

Takatsu

Hiraga

Tatebe

2017

Journal of Information Processing

Self Cite

View full text Add to dashboard Cite

Abstract:The fusion of the research field of high-performance computing (HPC) with that of big data, which has become known as the field of extreme big data, is problematic in that file creation in storage systems such as distributed file systems is not optimized. That is, the large workload leads to simultaneous creations of many files by many processes when creating checkpoints. The need to improve the file creation processes prompted us to design a scale-out distributed file system for post-petascale systems named PPFS. PPFS consists of PPMDS, which is a scale-out distributed metadata server, and PPOSS, which is a scalable distributed storage server for flash storage. The high file creation performance of PPMDS was achieved by using a key-value store for metadata storage and non-blocking distributed transactions to update multiple entries simultaneously. PPOSS depends on PPOST, which is an object storage system that manages the underlying low-level storage, such as Fusion IO ioDrive, a flash device connected through PCI express supporting OpenNVM. The high file creation performance was attained by implementing the PPFS prototype using file creation optimization, termed bulk creation, to reduce the amount of communication between PPMDS and PPOSS. And, to enhance the I/O performance of PPOSS when the client process and PPOSS run on the same node, PPOSS accesses a local storage device directly. The prototype implementation of PPFS with a further file creation optimization called object prefetching achieves 138,000 Operations Per Second for file creation when using five metadata servers and 128 client processes, thereby exceeding the performance of IndexFS by 2.52 times. With local access optimization, PPOSS reached its limit at a block size of 16 KiB, which is an improvement of 1.5 times compared to before optimization. Furthermore, this evaluation indicates that PPFS has a good scalability on file creation and IO performance, that is required for post-petascale systems.

show abstract

Design of Object Storage Using OpenNVM for High-performance Distributed File System

Cited by 5 publications

References 20 publications

Scalable Distributed Metadata Server Based on Nonblocking Transactions

Scalable Distributed Metadata Server Based on Nonblocking Transactions

Survey of Storage Systems for High-Performance Computing

PPFS: A Scale-out Distributed File System for Post-petascale Systems

Contact Info

Product

Resources

About