DeltaFS

Zheng, Qing; Ren, Kai; Gibson, Garth A.; Settlemyer, Bradley W.; Grider, Gary

doi:10.1145/2834976.2834984

Cited by 43 publications

(5 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They foresee four major challenges in moving ADIOS towards exascale: data integrity, fault recovery, automated tuning, and data consistency. DeltaFS [188] is a file system that improves the scalability of file systems by letting compute nodes manage metadata instead of a centralized server (common in traditional distributed file systems). Kunkel et al [105] review three methods to improve I/O performance at exascale: re-computation (found to be beneficial for infrequent accesses),…”

Section: I/o Middleware I/o Performance Can Also Be Improved By Solumentioning

confidence: 99%

The Landscape of Exascale Research

et al. 2020

View full text Add to dashboard Cite

The next generation of supercomputers will break the exascale barrier. Soon we will have systems capable of at least one quintillion (billion billion) floating-point operations per second (10 18 FLOPS). Tremendous amounts of work have been invested into identifying and overcoming the challenges of the exascale era. In this work, we present an overview of these efforts and provide insight into the important trends, developments, and exciting research opportunities in exascale computing. We use a three-stage approach in which we (1) discuss various exascale landmark studies, (2) use data-driven techniques to analyze the large collection of related literature, and (3) discuss eight research areas in depth based on influential articles. Overall, we observe that great advancements have been made in tackling the two primary exascale challenges: energy efficiency and fault tolerance. However, as we look forward, we still foresee two major concerns: the lack of suitable programming tools and the growing gap between processor performance and data bandwidth (i.e., memory, storage, networks). Although we will certainly reach exascale soon, without additional research, these issues could potentially limit the applicability of exascale computing.

show abstract

Section: I/o Middleware I/o Performance Can Also Be Improved By Solumentioning

confidence: 99%

The Landscape of Exascale Research

et al. 2020

View full text Add to dashboard Cite

show abstract

“…To be coherent with the trie-based design that DART is following, we use ART [21] as the server-side indexing data structure for DART. We use hash table as the server-side local data structure for two DHTbased approaches, "Full String Hashing" and "Initial Hashing", since such combination is prevalently used in many existing studies, such as FusionFS [49], DeltaFS [50], SoMeta [43], etc. For all experiments, we set up an equal number of client instances and server instances.…”

Section: Evaluation 51 Experimental Setupmentioning

confidence: 99%

“…Moreover, these approaches do not scale well in HPC environments and it is di cult to ensemble them into a HPC storage system as a pluggable component. Also, as a common practice for managing distributed index in many HPC storage systems (such as SoMeta [43], IndexFS [35], FusionFS [49], and DeltaFS [50]), distributed hash table (DHT) [47] has been used to build inverted index for metadata [48]. However, when it comes…”

Section: Introductionmentioning

confidence: 99%

“…The term-partitioned approach takes each indexed keyword as the cue for index partitioning. The typical case of such approach designed for HPC systems is the use of DHT, which can be found in many recent distributed metadata and key value store solutions, such as SoMeta [43], IndexFS [35], FusionFS [49], and DeltaFS [50]. However, the application of DHT still remains inappropriate for a x-based keyword search in two-fold.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dart

Zhang

Tang

Byna

et al. 2018

Proceedings of the 27th International Conference on Parallel Architectures and Compilation Techniques

View full text Add to dashboard Cite

A x-based search is a fundamental functionality for storage systems. It allows users to nd desired datasets, where attributes of a dataset match an a x. While building inverted index to facilitate e cient a xbased keyword search is a common practice for standalone databases and for desktop le systems, building local indexes or adopting indexing techniques used in a standalone data store is insu cient for highperformance computing (HPC) systems due to the massive amount of data and distributed nature of the storage devices within a system. In this paper, we propose Distributed Adaptive Radix Tree (DART), to address the challenge of distributed a x-based keyword search on HPC systems. This trie-based approach is scalable in achieving ecient a x-based search and alleviating imbalanced keyword distribution and excessive requests on keywords at scale. Our evaluation at different scales shows that, comparing with the "full string hashing" use case of the most popular distributed indexing technique-Distributed Hash Table (DHT), DART achieves up to 55× better throughput with pre x search and with su x search, while achieving comparable throughput with exact and in x searches. Also, comparing to the "initial hashing" use case of DHT, DART maintains a balanced keyword distribution on distributed nodes and alleviates excessive query workload against popular keywords.

show abstract

“…We have implemented DeltaFS IMDs as a special directory type that runs on the clients of a distributed filesystem [104,105]. To transform data to a read-optimized format, our implementation uses the popular key-value interface [33] to enable applications to define the data to be transformed.…”

mentioning

confidence: 99%

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

et al. 2020

Self Cite

View full text Add to dashboard Cite

Complex storage stacks providing data compression, indexing, and analytics help leverage the massive amounts of data generated today to derive insights. It is challenging to perform this computation, however, while fully utilizing the underlying storage media. This is because, while storage servers with large core counts are widely available, single-core performance and memory bandwidth per core grow slower than the core count per die. Computational storage offers a promising solution to this problem by utilizing dedicated compute resources along the storage processing path. We present DeltaFS Indexed Massive Directories (IMDs), a new approach to computational storage. DeltaFS IMDs harvest available (i.e., not dedicated) compute, memory, and network resources on the compute nodes of an application to perform computation on data. We demonstrate the efficiency of DeltaFS IMDs by using them to dynamically reorganize the output of a real-world simulation application across 131,072 CPU cores. DeltaFS IMDs speed up reads by 1,740× while only slightly slowing down the writing of data during simulation I/O for in situ data processing.

show abstract

DeltaFS

Cited by 43 publications

References 27 publications

The Landscape of Exascale Research

The Landscape of Exascale Research

Dart

Streaming Data Reorganization at Scale with DeltaFS Indexed Massive Directories

Contact Info

Product

Resources

About