PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets

Kumar, Sidharth; Vishwanath, Venkatram; Carns, Philip; Summa, Brian; Scorzelli, Giorgio; Pascucci, Valerio; Ross, Robert; Chen, J.; Kolla, Hemanth; Grout, Ray

doi:10.1109/cluster.2011.19

Cited by 24 publications

(18 citation statements)

References 19 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This strategy is complementary to the approach followed in one-phase writes, where every process first calculates the HZ order and corresponding HZ level for its sub-volume, and then uses independent MPI-I/O write operations to store each level. For both reading and writing this method entails a high degree of noncontiguous data access of the file system, dramatically impeding scalability [15]. See figure 6a.…”

Section: Read Implementationmentioning

confidence: 99%

“…Each run of S3D generates four variables: pressure, temperature, velocity (3 samples) and species (11 samples). Details of integration of the PIDX I/O library with S3D are described in [15].…”

Section: Experiments Setupmentioning

confidence: 99%

“…In [13], we demonstrated the use of PIDX in coordinating parallel write access to multidimensional, regular datasets, and explored several low-level encoding and file access optimizations. In subsequent work [15], we introduced a two-phase I/O strategy [9] in which each process first performs a local HZ encoding of its own data. The data is then aggregated by a subset of processes (aggregators) that in turn write it to disk.…”

Section: Pidx: Parallel Idxmentioning

confidence: 99%

“…IDX is a hierarchical multiresolution data format with support for both lightweight serial and fast distributed parallel I/O. Parallel I/O is performed using PIDX [15], a fast parallel library for reading and writing IDX multiresolution data.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Fast Multiresolution Reads of Massive Simulation Datasets

Kumar

Christensen

Schmidt

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. Today's massively parallel simulation codes can produce output ranging up to many terabytes of data. Utilizing this data to support scientific inquiry requires analysis and visualization, yet the sheer size of the data makes it cumbersome or impossible to read without computational resources similar to the original simulation. We identify two broad classes of problems for reading data and present effective solutions for both. The first class of data reads depends on user requirements and available resources. Tasks such as visualization and user-guided analysis may be accomplished using only a subset of variables with a restricted spatial extent at a reduced resolution. The other class of reads requires full resolution multivariate data to be loaded, for example to restart a simulation. We show that utilizing the hierarchical multiresolution IDX data format enables scalable and efficient serial and parallel read access on a variety of hardware from supercomputers down to portable devices. We demonstrate interactive view-dependent visualization and analysis of massive scientific datasets using low-power commodity hardware, and we compare read performance with other parallel file formats for both full and partial resolution data.

show abstract

Section: Read Implementationmentioning

confidence: 99%

Section: Experiments Setupmentioning

confidence: 99%

Section: Pidx: Parallel Idxmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Fast Multiresolution Reads of Massive Simulation Datasets

Kumar

Christensen

Schmidt

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…Kumar et al [49] implement a two-phase I/O for multiple files in the context of the PIDX library. In a following work, Kumar et al [48] discuss a modification in their two-phase I/O strategy.…”

Section: Requests Aggregation and Reorderingmentioning

confidence: 99%

A Checkpoint of Research on Parallel I/O for High-Performance Computing

et al. 2018

View full text Add to dashboard Cite

We present a comprehensive survey on parallel I/O in the high performance computing (HPC) context. This is an important field for HPC because of the historic gap between processing power and storage latencies, which causes applications performance to be impaired when accessing or generating large amounts of data. As the available processing power and amount of data increase, I/O remains a central issue for the scientific community. In this survey, we focus on a traditional I/O stack, with a POSIX parallel file system. We present background concepts everyone could benefit from. Moreover, through the comprehensive study of publications from the most important conferences and journals in a five-year time window, we discuss the state of the art of I/O optimization approaches, access pattern extraction techniques, and performance modeling, in addition to general aspects of parallel I/O research. Through this approach, we aim at identifying the general characteristics of the field and the main current and future research topics.

show abstract

A coarse‐grained page cache aware multivariate analytical model for the storage performance of a parallel file system

Inacio

Dantas

2017

Concurrency and Computation

View full text Add to dashboard Cite

Summary Huge requirements of emerging Big Data applications combined with the performance impairment of current I/O subsystems pose a great challenge on data storage, management, and access performance. In order to design efficient storage systems, a clear understanding about how factors in the I/O path affect the performance of a data‐intensive application is of utmost importance. This paper reports our ongoing research toward addressing this issue, presenting a coarse‐grained page cache aware multivariate analytical model for the performance of write operations in a parallel file system. The proposed model was developed to reflect the performance behavior observed in an extensive experimental effort, in which the impact of 14 parameters in the response time and throughput of the OrangeFS was investigated. More than one million experiments were carried out using four distinct computing infrastructures, providing a detailed performance characterization. Additionally, a thorough evaluation of the proposed model, covering more than 14 000 scenarios, is reported, discussing both qualitative and quantitative aspects. Evaluation results indicate that the model succeeded in representing the behavior of the parallel file system performance, achieving a Mean Absolute Percentage Error of 39.94%.

show abstract

PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets

Cited by 24 publications

References 19 publications

Fast Multiresolution Reads of Massive Simulation Datasets

Fast Multiresolution Reads of Massive Simulation Datasets

A Checkpoint of Research on Parallel I/O for High-Performance Computing

A coarse‐grained page cache aware multivariate analytical model for the storage performance of a parallel file system

Contact Info

Product

Resources

About