Verifying IO Synchronization from MPI Traces

Yellapragada, Sushma; Wang, Chen; Snir, Marc

doi:10.1109/pdsw54622.2021.00012

Cited by 2 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This workflow simulates the typical scenarios where a single file may be accessed concurrently by HPC applications and multiple versions of a dataset may be generated accordingly. As shown in [34] and Recorder [35] collect coarse-grained statistics of I/O performance, there are no tools to extract data access information and the cost of those operations. Fine-grained information such as the total number of each type of HDF5 I/O operations incurred during the workflow, the accumulated time cost for each type of operations, the distribution of operations and time overhead, the HDF5 APIs invoked at a specific time point, etc.…”

Section: H5bench -Data Usage and I/o Performancementioning

confidence: 99%

“…Non-Provenance Tools. In addition, great efforts have been made to manage workflows [59], [60] or log I/O events for various purposes [34], [35], [61]- [75]. While they are effective for their original goals, they are insufficient to address provenance needs in general due to a number of reasons: (1) no relational model to support tracking or querying rich provenance (e.g., various relations defined in W3C PROV-DM [28]); (2) agnostic to the fine-grained semantics in HPC I/O libraries (e.g., HDF5 attributes); (3) little portability across different I/O libraries or workflow environments; (4) no programmable interface to specify customized provenance needs.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Prov-Io

Han

Byna

Tang

et al. 2022

Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Data provenance, or data lineage, describes the life cycle of data. In scientific workflows on HPC systems, scientists often seek diverse provenance (e.g., origins of data products, usage patterns of datasets). Unfortunately, existing provenance solutions cannot address the challenges due to their incompatible provenance models and/or system implementations.In this paper, we analyze three representative scientific workflows in collaboration with the domain scientists to identify concrete provenance needs. Based on the first-hand analysis, we propose a provenance framework called PROV-IO, which includes an I/O-centric provenance model for describing scientific data and the associated I/O operations and environments precisely. Moreover, we build a prototype of PROV-IO to enable end-to-end provenance support on real HPC systems with little manual effort. The PROV-IO framework provides flexibility in selecting various classes of provenance. Our experiments with realistic workflows show that PROV-IO can address the provenance needs of the domain scientists effectively with reasonable performance (e.g., less than 3.5% tracking overhead for most experiments). Moreover, PROV-IO outperforms a state-of-the-art system (i.e., ProvLake) in our experiments. CCS CONCEPTS• Computer systems organization → Parallel architectures; • Information systems → Data management systems.

show abstract

Section: H5bench -Data Usage and I/o Performancementioning

confidence: 99%