Measuring I/O Performance of Lustre and the Temporary File System for Tradespace Applications on HPC Systems

Kosta, Leonard; Hunter, Harrison; George, Glover E.; Strelzoff, Andrew; Matthews, Suzanne J.

doi:10.1145/3077286.3077326

Cited by 2 publications

(1 citation statement)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This could be the correct option in some HPC scenarios, but it is not feasible in a Lustre file-system environment. Creating what could be thousands of files is the well-documented bane of the Lustre file-system (Kosta et al 2017;Alam et al 2011), which is the file-system used by all ERDC HPC machines. The second parallel approach involves a feature of MPI, namely MPI-IO.…”

Section: Approachmentioning

confidence: 99%

Parallel file I/O for geometric models : formats and methods

Hunter¹,

White²,

Patel³

et al. 2020

View full text Add to dashboard Cite

Processing large amounts of data in a High-Performance Computing (HPC) environment can be throttled quickly without an efficient method for utilizing disk I/O. The Geometry Engine component of the Virtual Environment for Sensor Performance Assessment (VESPA) uses MPI-IO to load the geometric data in parallel and avoid creating a bottleneck on disk I/O interactions. This parallel I/O method requires formatting the data into specific binary file formats so each MPI process of the parallel program can determine where to read or write data without colliding with other MPI processes. Addressing the collision problem resulted in the development of two binary file formats, the Mesh Binary file (.mb) and the Scene Chunk Pack file (.scp). The Mesh Binary file contains the essential data required to recreate the landscape and vegetation geometry used by the Geometry Engine. The Scene Chunk Pack file is used to write the partitioned geometry to disk, so the ray casting engine can reload the distributed geometry without repeating the partitioning process. Both of these files together support reading and writing for the partitioning phase and the ray casting phase of the Geometry Engine. This report discusses these formats in detail and outlines how the Geometry Engine reads and writes these files in parallel on HPC. DISCLAIMER:The contents of this report are not to be used for advertising, publication, or promotional purposes. Citation of trade names does not constitute an official endorsement or approval of the use of such commercial products. All product names and trademarks cited are the property of their respective owners. The findings of this report are not to be construed as an official Department of the Army position unless so designated by other authorized documents.DESTROY THIS REPORT WHEN NO LONGER NEEDED. DO NOT RETURN IT TO THE ORIGINATOR.

show abstract

Section: Approachmentioning

confidence: 99%