The impact of spatial layout of jobs on I/O hotspots in mesh networks

Mache, Jens; Lo, Virginia; Garg, Sharad

doi:10.1016/j.jpdc.2005.04.020

Cited by 7 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is a typical behavior because the file system used in our testbed is a shared resource and therefore prone to contention from other users. In addition, as observed in a previous study [21], many other issues, such as the complexity of the I/O software stack and the contention of non-IO MPI communication, could also contribute to the variability. To obtain reasonable peak I/O performance values, we repeat each experiment at least 5 times over the course of several weeks.…”

Section: Methodsmentioning

confidence: 66%

Optimizing fastquery performance on lustre file system

Lin

Byna

Chou

et al. 2013

Proceedings of the 25th International Conference on Scientific and Statistical Database Management

View full text Add to dashboard Cite

FastQuery is a parallel indexing and querying system we developed for accelerating analysis and visualization of scientific data. We have applied it to a wide variety of HPC applications and demonstrated its capability and scalability using a petascale trillion-particle simulation in our previous work. Yet, through our experience, we found that performance of reading and writing data with FastQuery, like many other HPC applications, could be significantly affected by various tunable parameters throughout the parallel I/O stack. In this paper, we describe our success in tuning the performance of FastQuery on a Lustre parallel file system. We study and analyze the impact of parameters and tunable settings at file system, MPI-IO library, and HDF5 library levels of the I/O stack. We demonstrate that a combined optimization strategy is able to improve performance and I/O bandwidth of FastQuery significantly. In our tests with a trillion-particle dataset, the time to index the dataset reduced by more than one half.

show abstract

Section: Methodsmentioning

confidence: 66%

Optimizing fastquery performance on lustre file system

Lin

Byna

Chou

et al. 2013

Proceedings of the 25th International Conference on Scientific and Statistical Database Management

View full text Add to dashboard Cite

show abstract

“…Continuously keeping metadata updated in order to have fully consistent reads and writes (POSIX metadata management), requires writing a new value for the file's last-accessed time (POSIX atime) every time a file is read, imposing a significant burden on parallel file system 76 . It was observed that contention for the interconnect between OSTs and compute nodes due to MPI communication may lead to variable performance in I/O measurements 77 . Conversely, our data suggest that single-shared-file I/O on Lustre can negatively affect MPI communication as well, even at moderate numbers (tens to hundreds) of concurrent requests, similar to recent network simulations that predicted interference between MPI and I/O traffic 78 .…”

Section: Likely Causes Of Stragglersmentioning

confidence: 99%

Parallel performance of molecular dynamics trajectory analysis

Khoshlessan

Paraskevakos

Fox

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

The performance of biomolecular molecular dynamics (MD) simulations has steadily increased on modern high performance computing (HPC) resources but acceleration of the analysis of the output trajectories has lagged behind so that analyzing simulations is becoming a bottleneck.To close this gap, we studied the performance of parallel trajectory analysis with MPI and the Python MDAnalysis library on three different XSEDE supercomputers where trajectories were read from a Lustre parallel file system. Strong scaling performance was impeded by stragglers, MPI processes that were slower than the typical process. Stragglers were less prevalent for compute-bound workloads, thus pointing to file reading as a crucial bottleneck for scaling. However, a more complicated picture emerged in which both the computation and the ingestion of data exhibited close to ideal strong scaling behavior whereas stragglers were primarily caused by either large MPI communication costs or long times to open the single shared trajectory file. We improved overall strong scaling performance by either subfiling (splitting the trajectory into separate files) or MPI-IO with Parallel HDF5 trajectory files. We obtained near ideal strong scaling on up to 384 cores (16 nodes), thus reducing trajectory analysis times by two orders of magnitude compared to the serial approach.

show abstract

“…One possible application for which the average distance is minimized is in the allocation of jobs to nodes in a supercomputer [1][2][3][4][5]. Another possible application is in VLSI layout, where the objective is to cluster nearby components of a circuit on a chip [6].…”

Section: Introductionmentioning

confidence: 99%

Optimal shape of a blob

Bender

2007

Journal of Mathematical Physics

View full text Add to dashboard Cite

This paper presents the solution to the following optimization problem: What is the shape of the two-dimensional region that minimizes the average L p distance between all pairs of points if the area of this region is held fixed? [The L p distance between two points x = (x 1 , x 2 ) andVariational techniques are used to show that the boundary curve of the optimal region satisfies a nonlinear integral equation. The special case p = 2 is elementary and for this case the integral equation reduces to a differential equation whose solution is a circle. Two nontrivial special cases, p = 1 and p = ∞, have already been examined in the literature. For these two cases the integral equation reduces to nonlinear secondorder differential equations, one of which contains a quadratic nonlinearity and the other a cubic nonlinearity.

show abstract

The impact of spatial layout of jobs on I/O hotspots in mesh networks

Cited by 7 publications

References 34 publications

Optimizing fastquery performance on lustre file system

Optimizing fastquery performance on lustre file system

Parallel performance of molecular dynamics trajectory analysis

Optimal shape of a blob

Contact Info

Product

Resources

About