Monitoring HPC applications in the production environment

Abstract

The advancement of HPC systems brings with it a need for more introspection into the run-time environment and performance of longrunning applications. Software and hardware fault tolerance, scaling performance issues, soft error effect on computations, and even large-scale computational progress will require more capable runtime monitoring of applications during production runs. Current HPC toolsets, however, are geared towards heavyweight program introspection during the development, debugging, and optimization phases of software development, and are not suitable for production monitoring. To this end this work has developed PROMON, a framework for the production monitoring of HPC applications. PROMON is part of a vision for automatic monitoring of HPC applications without requiring developers or users to do significant work to support it. This paper presents PROMON, motivates its purpose, and shows that production monitoring can be performed efficiently.

PMIx

“…These situations can result in a significant waste of resources as the SMS is unaware of the problem, and thus cannot terminate the job. Various watchdog methods have been developed for detecting this situation, including requiring a periodic "heartbeat" from the application and monitoring a specified file for changes in size [15,18].…”