Enabling comprehensive data-driven system management for large computational facilities

Abstract

This paper presents a tool chain, based on the open source tool TACC_Stats, for systematic and comprehensive job level resource use measurement for large cluster computers, and its incorporation into XDMoD, a reporting and analytics framework for resource management that targets meeting the information needs of users, application developers, systems administrators, systems management and funding managers. Accounting, scheduler and event logs are integrated with system performance data from TACC_Stats.TACC_Stats periodically records resource use including many hardware counters for each job running on each node. Furthermore, system level metrics are obtained through aggregation of the node (job) level data. Analysis of this data generates many types of standard and custom reports and even a limited predictive capability that has not previously been available for open-source, Linux-based software systems. This paper presents case studies of information that can be applied for effective resource management. We believe this system to be the first fully comprehensive system for supporting the information needs of all stakeholders in open-source software based HPC systems.

Failure Diagnosis for Cluster Systems using Partial Correlations

“…The Lustre filesystem is commonly used to provide highspeed data I/O on many HPC systems. However, I/O problems on Lustre has been widely reported [33]. To identify the dates when Lustre experienced I/O problem, we obtain the lists of partial correlated counters and partial correlated messages and identified cases of I/O problems on Lustre.…”

Managing computational gateway resources with XDMoD

“…The figure clearly displays a substantial difference in resource utilization between the two jobs; Figure 7a shows inefficient use of the requested resources, with CPU User less than 3% for 24 cores; Figure 7b shows CPU User near 100% for 24 cores. 5 The Job Viewer lets user support staff and end users examine their job data in detail, enabling improved identification and diagnosis of inefficient jobs, and better use of resources.…”

“…The XD Metrics on Demand (XDMoD) tool provides stakeholders with ready access to data about utilization, performance, and quality of service for High Performance Computing (HPC) resources. [1]- [5] This comprehensive tool was originally developed to support resources for the National Science Foundation (NSF) XSEDE program; it was later opensourced and made available to general HPC resources at universities, government laboratories, and commercial entities. [6] XDMoD enables users, managers, and operations staff to monitor, assess and maintain quality of service for their computational resources.…”

Performance Analysis Tool for HPC and Big Data Applications on Scientific Clusters

“…Yadwadkar et al [39] proposed to use the the Support Vector Machine (SVM) [15] to proactively predict stragglers from cluster resource utilization counters. Browne et al [8] proposed a comprehensive resource management tool by combining data from event logs, schedulers, and performance counters. In addition, Chuah et al [13] proposed to link resource usage anomalies with system failures.…”