Using Message Logs and Resource Use Data for Cluster Failure Diagnosis

Abstract

Copyright and reuse:The Warwick Research Archive Portal (WRAP) makes this work by researchers of the University of Warwick available open access under the following conditions. Copyright © and all moral rights to the version of the paper presented here belong to the individual author(s) and/or other copyright owners. To the extent reasonable and practicable the material made available in WRAP has been checked for eligibility before being made available.Copies of full items can be used for personal research or study, educational, or not-for profit purposes without prior permission or charge. Provided that the authors, title and full bibliographic details are credited, a hyperlink and/or URL is given for the original metadata page and the content is not changed in any way.Publisher's statement: © 2016 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting /republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. A note on versions:The version presented here may differ from the published version or, version of record, if you wish to cite this item you are advised to consult the publisher's version. Please see the 'permanent WRAP url' above for details on accessing the published version and note that access may require a subscription. Abstract-Failure diagnosis for large compute clusters using only message logs is known to be incomplete. Recent availability of resource use data provides another potentially useful source of data for failure detection and diagnosis. Early work combining message logs and resource use data for failure diagnosis has shown promising results. This paper describes the CRUMEL framework which implements a new approach to combining rationalized message logs and resource use data for failure diagnosis. CRUMEL identifies patterns of errors and resource use and correlates these patterns by time with system failures. Application of CRUMEL to data from the Ranger supercomputer has yielded improved diagnoses over previous research. CRUMEL has: (i) showed that more events correlated with system failures can only be identified by applying different correlation algorithms, (ii) confirmed six groups of errors, (iii) identified Lustre I/O resource use counters which are correlated with occurrence of Lustre faults which are potential flags for online detection of failures, (iv) matched the dates of correlated error events and correlated resource use with the dates of compute node hangups and (v) identified two more error groups associated with compute node hang-ups. The pre-processed data will be put on the public domain in September, 2016.

Tracking System Behavior from Resource Usage Data

“…In future, we plan to integrate the information from system messages into the analysis, rather than using that for postvalidation. Other researchers have reported promising results by analyzing the two sources of information together [10], [2]. While the proposed error statistic allows for a visual tracking of the system performance, we plan to build a statistical anomaly detection model that leverages the expected behavior of the statistic to identify appropriate thresholds for flagging an anomalous event.…”

“…The four anomalies that lie above the threshold are verified using the system log data. relied on message logs [23], [16], [8], [20], [19] or resource usage data [9], [1], or both [3], [10], [2]. Since this paper focuses on the detection task, we present a brief overview of related methods that deal with detecting faults.…”

Enabling Dependability-Driven Resource Use and Message Log-Analysis for Cluster System Diagnosis

“…Recent work which use resource use data and message logs for failure diagnosis [21], [22] and error detection [23], [24] has shown increased accuracy over using message logs alone. [21] provides partial diagnosis of system failures by using resource use data to identify resource anomalies, and provides a more precise diagnosis by using message log-analysis.…”

“…[24] combines analyses of message logs and resource use data but the focus is on error detection. [23] uses message logs and resource use data to increase the error handling time window, and [22] is focused on correlating resource usage and message logs with system failures. [25] combines analysis of RAS logs and job logs but the focus is on identifying failure characteristics in a cluster system.…”

“…[25] combines analysis of RAS logs and job logs but the focus is on identifying failure characteristics in a cluster system. Previous work [21], [22], [26] have correlated errors only with failures, but there is little work which focuses on other dependability-related issues such as error propagation and error recovery. The knowledge of error propagation paths can be useful to system designers and implementators for improving the effectiveness of error recovery protocols.…”