Establishing Hypothesis for Recurrent System Failures from Cluster Log Files

Abstract

A goal for the analysis of supercomputer logs is to establish causal relationships among events which reflect significant state changes in the system. Establishing these relationships is at the heart of failure diagnosis. In principle, a log analysis tool could automate many of the manual steps systems administrators must currently use to diagnose system failures. However, supercomputer logs are unstructured, incomplete and contain considerable ambiguity so that direct discovery of causal relationships is difficult. This paper describes the second generation FDiag logbased failure diagnostics framework that provides automation of the manual failure diagnosis process and determines with high confidence, the likely cause of the failure, the components involved and the event sequences which contain the times of the causal and terminal events. FDiag extracts relevant events from the system logs, performs correlation analysis on these events and from these correlations determines the components involved and the event sequences. The diagnostics capabilities of FDiag are validated by comparing its assessments on known instances of recurrent failures on the Ranger supercomputer at the University of Texas at Austin. We believe FDiag is the first log analyzer to demonstrate this level of diagnostics capability from the system logs of an open source software stack incorporating Linux and the Lustre file system. FDiag will be put into production use for support of failure diagnosis on Ranger in September, 2011.

Mining Complex Event Patterns in Computer Networks

“…The outliers indicate particularly interesting events. Furthermore, causal analysis of event sequences, e.g., [3] complementing the association and correlation analysis, would be interesting as well, since it could directly provide more actionable knowledge.…”

“…If there is no other, later minimal occurrence [S A , E A ] ∈ µ α , where S A < S A , with the same property, then [S A , E B ] is a minimal occurrence of γ. For example, for the sequence (1, a), (2, b), (3, a), (4, c), (5, b), (6, c), (7, d) and the episodes α = (a, b, c) and β = (b, c, d), we get µ α = { [1,4], [3,6]}, µ β = { [5,7]}, γ = (a, b, c, d), and µ γ = { [3,7]}.…”

“…In the case of injective episodes, this restriction can be left out. E.g., in the sequence (1, a), (2, a) the episode α = β = (a) has the set µ α = { [1,2], [2,3]} of minimal occurrences, and the combination γ = (a, a) has only a single minimal occurrence:…”

“…[ (1, Control), (2, Manager), (3,7036), (4, Service), (5, "McAfee), (6, McShield"), (7, now), (8, is), (9, in), (10, the), (11, status) ]…”

Linking Resource Usage Anomalies with System Failures from Cluster Log Data

“…ANCOR builds upon and significantly extends a previously described log analysis system, FDiag [15]. ANCOR evaluates multiple anomaly extraction algorithms.…”