Dynamic syslog mining for network failure monitoring

Abstract

Syslog monitoring technologies have recently received vast attentions in the areas of network management and network monitoring. They are used to address a wide range of important issues including network failure symptom detection and event correlation discovery. Syslogs are intrinsically dynamic in the sense that they form a time series and that their behavior may change over time. This paper proposes a new methodology of dynamic syslog mining in order to detect failure symptoms with higher confidence and to discover sequential alarm patterns among computer devices.The key ideas of dynamic syslog mining are 1) to represent syslog behavior using a mixture of Hidden Markov Models, 2) to adaptively learn the model using an on-line discounting learning algorithm in combination with dynamic selection of the optimal number of mixture components, and 3) to give anomaly scores using universal test statistics with a dynamically optimized threshold. Using real syslog data we demonstrate the validity of our methodology in the scenarios of failure symptom detection, emerging pattern identification, and correlation discovery.

Monitoring strategies for scalable dynamic checkpointing

“…Several studies about system monitoring have demonstrated the feasibility of such an approach [28], [29]. Although those works present important results about system monitoring in HPC, they do not try to make the connection with different failure regimes and dynamic resilience runtime in order to minimize wasted time.…”

META: Multi-resolution Framework for Event Summarization

“…Different analysis tasks pay attention to different application aspects, such as system failure tracing [19,10], event correlation discovery [20,18,16], and event based trend analysis [5,6,7]. In practice, these methods are often conducted when the analysts already have some prior knowledge about the data.…”

“…property values description #types 20-100 step 20 The number of event types. #events 60k-140k step 20k The number of event occurrences.…”

A comparison of syslog and IS-IS for network failure analysis

“…As a result such datasets are rarely available in operational networks and were not available when we first analyzed the CENIC network. Thus, in many cases, including our own previous work, network analysis is conducted using syslog data [4,10,14,15,21]. Worryingly, it remains unknown what sacrifices a syslogbased approach makes in terms of accuracy when compared to the ground truth revealed via routing protocol messages.…”