An integrated framework on mining logs files for computing system management

Li, Tao; Feng, Liang; Ma, Sheng; Peng, Wei

doi:10.1145/1081870.1081972

Cited by 57 publications

(39 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Discovering time-related patterns from system logs is considered in [18], [33]. In our study, the duration time of a pattern depends on a couple of factors such as actual delay time and acceptable SLA thresholds.…”

Section: Related Workmentioning

confidence: 99%

An integrated framework for optimizing automatic monitoring systems in large IT infrastructures

Tang

Shwartz

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

The competitive business climate and the complexity of IT environments dictate efficient and cost-effective service delivery and support of IT services. These are largely achieved by automating routine maintenance procedures, including problem detection, determination and resolution. System monitoring provides an effective and reliable means for problem detection. Coupled with automated ticket creation, it ensures that a degradation of the vital signs, defined by acceptable thresholds or monitoring conditions, is flagged as a problem candidate and sent to supporting personnel as an incident ticket. This paper describes an integrated framework for minimizing false positive tickets and maximizing the monitoring coverage for system faults.In particular, the integrated framework defines monitoring conditions and the optimal corresponding delay times based on an off-line analysis of historical alerts and incident tickets. Potential monitoring conditions are built on a set of predictive rules which are automatically generated by a rule-based learning algorithm with coverage, confidence and rule complexity criteria. These conditions and delay times are propagated as configurations into run-time monitoring systems. Moreover, a part of misconfigured monitoring conditions can be corrected according to false negative tickets that are discovered by another text classification algorithm in this framework. This paper also provides implementation details of a program product that uses this framework and shows some illustrative examples of successful results.

show abstract

Section: Related Workmentioning

confidence: 99%

An integrated framework for optimizing automatic monitoring systems in large IT infrastructures

Tang

Shwartz

et al. 2013

Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

Self Cite

View full text Add to dashboard Cite

show abstract

“…Salfner et al proposed a logfile structure consisting of hierarchical numbering of event types and sources such that it is amenable for automatic log analysis techniques like clustering [16]. Li et al proposed an integrated framework to mine logs to infer temporal dependency between log events from the cumulative distribution function of the events' waiting times [12]. Palatin et al employed a distributed outlier detection algorithm HilOut, a variant of the nearest neighbor approach, over processed log files stored in different nodes of a grid system to identify misconfigured machines [15].…”

Section: Related Workmentioning

confidence: 99%

ELT: Efficient Log-based Troubleshooting System for Cloud Computing Infrastructures

2011

2011 IEEE 30th International Symposium on Reliable Distributed Systems

View full text Add to dashboard Cite

Abstract-We present an Efficient Log-based Troubleshooting(ELT) system for cloud computing infrastructures. ELT adopts a novel hybrid log mining approach that combines coarse-grained and fine-grained log features to achieve both high accuracy and low overhead. Moreover, ELT can automatically extract key log messages and perform invariant checking to greatly simplify the troubleshooting task for the system administrator. We have implemented a prototype of the ELT system and conducted an extensive experimental study using real management console logs of a production cloud system and a Hadoop cluster. Our experimental results show that ELT can achieve more efficient and powerful troubleshooting support than existing schemes. More importantly, ELT can find software bugs that cannot be detected by current cloud system management practice.

show abstract

“…Detecting message patterns may focus on frequency of occurrences [21]. Another method is to analyze time correlation and utilize text analysis algorithms [15,20].…”

Section: Security-related Eventsmentioning

confidence: 99%