Real-time failure prediction in online services

Shatnawi, Mohammed Q.; Hefeeda, Mohamed

doi:10.1109/infocom.2015.7218516

Cited by 17 publications

(7 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using log files for failure detection, diagnosis and detection has been widely applied in ISP networks [13], [16], computers [14], [17]- [21], [40], [41], and online ad services [42]. Liang et al investigated the RAS event logs and developed three simple failure prediction techniques based on not only the characteristics of failure events, but also the correlation between failure events and non-failure events [40].…”

Section: Related Workmentioning

confidence: 99%

Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

et al. 2020

View full text Add to dashboard Cite

Syslog parsing is of vital importance for the detection, diagnosis and prediction of network device failures in a datacenter. A common approach to syslog parsing is to extract templates from historical syslogs, after which syslogs are matched to these templates. To address the problems in the existing syslog parsing techniques, we propose a novel framework, Craftsman, which identifies frequent combinations of (syslog) words and then applies them as templates. Craftsman empirically extracts templates accurately, is extremely efficient in template matching, and naturally supports incremental learning. To compare the performance of Craftsman and three other template learning techniques designed for network devices, we experiment them on two-years' worth of syslogs collected from network devices deployed across 10+ datacenters of a tier-one service provider. The experiments demonstrate that Craftsman achieves a close-toone accuracy (as measured by rand index), and improves the computational efficiency by 6.88 to 10.25 times in template matching, and by 730 to 6847 times in syslog parsing. It also improves the accuracy (as measured by F1 measure) of failure prediction by 13.07% to 188%. In addition, we demonstrate Craftsman's superior generality by comparing it with three widely-applied log parsing methods over five large log datasets collected from servers, distributed systems and applications.INDEX TERMS Syslog parsing, network device, prefix tree, datacenter network, frequent pattern.

show abstract

Section: Related Workmentioning

confidence: 99%

Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Many studies have been conducted for finding anomalies and their root causes [2], [4], [5] in log data. Zhong et al [3] proposed an anomaly detection method for both device and network errors with fine log time series feature creation.…”

Section: Related Workmentioning

confidence: 99%

Latent Variable Based Anomaly Detection in Network System Logs

Otomo

Kobayashi

Fukuda

et al. 2019

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nationwide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.

show abstract

“…Many studies have been conducted for finding anomalies and their root causes [1,12,14] in log data. Zhong et al [18] proposed an anomaly detection method for both device and network errors with fine log time series feature creation.…”

Section: Related Workmentioning

confidence: 99%

“…One day long data(1,440 min) for finding anomalies and their root causes have been proposed to overcome this problem [1,9,11,12,14,18]. In many cases, one first classifies the logs by their message type (i.e., log template) then treats them as statistical time series to be later processed through statistical analysis.…”

Section: Introductionmentioning

confidence: 99%

Finding Anomalies in Network System Logs with Latent Variables

Otomo

Kobayashi²,

Fukuda

et al. 2018

Proceedings of the 2018 Workshop on Big Data Analytics and Machine Learning for Data Communication Networks

View full text Add to dashboard Cite

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for largescale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 15 months of system logs obtained from a nationwide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method detects anomalies and ease troubleshooting of network system faults.

show abstract

Real-time failure prediction in online services

Cited by 17 publications

References 8 publications

Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

Latent Variable Based Anomaly Detection in Network System Logs

Finding Anomalies in Network System Logs with Latent Variables

Contact Info

Product

Resources

About