An Analysis of Burstiness and Causality of System Logs

Otomo, Kazuki; Kobayashi, Satoru; Fukuda, Kensuke; Esaki, Hiroshi

doi:10.1145/3154970.3154973

Cited by 4 publications

(4 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper focuses on discovering event correlations in an alert event sequence. According to our investigation, most correlation mining methods are generally based on pairwise correlations and assume that multiple correlations are composed of pairwise correlations from bottom to top [6,8,9]. As shown in Fig.…”

Section: Motivationmentioning

confidence: 99%

“…Zhang et al [22] proposed CloudPin, a root cause localization framework employing a multidimensional algorithm with three submodels for prediction deviation, anomaly amplitude, and shape similarity. Otomo et al [9] focused on the burstiness and causality of log time series data to extract meaningful information for troubleshooting. Zhang et al [23] proposed a root cause analysis framework called CloudRCA, which uses heterogeneous multisource data, including KPIs, logs, and topologies.…”

Section: Correlation and Causality Miningmentioning

confidence: 99%

See 1 more Smart Citation

AlertInsight: Mining Multiple Correlation For Alert Reduction

Yu¹,

Zhang²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

Modern cloud services are monitored by numerous multidomain and multivendor monitoring tools, which generate massive numbers of alerts and events that are not actionable. These alerts usually carry isolated messages that are missing service contexts. Administrators become inundated with tickets caused by such alert events when they are routed directly to incident management systems. Noisy alerts increase the risk of crucial warnings going undetected and leading to service outages. One of the feasible ways to cope with the above problems involves revealing the correlations behind a large number of alerts and then aggregating the related alerts according to their correlations. Based on these guidelines, AlertInsight, a framework for alert event reduction, is proposed in this paper. In AlertInsight, the correlations among event sources are found by mining a sequence of historical events. Then, event correlation knowledge is employed to build an online detector targeting the correlated events that are hidden in the event stream. Finally, the correlated events are aggregated into a single high-level event for alert reduction. Because of the weaknesses of the commonly used pairwise correlation analysis methods in complex environments, an innovative approach for multiple correlation mining, which overcomes computational complexity challenges by scanning panoramic views of historical episodes from the perspective of holism, is proposed in this paper. In addition, a neural network-based correlated event detector that can learn the event correlation knowledge generated from correlation mining and then detect the correlated events in a sequence online is proposed. Experiments are conducted to test the effectiveness of AlertInsight. The experimental results (precision = 0.92, recall = 0.93, and F1-score = 0.93) demonstrate the performance of AlertInsight for the recognition of multiple correlated alerts and its competence for alert reduction.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Correlation and Causality Miningmentioning

confidence: 99%

AlertInsight: Mining Multiple Correlation For Alert Reduction

Yu¹,

Zhang²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

show abstract

“…Otomo et al [24] conducted log analysis focusing on the burstiness and causality of log time series. They first removed trivial logs (e.g., periodic and very frequent logs) to prevent detecting trivial bursts then detected log bursts with Kleinberg's burst detection [8].…”

Section: Burst Detectionmentioning

confidence: 99%

Latent Variable Based Anomaly Detection in Network System Logs

Otomo

Kobayashi

Fukuda

et al. 2019

IEICE Trans. Inf. & Syst.

Self Cite

View full text Add to dashboard Cite

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nationwide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.

show abstract

“…The three algorithms are summarized in Table 2. 4.2.1 Burst detection. Otomo et al [13] conducted log analysis focusing on the burstiness and causality of log time series. They first removed trivial logs (e.g., periodic and very frequent logs) to prevent detecting trivial bursts then detected log bursts with Kleinberg's burst detection [8].…”

Section: Baseline Algorithmsmentioning

confidence: 99%

Finding Anomalies in Network System Logs with Latent Variables

Otomo

Kobayashi²,

Fukuda

et al. 2018

Proceedings of the 2018 Workshop on Big Data Analytics and Machine Learning for Data Communication Networks

Self Cite

View full text Add to dashboard Cite

System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for largescale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 15 months of system logs obtained from a nationwide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method detects anomalies and ease troubleshooting of network system faults.

show abstract

An Analysis of Burstiness and Causality of System Logs

Cited by 4 publications

References 14 publications

AlertInsight: Mining Multiple Correlation For Alert Reduction

AlertInsight: Mining Multiple Correlation For Alert Reduction

Latent Variable Based Anomaly Detection in Network System Logs

Finding Anomalies in Network System Logs with Latent Variables

Contact Info

Product

Resources

About