LogDP: Combining Dependency and Proximity for Log-Based Anomaly Detection

Xie, Yongzheng; Zhang, Hongyu; Zhang, Bo; Babar, Muhammad Ali; Lu, Sha

doi:10.1007/978-3-030-91431-8_47

Cited by 9 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Traditional ML-based methods, such as LR [8], SVM [9], PCA (Principal Component Analysis) [54] and LogCluster [10], are often more efficient compared with deep learning based methods in terms of time costs. Invariant relation mining-based methods, such as Invariants Mining [5], ADR [2] and LogDP [11], have the advantages of low labeling cost and interpretability because they usually work in semi-supervised or unsupervised mode and can capture meaningful relations. Despite these advantages, quantitative-based methods tend to suffer from unstable performance in some specific cases because they cannot capture sequential patterns and semantic information between log events.…”

Section: B Log-based Anomaly Detectionmentioning

confidence: 99%

See 1 more Smart Citation

LogGD:Detecting Anomalies from System Logs by Graph Neural Networks

Xie¹,

Zhang²,

Babar³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Log analysis is one of the main techniques engineers use to troubleshoot faults of large-scale software systems. During the past decades, many log analysis approaches have been proposed to detect system anomalies reflected by logs. They usually take log event counts or sequential log events as inputs and utilize machine learning algorithms including deep learning models to detect system anomalies. These anomalies are often identified as violations of quantitative relational patterns or sequential patterns of log events in log sequences. However, existing methods fail to leverage the spatial structural relationships among log events, resulting in potential false alarms and unstable performance. In this study, we propose a novel graph-based log anomaly detection method, LogGD, to effectively address the issue by transforming log sequences into graphs. We exploit the powerful capability of Graph Transformer Neural Network, which combines graph structure and node semantics for logbased anomaly detection. We evaluate the proposed method on four widely-used public log datasets. Experimental results show that LogGD can outperform state-of-the-art quantitative-based and sequence-based methods and achieve stable performance under different window size settings. The results confirm that LogGD is effective in log-based anomaly detection.

show abstract

Section: B Log-based Anomaly Detectionmentioning

confidence: 99%

“…We call such an approach quantitative-based approach. The representative methods of this approach include LR [8], SVM [9], LogCluster [10], Invariants Mining [5], ADR [2], and LogDP [11]. However, these methods tend to suffer from unstable performance on different datasets since their input only contains quantitative statistics.…”

Section: Introductionmentioning

confidence: 99%

LogGD:Detecting Anomalies from System Logs by Graph Neural Networks

Xie¹,

Zhang²,

Babar³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the event of a system failure, the most straightforward approach for maintenance personnel is to perform log analysis. However, manually identifying anomalies based on massive log data has become impractical [4,5]. Due to the rapid growth in log size, log analysis by experienced experts has become increasingly challenging [6].…”

Section: Introductionmentioning

confidence: 99%

LogEDL: Log Anomaly Detection via Evidential Deep Learning

Duan,

Xue,

Sun

et al. 2024

Preprint

View full text Add to dashboard Cite

With advancements in digital technologies such as 5G communications, big data, and cloud computing, the components of network operation systems have become increasingly complex, significantly complicating system monitoring and maintenance. Correspondingly, automated log anomaly detection has become a crucial means to ensure stable network operation and protect them from malicious attacks or failures. Conventional machine learning and deep learning methods assume consistent distributions between the training and testing data, adhering to a closed-set recognition paradigm. Nevertheless, in realistic scenarios, systems may encounter new anomalies that were not present in the training data, especially in log anomaly detection. Inspired by evidential learning, we propose a novel anomaly detector called LogEDL, which supervises training of the model through an evidential loss function. Unlike traditional loss functions, the evidential loss function not only focuses on correct classification, but also quantifies the uncertainty of predictions. This enhances the robustness and accuracy of the model in handling anomaly detection tasks, while achieving functionality similar to open-set recognition. To evaluate the proposed LogEDL method, we conduct extensive experiments on three datasets, i.e., HDFS, BGL, and Thunderbird, to detect anomalous log sequences. Experimental results demonstrate that our proposed LogEDL achieves state-of-the-art performance in anomaly detection.

show abstract