Improving Log-Based Anomaly Detection with Component-Aware Analysis

Yin, Kun; Yan, Meng; Xu, Lan; Zhou, Xu; Zhao, Li; Yang, Dan; Zhang, Xiaohong

doi:10.1109/icsme46990.2020.00069

Cited by 22 publications

(4 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, session windows can be used when a session identifier is available (e.g., the block ID present in the HDFS dataset). Generally, fixed or sliding windows are selected when these identifiers are unavailable (as with the ThunderBird dataset) [236]. Therefore, the window type is less a feature of the log anomaly detection method and more an adaptation based on the log being monitored.…”

Section: ) Encodingmentioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

As production system estates become larger and more complex, ensuring stability through traditional monitoring approaches becomes more challenging. Rule-based monitoring is common in industrial settings, but it has limitations. These include the difficulty of crafting rules capable of detecting unforeseen issues and the burden of manually maintaining rule sets. A potential solution to effectively manage complex system states is log anomaly detection. Workflows for log anomaly detection utilize several fundamental components. These include preprocessors for data cleansing, parsers to extract structured information from raw log data, encoding algorithms to convert extracted data into usable model input features, anomaly detection methods to isolate anomalous signals, and feedback mechanisms to incrementally improve model performance. This study explores the current state of research into online parser-supported log anomaly detection methods, investigates recent research trends, compares the performances of parser and anomaly detection methods using common public datasets and metrics, and assesses their performance evolution over time. Additionally, it classifies available methods using a newly introduced taxonomy, highlights current research gaps, and recommends future research directions.INDEX TERMS Log parsing, log template extraction, online algorithms, anomaly detection.

show abstract

Section: ) Encodingmentioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…In addition, there are rule-based [15], tree-based [16], statistical [17], as well as methods based on clustering [18,19,20,21]. Whereas recent anomaly detection methods are mainly designed with neural networks [22,23,24] and based on encoder architectures [25,26]. Equally, recent methods utilizing the attention mechanism that is often used in encoder architectures [27].…”

Section: Related Workmentioning

confidence: 99%

A2Log: Attentive Augmented Log Anomaly Detection

Wittkopp¹,

Acker²,

Nedelkoski³

et al. 2022

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

Anomaly detection becomes increasingly important for the dependability and serviceability of IT services. As log lines record events during the execution of IT services, they are a primary source for diagnostics. Thereby, unsupervised methods provide a significant benefit since not all anomalies can be known at training time. Existing unsupervised methods need anomaly examples to obtain a suitable decision boundary required for the anomaly detection task. This requirement poses practical limitations. Therefore, we develop A2Log, which is an unsupervised anomaly detection method consisting of two steps: Anomaly scoring and anomaly decision. First, we utilize a self-attention neural network to perform the scoring for each log message. Second, we set the decision boundary based on data augmentation of the available normal training data. The method is evaluated on three publicly available datasets and one industry dataset. We show that our approach outperforms existing methods. Furthermore, we utilize available anomaly examples to set optimal decision boundaries to acquire strong baselines. We show that our approach, which determines decision boundaries without utilizing anomaly examples, can reach scores of the strong baselines.

show abstract

“…-Failures [24], [34]- [36], [38]- [41], [55], [60], [64], [71], [73] OpenStack [20] 2017 Virtual machines Failures [20], [26], [34]- [36], [42], [44], [50], [52], [63], [76]- [78] Hadoop [90] 2016 High-perf. comp.…”

Section: Data Setmentioning

confidence: 99%

Deep Learning for Anomaly Detection in Log Data: A Survey

Landauer¹,

Onder²,

Skopik³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatic log file analysis enables early detection of relevant incidents such as system failures. In particular, selflearning anomaly detection techniques capture patterns in log data and subsequently report unexpected log event occurrences to system operators without the need to provide or manually model anomalous scenarios in advance. Recently, an increasing number of approaches leveraging deep learning neural networks for this purpose have been presented. These approaches have demonstrated superior detection performance in comparison to conventional machine learning techniques and simultaneously resolve issues with unstable data formats. However, there exist many different architectures for deep learning and it is nontrivial to encode raw and unstructured log data to be analyzed by neural networks. We therefore carry out a systematic literature review that provides an overview of deployed models, data pre-processing mechanisms, anomaly detection techniques, and evaluations. The survey does not quantitatively compare existing approaches but instead aims to help readers understand relevant aspects of different model architectures and emphasizes open issues for future work.

show abstract

Improving Log-Based Anomaly Detection with Component-Aware Analysis

Cited by 22 publications

References 18 publications

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

A2Log: Attentive Augmented Log Anomaly Detection

Deep Learning for Anomaly Detection in Log Data: A Survey

Contact Info

Product

Resources

About