Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

Zhang, Shenglin; Song, Lei; Zhang, Ming; Liu, Ying; Meng, Weibin; Bu, Jiahao; Yang, Sen; Sun, Yongqian; Pei, Dan; Xu, Jun; Zhang, Yuzhi

doi:10.1109/access.2020.2972691

Cited by 18 publications

(4 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Heuristic techniques are frequently coupled with fixed-depth parsing trees, as seen with Drain, FT-tree, Hue, OLMPT, and TCN-Log2Vec [14] [29] [199] [40] [125]. Many modern parsers (such as Craftsman) use parsing trees with variable depth [33].…”

Section: ) Parsingmentioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

As production system estates become larger and more complex, ensuring stability through traditional monitoring approaches becomes more challenging. Rule-based monitoring is common in industrial settings, but it has limitations. These include the difficulty of crafting rules capable of detecting unforeseen issues and the burden of manually maintaining rule sets. A potential solution to effectively manage complex system states is log anomaly detection. Workflows for log anomaly detection utilize several fundamental components. These include preprocessors for data cleansing, parsers to extract structured information from raw log data, encoding algorithms to convert extracted data into usable model input features, anomaly detection methods to isolate anomalous signals, and feedback mechanisms to incrementally improve model performance. This study explores the current state of research into online parser-supported log anomaly detection methods, investigates recent research trends, compares the performances of parser and anomaly detection methods using common public datasets and metrics, and assesses their performance evolution over time. Additionally, it classifies available methods using a newly introduced taxonomy, highlights current research gaps, and recommends future research directions.INDEX TERMS Log parsing, log template extraction, online algorithms, anomaly detection.

show abstract

Section: ) Parsingmentioning

confidence: 99%

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Lupton,

Washizaki,

Yoshioka

et al. 2024

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Therefore, R must be reviewed after its initial acquisition to eliminate redundancy. The situation may be complicated, such as assuming R [3] contains s u s v s w and R [4] contains s u s v s w s x and s u s v s w s y . The correct result has two possible orientations: 1) eliminating s u s v s w in R [3] and keeping only the two templates in R [4]; 2) s u s v s w in R [3] has the reasons to be kept independently, so it is necessary to keep all three templates.…”

Section: Algorithmmentioning

confidence: 99%

“…Thus, log analysis techniques have attracted considerable attention from researchers in the past decade. Many distinguished works have emerged, including detecting program running exceptions [1,2], monitoring network failures and traffic [3,4], diagnosing performance bottlenecks [5], and analyzing business [6] and user behavior [7].…”

Section: Introductionmentioning

confidence: 99%

An Efficient Way to Parse Logs Automatically for Multiline Events

Yu¹,

Zhang²

2023

Computer Systems Science and Engineering

View full text Add to dashboard Cite

In order to obtain information or discover knowledge from system logs, the first step is to perform log parsing, whereby unstructured raw logs can be transformed into a sequence of structured events. Although comprehensive studies on log parsing have been conducted in recent years, most assume that one event object corresponds to a single-line message. However, in a growing number of scenarios, one event object spans multiple lines in the log, for which parsing methods toward single-line events are not applicable. In order to address this problem, this paper proposes an automated log parsing method for multiline events (LPME). LPME finds multiline event objects via iterative scanning, driven by a set of heuristic rules derived from practice. The advantage of LPME is that it proposes a cohesion-based evaluation method for multiline events and a bottom-up search approach that eliminates the process of enumerating all combinations. We analyze the algorithmic complexity of LPME and validate it on four datasets from different backgrounds. Evaluations show that the actual time complexity of LPME parsing for multiline events is close to the constant time, which enables it to handle large-scale sample inputs. On the experimental datasets, the performance of LPME achieves 1.0 for recall, and the precision is generally higher than 0.9, which demonstrates the effectiveness of the proposed LPME.

show abstract

“…For instance, [82] developed a log parser based on BERT, [59] built NuLog self-supervised parser based on a transformerdecoder, and [69] built a GPT-2 based parser to preprocess Cowrie Secure Shell (SSH) honeypot logs. Several other log parsers are developed with different methods but none of them are transformer based, such as LogAider [22] deployed for IBM BlueGene HPC series [21], Craftsman [80], etc.…”

Section: Related Workmentioning

confidence: 99%

Clairvoyant

Alharthi

Jhumka

et al. 2022

Proceedings of the 36th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

System failures are expected to be frequent in the exascale era such as current Petascale systems. The health of such systems is usually determined from challenging analysis of large amounts of unstructured & redundant log data. In this paper, we leverage log data and propose Clairvoyant, a novel self-supervised (i.e., no labels needed) model to predict node failures in HPC systems based on a recent deep learning approach called transformer-decoder and the self-attention mechanism. Clairvoyant predicts node failures by (i) predicting a sequence of log events and then (ii) identifying if a failure is a part of that sequence. We carefully evaluate Clairvoyant and another state-of-the-art failure prediction approach -Desh, based on two real-world system log datasets. Experiments show that Clairvoyant is significantly better: e.g., it can predict node failures with an average Bleu, Rouge, and MCC scores of 0.90, 0.78, and 0.65 respectively while Desh scores only 0.58, 0.58, and 0.25. More importantly, this improvement is achieved with faster training and prediction time, with Clairvoyant being about 25× and 15× faster than Desh respectively.

show abstract

Efficient and Robust Syslog Parsing for Network Devices in Datacenter Networks

Cited by 18 publications

References 44 publications

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection Methods

An Efficient Way to Parse Logs Automatically for Multiline Events

Clairvoyant

Contact Info

Product

Resources

About