LogStamp

Abstract

Logs are one of the most critical data for service management. It contains rich runtime information for both services and users. Since size of logs are often enormous in size and have free handwritten constructions, a typical log-based analysis needs to parse logs into structured format first. However, we observe that most existing log parsing methods cannot parse logs online, which is essential for online services. In this paper, we present an automatic online log parsing method, name as LogStamp. We extensively evaluate LogStamp on five public datasets to demonstrate the effectiveness of our proposed method. The experiments show that our proposed method can achieve high accuracy with only a small portion of the training set. For example, it can achieve an average accuracy of 0.956 when using only 10% of the data training.

Log-based Anomaly Detection Without Log Parsing

“…During the pre-training phase, the model learns to predict randomly masked input tokens. Based on the idea that log is actually a natural language sequence [2], some studies [16], [33], [34] have leveraged pretrained language models such as BERT [31] to analyse log data. Language models are pre-trained on large-scale unlabelled corpus and then fine-tuned to perform downstream tasks.…”

“…Data-driven Log Parsing: Log parsing has become an active research topic in recent years [12], [13], [20], [60]. Recently, to address the limitations of traditional log parsers and improve the parsing accuracy, some approaches [34], [19] proposed to use token classification for log parsing. LogStamp [34] converts the log parsing task into a sequence labelling problem.…”

“…Recently, to address the limitations of traditional log parsers and improve the parsing accuracy, some approaches [34], [19] proposed to use token classification for log parsing. LogStamp [34] converts the log parsing task into a sequence labelling problem. It leverages the BERT [31] model to classify words in log messages.…”

“…To achieve better log parsing accuracy, many data-driven approaches, such as those based on clustering [13], [14], frequent pattern mining [15], [16], and heuristics [9], [10], [17], have been proposed to automatically distinguish the constant and variable parts of log messages [11], [12], [18]. Recent studies adopt pre-trained language models for representing [3], [18], [19] log data. However, these methods still require either training models from scratch [20] or tuning a pre-trained language model with labelled data [3], [18], which could be impractical due to the scarcity of computing resources and labelled data.…”

DLLog: An Online Log Parsing Approach for Large-Scale System

“…Importantly, developers have not established a unifed standard for log format, making traditional manual log parsing methods impractical. Static code-based parsing methods exhibit high limitations [7][8][9] because obtaining system source code during the parsing process is challenging. While frequent pattern mining-based log parsing methods demonstrate competitive parsing efciency, they struggle to match rare logs with low frequency to any log template, resulting in suboptimal parsing results [10][11][12].…”