UniParser: A Unified Log Parser for Heterogeneous Log Data

Liu, Yudong; Zhang, Xu; He, Shilin; Zhang, Hongyu; Li, Liqun; Kang, Yu; Yong, Xu; Ma, Minghua; Lin, Qingwei; Dang, Yingnong; Rajmohan, Saravan; Zhang, Dongmei

doi:10.1145/3485447.3511993

Cited by 45 publications

(29 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, a few works brings attention to the semantics of templates [15,16,18]. These methods apply deep learning to enrich the semantics of templates, which require a large amount of training data.…”

Section: Semantic Aware Log Parsingmentioning

confidence: 99%

“…With the growing scale and complexity of Web applications such as cloud computing and micro-service systems [15,16,18], system event logs (we call them logs for brevity) provide first-hand information for engineers to monitor the health status of the system and troubleshoot [13]. The raw logs are of a vast volume containing much redundant information, making it difficult for engineers to analyze them.…”

Section: Introductionmentioning

confidence: 99%

“…A number of template mining algorithms were proposed [2,13,16,[24][25][26], which enable the automatic extraction of templates. However, previous template mining algorithms do not support interactive log analysis, which is highly desirable in real-world diagnosis or troubleshooting of Web applications.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Interactive Log Parsing via Light-weight User Feedbacks

Wang¹,

Xie²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

Template mining is one of the foundational tasks to support log analysis, which supports the diagnosis and troubleshooting of large scale Web applications. This paper develops a human-in-the-loop template mining framework to support interactive log analysis, which is highly desirable in real-world diagnosis or troubleshooting of Web applications but yet previous template mining algorithms fails to support it. We formulate three types of light-weight user feedbacks and based on them we design three atomic humanin-the-loop template mining algorithms. We derive mild conditions under which the outputs of our proposed algorithms are provably correct. We also derive upper bounds on the computational complexity and query complexity of each algorithm. We demonstrate the versatility of our proposed algorithms by combining them to improve the template mining accuracy of five representative algorithms over sixteen widely used benchmark datasets.

show abstract

Section: Semantic Aware Log Parsingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Interactive Log Parsing via Light-weight User Feedbacks

Wang¹,

Xie²,

Li³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…• Accuracy: Existing log parsers extract common parts as templates using statistical features (e.g., word length, log length, frequency) and ignore the semantic meaning of log messages. Without considering the semantic information, traditional log parsers tend to misidentify parameters as keywords [19] in many cases (e.g., when encountering previously unseen log templates).…”

Section: Introductionmentioning

confidence: 99%

“…For example, without the pre-processing step, the parsing accuracy can decline by 6.1%-73.5% [21]. When applying the existing log parsers to a new log dataset, due to different logging formats and behaviours, time-consuming adjustment of hyper-parameters and regular expressions are needed [19]. To overcome the above-mentioned limitations, in this paper, we propose LogPPT, a novel log parser with prompt-based few-shot learning.…”

Section: Introductionmentioning

confidence: 99%

Log-based Anomaly Detection Without Log Parsing

Zhang

2021

2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)

Self Cite

106

View full text Add to dashboard Cite

Logs generated by large-scale software systems provide crucial information for engineers to understand the system status and diagnose problems of the systems. Log parsing, which converts raw log messages into structured data, is the first step to enabling automated log analytics. Existing log parsers extract the common part as log templates using statistical features. However, these log parsers often fail to identify the correct templates and parameters because: 1) they often overlook the semantic meaning of log messages, and 2) they require domain-specific knowledge for different log datasets. To address the limitations of existing methods, in this paper, we propose LogPPT to capture the patterns of templates using prompt-based few-shot learning. LogPPT utilises a novel prompt tuning method to recognise keywords and parameters based on a few labelled log data. In addition, an adaptive random sampling algorithm is designed to select a small yet diverse training set. We have conducted extensive experiments on 16 public log datasets. The experimental results show that LogPPT is effective and efficient for log parsing.

show abstract