amulog: A General Log Analysis Framework for Diverse Template Generation Methods

Kobayashi, Satoru; Yamashiro, Yuya; Otomo, Kazuki; Fukuda, Kensuke

doi:10.23919/cnsm50824.2020.9269049

Cited by 4 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…CRF 30,47 is an NLP-like approach that estimates template structures (i.e., descriptions and variables in a log message) for log messages. CRF requires manually labeled training data for supervised learning, but for tagging, it does † The amulog implementation is highly improved from the past paper, 1 and now it only takes less than 50% processing time compared with the previous version. ‡ POP, 46 an extended method of Drain, is offline-only but can be processed in parallel.…”

Section: Template Generation Methods For Comparisonmentioning

confidence: 99%

“…In addition, we conduct a comparison of six log template generation methods to determine the applicability of amulog to various algorithms (Section 5). This comprehensive comparison work is a main new contribution from the original paper 1 . Through the comparison, we provide a number of findings on how to compare log template generation methods; In summary, we need to select appropriate measurement procedure and accuracy metrics considering the further usage of generated templates.…”

Section: Introductionmentioning

confidence: 99%

“…This comprehensive comparison work is a main new contribution from the original paper. 1 Through the comparison, we provide a number of findings on how to compare log template generation methods; In summary, we need to select appropriate measurement procedure and accuracy metrics considering the further usage of generated templates. In the comparison, we demonstrate that combining our template matching method with existing log template generation methods largely decreases the processing time and in some cases improves the accuracy of template generation.…”

mentioning

confidence: 99%

See 2 more Smart Citations

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

et al. 2021

Self Cite

View full text Add to dashboard Cite

Summary One of the ways to analyze unstructured log messages from large‐scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no common knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for comparing and combining diverse log template generation methods. Amulog consists of three key functions: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template‐matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time‐series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation‐wide academic network and demonstrate that it classifies the log data in a reasonable amount of time even with over 100,000 log template candidates. The template‐matching method in amulog also reduces 75% processing time for template generation and keeps the accuracy when combined with an existing structure‐based template generation method. In order to show the effectiveness of amulog in comparing log template generation methods, we demonstrate that the appropriate template generation methods and accuracy metrics largely depend on the purpose of further analysis by comparing the accuracy of six existing log template generation methods with 10 different accuracy metrics on amulog.

show abstract

Section: Template Generation Methods For Comparisonmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

Landscape of Automated Log Analysis: A Systematic Literature Review and Mapping Study

Korzeniowski

Goczyła

2022

IEEE Access

View full text Add to dashboard Cite

Logging is a common practice in software engineering to provide insights into working systems. The main uses of log files have always been failure identification and root cause analysis. In recent years, novel applications of logging have emerged that benefit from automated analysis of log files, for example, real-time monitoring of system health, understanding users' behavior, and extracting domain knowledge. Although nearly every software system produces log files, the biggest challenge in log analysis is the lack of a common standard for both the content and format of log data. This paper provides a systematic review of recent literature (covering the period between 2000 and June 2021, concentrating primarily on the last five years of this period) related to automated log analysis. Our contribution is threefold: we present an overview of various research areas in the field; we identify different types of log files that are used in research, and we systematize the content of log files. We believe that this paper serves as a valuable starting point for new researchers in the field, as well as an interesting overview for those looking for other ways of utilizing log information.

show abstract

System Log Parsing: A Survey

Zhang

Qiu

Castellano

et al. 2023

IEEE Trans. Knowl. Data Eng.

View full text Add to dashboard Cite

amulog: A General Log Analysis Framework for Diverse Template Generation Methods

Cited by 4 publications

References 27 publications

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

Landscape of Automated Log Analysis: A Systematic Literature Review and Mapping Study

System Log Parsing: A Survey

Contact Info

Product

Resources

About