Tools and Benchmarks for Automated Log Parsing

Abstract

Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The increasing scale and complexity of modern software systems, however, make the volume of logs explodes. In many cases, the traditional way of manual log inspection becomes impractical. Many recent studies, as well as industrial tools, resort to powerful text search and machine learning-based analytics solutions. Due to the unstructured nature of logs, a first crucial step is to parse log messages into structured data for subsequent analysis. In recent years, automated log parsing has been widely studied in both academia and industry, producing a series of log parsers by different techniques. To better understand the characteristics of these log parsers, in this paper, we present a comprehensive evaluation study on automated log parsing and further release the tools and benchmarks for easy reuse. More specifically, we evaluate 13 log parsers on a total of 16 log datasets spanning distributed systems, supercomputers, operating systems, mobile systems, server applications, and standalone software. We report the benchmarking results in terms of accuracy, robustness, and efficiency, which are of practical importance when deploying automated log parsing in production. We also share the success stories and lessons learned in an industrial application at Huawei. We believe that our work could serve as the basis and provide valuable guidance to future research and deployment of automated log parsing.

Logzip: Extracting Hidden Structures via Iterative Clustering for Log Compression

“…Therefore, template extraction from software logs is the most widely-applicable approach and thus logzip proposes an iterative clustering algorithm to extract templates from logs automatically. According to the benchmark by Zhu et al [17], existing template extraction approaches could perform accurately on software logs. However, these methods require all the historical logs as input, leading to severe inefficiency and hindering them from adoption in practice.…”

“…LKE and IPLoM are offline parsers, and SHISO and Drain could parse online in a streaming manner. These parsers are evaluated and compared in the benchmark by Zhu et al [17]. The parsers could extract hidden structures but they take all logs as input thus are not efficient compared with the proposed ISE.…”

“…In particular, loghub hosts a large collection of system log datasets. Logparser provides a toolkit and benchmarks for automated log parsing [17], [46], [47]. Loglizer implements a number of machine-learning based log analysis techniques for automated anomaly detection [18], [48].…”

“…We evaluate logzip on five real-world log datasets (i.e., HDFS, Spark, Andriod, Windows, and Thunderbird) from the loghub repository [17]. They are chosen to span multiple types of systems, including distributed systems, mobile systems, operating systems, and supercomputing systems, and also have different sizes ranging from 1.58 GB to 29.6 GB.…”

An online log template extraction method based on hierarchical clustering

“…The log data set BlueGene/L (BGL) is a public, partially tagged data set from IBM's renowned high-performance computing lab (Lawrence Livermore National Labs, LLNL). The BGL data set [33,34] contains 4,747,963 raw log messages of 215 days, with a size of 708 M. HPC data set [33,34] is a high-performance cluster log collected by Los Alamos National Labs, containing 433,490 raw log messages. HDFS data set [33,34] is a log data set collected from the 203 node cluster of the Amazon EC2 platform.…”