With Semantics and Hidden Markov Models to an Adaptive Log File Parser

Kuhnert, Nadine; Maier, Andreas

doi:10.5121/ijnlc.2019.8603

Cited by 2 publications

(2 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, an adaptive solution using HMMs was proposed with the goal to parse texts automatically and flexibly. 35 However, their method was computationally expensive and has only been tested on one data set. Zhu et al 36 proposed a logbased anomaly detection method called LogRobust.…”

Section: Related Workmentioning

confidence: 99%

FlexParser—The adaptive log file parser for continuous results in a changing world

Rücker

Maier

2022

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi-structured log messages. State-of-the-art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes "mutation" and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel and flexible parser, called FlexParser, which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several examples. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1-Score of 0.98, it outperforms other Deep Learning methods as well as state-of-the-art unsupervised parsers.

show abstract

Section: Related Workmentioning

confidence: 99%

FlexParser—The adaptive log file parser for continuous results in a changing world

Rücker

Maier

2022

J Software Evolu Process

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, with software development not only new information can be added to log files, but also events can get modified or even dismissed. Thus, an adaptive solution has been proposed using HMMs with the goal to parse texts automatically and flexibly 34 . However, their method is computationally expensive and has only been tested on one data set.…”

mentioning

confidence: 99%

FlexParser -- the adaptive log file parser for continuous results in a changing world

Ruecker,

Maier

2021

Preprint

Self Cite

View full text Add to dashboard Cite

Any modern system writes events into files, called log files. Those contain crucial information which are subject to various analyses. Examples range from cybersecurity, intrusion detection over usage analyses to trouble shooting. Before data analysis is possible, desired information needs to be extracted first out of the semi-structured log messages. State of the art event parsing often assumes static log events. However, any modern system is updated consistently and with updates also log file structures can change. We call those changes 'mutations' and study parsing performance for different mutation cases. Latest research discovers mutations using anomaly detection post mortem, however, does not cover actual continuous parsing. Thus, we propose a novel, flexible parser, called FlexParser which can extract desired values despite gradual changes in the log messages. It implies basic text preprocessing followed by a supervised Deep Learning method. We train a stateful LSTM on parsing one event per data set. Statefulness enforces the model to learn log message structures across several messages. Our model was tested on seven different, publicly available log file data sets and various kinds of mutations. Exhibiting an average F1-Score of 0.98, it outperforms other Deep Learning methods as well as state-of-the-art unsupervised parsers.

show abstract

With Semantics and Hidden Markov Models to an Adaptive Log File Parser

Cited by 2 publications

References 10 publications

FlexParser—The adaptive log file parser for continuous results in a changing world

FlexParser—The adaptive log file parser for continuous results in a changing world

FlexParser -- the adaptive log file parser for continuous results in a changing world

Contact Info

Product

Resources

About