Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection

Bertero, Christophe; Roy, Matthieu; Sauvanaud, Carla; Trédan, Gilles

doi:10.1109/issre.2017.43

Cited by 110 publications

(91 citation statements)

References 20 publications

(32 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…1 presents one of the diff regions, i.e., deletion of a set of vertices T 1103-T 1109 (represented as dashed) from class Fetcher.java and addition of new vertices EXT 0-EXT 5 (double circled) from apparently third party library (prefixed with EXT). We manually investigated this diff region and found that the code fragment corresponding to templates T 1103-T 1109 has been moved from Fetcher.java to FetchItemQueue.java 4 . Inspecting FetchItemQueue.java we found that FetchItemQueues is used as logger instead of FetchItemQueue.…”

Section: Motivating Example: a Bug In Nutchmentioning

confidence: 99%

Runtime Monitoring in Continuous Deployment by Differencing Execution Behavior Model

Gupta¹,

Mandal²,

Dasgupta³

et al. 2018

Service-Oriented Computing

View full text Add to dashboard Cite

DOI to the publisher's website. • The final author version and the galley proof are versions of the publication after peer review. • The final published version features the final layout of the paper including the volume, issue and page numbers. Link to publication General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal. If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the "Taverne" license above, please follow below link for the End User Agreement:

show abstract

Section: Motivating Example: a Bug In Nutchmentioning

confidence: 99%

Runtime Monitoring in Continuous Deployment by Differencing Execution Behavior Model

Gupta¹,

Mandal²,

Dasgupta³

et al. 2018

Service-Oriented Computing

View full text Add to dashboard Cite

show abstract

“…One of the conventional approaches to deal with log preprocessing and comparison is transforming log entries to vectors, using the Word2Vec algorithm [33], [34]. A recent attempt to leverage Word2Vec for root cause classification is described in [35]. The authors propose a method for processing logs with a Word2Vec model and then using a Bayesian classifier.…”

Section: Mining Logs For Root Cause Classification and Diagnosticsmentioning

confidence: 99%

Next Stop "NoOps": Enabling Cross-System Diagnostics Through Graph-Based Composition of Logs and Metrics

Zasadzinski¹,

Solé²,

Brandon

et al. 2018

2018 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Performing diagnostics in IT systems is an increasingly complicated task, and it is not doable in satisfactory time by even the most skillful operators. Systems and their architecture change very rapidly in response to business and user demand. Many organizations see value in the maintenance and management model of NoOps that stands for No Operations. One of the implementations of this model is a system that is maintained automatically without any human intervention. The path to NoOps involves not only precise and fast diagnostics but also reusing as much knowledge as possible after the system is reconfigured or changed. The biggest challenge is to leverage knowledge on one IT system and reuse this knowledge for diagnostics of another, different system. We propose a framework of weighted graphs which can transfer knowledge, and perform high-quality diagnostics of IT systems. We encode all possible data in a graph representation of a system state and automatically calculate weights of these graphs. Then, thanks to the evaluation of similarity between graphs, we transfer knowledge about failures from one system to another and use it for diagnostics. We successfully evaluate the proposed approach on Spark, Hadoop, Kafka and Cassandra systems.

show abstract

“…Due to system logs do contain critical alarm information, they are widely used in predicting system failures [Stearley and Oliner, 2008]. Many efforts have been put into the field of anomaly detection by analysis of the log files [Stearley, 2004, Reuning, 2004, Vaarandi, 2004, Salfner et al, 2004, Prewett and James, 2004, Liang et al, 2006, 2007b,a, Oliner and Stearley, 2007, Bezerra and Wainer, 2013, Sipos et al, 2014, Ghanbari et al, 2014, Juvonen and Hamalainen, 2014, Du and Cao, 2015, Breier and Branišová, 2015, He et al, 2016, Li et al, 2017, Bertero et al, 2017, Vinayakumar et al, 2017, Landauer et al, 2018, Hu et al, 2018. Given the system logs, we concentrate on learning the correlations between KPI data and alarm cases on the basis of machine learning methods.…”

Section: Introductionmentioning

confidence: 99%

An anomaly prediction framework for financial IT systems using hybrid machine learning methods

Wang

Liu

et al. 2019

J Ambient Intell Human Comput

View full text Add to dashboard Cite

Informatization grows rapidly in all walks of life, going with the enhancement of dependence on IT systems. It is of vital importance to ensure the safe and stable running of the system especially in the field of finance. This paper puts forward a machine learningbased framework for predicting the occurrence of the alarm cases of a financial IT system. We extracted the features from the system logs then build three sub modules which are time-series prediction module, alarm classification module and level division module that composing the whole work flow. We take multiple methods to deal with the problems facing the obstacles in each module. We built the time-series prediction model in terms of time and accuracy performance. To gain higher performance, we introduced ensemble learning methods in designing alarm classifier and alleviated the class-imbalance problem in alarm level division process. The evaluation results from all sides show that our framework could be exploited for real time applications with the veracity and reliability ensured.

show abstract

Experience Report: Log Mining Using Natural Language Processing and Application to Anomaly Detection

Cited by 110 publications

References 20 publications

Runtime Monitoring in Continuous Deployment by Differencing Execution Behavior Model

Runtime Monitoring in Continuous Deployment by Differencing Execution Behavior Model

Next Stop "NoOps": Enabling Cross-System Diagnostics Through Graph-Based Composition of Logs and Metrics

An anomaly prediction framework for financial IT systems using hybrid machine learning methods

Contact Info

Product

Resources

About