LogLens: A Real-Time Log Analysis System

Debnath, Biplob; Solaimani, Mohiuddin; Gulzar, Muhammad Ali; Arora, Nipun; Lumezanu, Cristian; Xu, Jing; Zong, Bo; Zhang, Hui; Jiang, Guofei; Khan, Latifur

doi:10.1109/icdcs.2018.00105

Cited by 79 publications

(34 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After feature extraction, several machine learning models are used for anomaly detection, such as Regression [30], Random Forest [29,32], and Clustering [33,38,42].…”

Section: Anomaly Detectionmentioning

confidence: 99%

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Wang

Tang

et al. 2020

Sensors

View full text Add to dashboard Cite

Log anomaly detection is an efficient method to manage modern large-scale Internet of Things (IoT) systems. More and more works start to apply natural language processing (NLP) methods, and in particular word2vec, in the log feature extraction. Word2vec can extract the relevance between words and vectorize the words. However, the computing cost of training word2vec is high. Anomalies in logs are dependent on not only an individual log message but also on the log message sequence. Therefore, the vector of words from word2vec can not be used directly, which needs to be transformed into the vector of log events and further transformed into the vector of log sequences. To reduce computational cost and avoid multiple transformations, in this paper, we propose an offline feature extraction model, named LogEvent2vec, which takes the log event as input of word2vec to extract the relevance between log events and vectorize log events directly. LogEvent2vec can work with any coordinate transformation methods and anomaly detection models. After getting the log event vector, we transform log event vector to log sequence vector by bary or tf-idf and three kinds of supervised models (Random Forests, Naive Bayes, and Neural Networks) are trained to detect the anomalies. We have conducted extensive experiments on a real public log dataset from BlueGene/L (BGL). The experimental results demonstrate that LogEvent2vec can significantly reduce computational time by 30 times and improve accuracy, comparing with word2vec. LogEvent2vec with bary and Random Forest can achieve the best F1-score and LogEvent2vec with tf-idf and Naive Bayes needs the least computational time.

show abstract

“…After feature extraction, several machine learning models are used for anomaly detection, such as Regression [30], Random Forest [29,32], and Clustering [33,38,42].…”

Section: Anomaly Detectionmentioning

confidence: 99%

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

Wang

Tang

et al. 2020

Sensors

View full text Add to dashboard Cite

show abstract

“…Aver − sim(S, P)+ � J(S, S pi ) (4) end for (5) for j � 1 ⟶ n do (6) Aver − sim(S, F)+ � J(S, S fi ) (7) end for (8) if Aver − sim(S, P)+ � J(S, S pi ) ≥ Aver − sim(S, F)+ � J(S, S fi ) then (9) return normal ( 10) else (11) return abnormal (12) end if (21) end function ALGORITHM 4: e strategy of anomaly detection. As described in Section 3, the log-based anomaly detection method proposed in this paper is based on kNN algorithm, which is an outlier detection method in machine learning.…”

Section: Log Datamentioning

confidence: 99%

“…Reference [6] uses a clustering algorithm to sort the log sequences, which is an outlier detection method and the same as our method. e methods in [8,9] are not outlier detection methods; [8] uses an anomaly detection method based on finite state automaton and [9] uses the information entropy of log messages for identifying exceptions.…”

Section: Log Datamentioning

confidence: 99%

See 1 more Smart Citation

A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection

Wang

Song

Yang

2020

Scientific Programming

View full text Add to dashboard Cite

Using the k-nearest neighbor (kNN) algorithm in the supervised learning method to detect anomalies can get more accurate results. However, when using kNN algorithm to detect anomaly, it is inefficient at finding k neighbors from large-scale log data; at the same time, log data are imbalanced in quantity, so it is a challenge to select proper k neighbors for different data distributions. In this paper, we propose a log-based anomaly detection method with efficient selection of neighbors and automatic selection of k neighbors. First, we propose a neighbor search method based on minhash and MVP-tree. The minhash algorithm is used to group similar logs into the same bucket, and MVP-tree model is built for samples in each bucket. In this way, we can reduce the effort of distance calculation and the number of neighbor samples that need to be compared, so as to improve the efficiency of finding neighbors. In the process of selecting k neighbors, we propose an automatic method based on the Silhouette Coefficient, which can select proper k neighbors to improve the accuracy of anomaly detection. Our method is verified on six different types of log data to prove its universality and feasibility.

show abstract

“…Lastly, Debnath et al [32] envision LogLens, an anomaly detection system that analyzes log files in real time. It works with minimal or even no target system knowledge and user specification: it learns normal behavior and builds a finitestate machine that captures it, with which it then detects anomalies.…”

Section: Introductionmentioning

confidence: 99%

Automating Root Cause Analysis via Machine Learning in Agile Software Testing Environments

Kahles

Torronen

Huuhtanen

et al. 2019

2019 12th IEEE Conference on Software Testing, Validation and Verification (ICST)

View full text Add to dashboard Cite

We apply machine learning to automate the root cause analysis in agile software testing environments. In particular, we extract relevant features from raw log data after interviewing testing engineers (human experts). Initial efforts are put into clustering the unlabeled data, and despite obtaining weak correlations between several clusters and failure root causes, the vagueness in the rest of the clusters leads to the consideration of labeling. A new round of interviews with the testing engineers leads to the definition of five ground-truth categories. Using manually labeled data, we train artificial neural networks that either classify the data or pre-process it for clustering. The resulting method achieves an accuracy of 88.9%. The methodology of this paper serves as a prototype or baseline approach for the extraction of expert knowledge and its adaptation to machine learning techniques for root cause analysis in agile environments.

show abstract

LogLens: A Real-Time Log Analysis System

Cited by 79 publications

References 12 publications

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

LogEvent2vec: LogEvent-to-Vector Based Anomaly Detection for Large-Scale Logs in Internet of Things

A Log-Based Anomaly Detection Method with Efficient Neighbor Searching and Automatic K Neighbor Selection

Automating Root Cause Analysis via Machine Learning in Agile Software Testing Environments

Contact Info

Product

Resources

About