Learning Latent Events From Network Message Logs

Abstract

We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a nonparametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the changepoint detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.

Summarizing Unstructured Logs in Online Services

“…We apply the compression ratio, i.e., size of summaries size of original logs , to evaluate the log compression performance. We compare LogSummary with three extractive summarization methods, namely, TF-IDF [38], LDA [13], TextRank (sentence summary) [27]).…”

“…Satpathi et al [13] proposed the first closely related work in our scenario. Their focus is different from ours as they aim to mine the distribution of messages for each anomalous event and their occurrences in the logs getting an event signature that represents it formed by keywords.…”

“…It is very difficult for operators to timely comprehend these new types of logs. Although several works have been proposed for log compression [3], [11] or system interpretability through logs [12], [13], to the best of our knowledge, there is no previous work aiming to help operators efficiently and effectively summarize a given log sequence in an interpretable and readable manner.…”

Failure Detection Using Semantic Analysis and Attention-Based Classifier Model for IT Infrastructure Log Data

“…The system log is the first place where the system administrator investigates issues on failure alerts. Consequently, system logs have been widely used in anomaly and failure detection or prediction because of their directness and usefulness [5][6] [7] [8]. Complex computer systems record a massive collection of logs that can make useful information available; on the other hand, analyzing colossal data is challenging.…”

MoniLog: An Automated Log-Based Anomaly Detection System for Cloud Computing Infrastructures

“…Because of its simplicity and effectiveness, logging has been commonly adopted in practice [6]. Logs are therefore one of the most valuable data sources for anomaly detection [7]- [9].…”