Data-Driven Techniques in Computing System Management

Li, Tao; Zeng, Chunqiu; Jiang, Yexi; Zhou, Wubai; Tang, Liang; Liu, Zheng; Huang, Yue

doi:10.1145/3092697

Cited by 19 publications

(21 citation statements)

References 93 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The primary goal of this "knowledge" extraction is to assist in diagnosing the underlying problems responsible for log-messages and events. Two excellent resources for the large body of work done in the area are [1], [2]. Next, we outline some of the key challenges in this knowledge extraction, associated research in the area, and our problem in the context of existing work.…”

Section: B Context and Related Workmentioning

confidence: 99%

“…Thus it is very useful to have a global summarized snapshot of messages based on logs. Most works in this area exploit the inter-arrival distribution and cooccurrence of events [16], [17], [18], [19], [2] to produce summarized correlation between events. These methods are useful when the event-stream is available and possible eventtypes are known in advance.…”

Section: B Context and Related Workmentioning

confidence: 99%

“…For example, a syslog message "service wqffv failed due to connection failure to IP address a.b.c.d using port 8231" would reduce to template "service wqffv failed due to connection failure to IP address * using port * ." There are many existing methods to extract such templates [1], [2], ranging from tree-based methods to NLP based methods. In our work, we have a template-extraction pre-processing step before applying our methods.…”

Section: Problem Statement and Preliminariesmentioning

confidence: 99%

“…• The number of E events which generated the data set. • The signatures of these events: p (1) , p (2) , . .…”

Section: Problem Statement and Preliminariesmentioning

confidence: 99%

“…The logs from different network elements could be pooled together in a central database for subsequent analysis. While mining error logs have been studied extensively in different contexts, (see [1], [2] for excellent surveys; also see Section I-B), there are some fundamental differences in our setting. Modern data center and communication networks consist of components bought from different vendors, and each component is designed to generate an error message when it cannot execute a job.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learning Latent Events From Network Message Logs

Satpathi

Deb

Srikant

et al. 2019

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

We consider the problem of separating error messages generated in large distributed data center networks into error events. In such networks, each error event leads to a stream of messages generated by hardware and software components affected by the event. These messages are stored in a giant message log. We consider the unsupervised learning problem of identifying the signatures of events that generated these messages; here, the signature of an error event refers to the mixture of messages generated by the event. One of the main contributions of the paper is a novel mapping of our problem which transforms it into a problem of topic discovery in documents. Events in our problem correspond to topics and messages in our problem correspond to words in the topic discovery problem. However, there is no direct analog of documents. Therefore, we use a nonparametric change-point detection algorithm, which has linear computational complexity in the number of messages, to divide the message log into smaller subsets called episodes, which serve as the equivalents of documents. After this mapping has been done, we use a well-known algorithm for topic discovery, called LDA, to solve our problem. We theoretically analyze the changepoint detection algorithm, and show that it is consistent and has low sample complexity. We also demonstrate the scalability of our algorithm on a real data set consisting of 97 million messages collected over a period of 15 days, from a distributed data center network which supports the operations of a large wireless service provider.

show abstract

Section: B Context and Related Workmentioning

confidence: 99%