Mining Causality of Network Events in Log Data

Kobayashi, Satoru; Otomo, Kazuki; Fukuda, Kensuke; Esaki, Hiroshi

doi:10.1109/tnsm.2017.2778096

Cited by 46 publications

(34 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…More specifically, we used 85% of the recording (130 days) to train our method (i.e., optimising the free parameters) and the remaining 15% (21 days) to compute the functional connectivity. To maintain consistency with the method of Kobayashi et al [3], the window (i.e., the unit of adaptation time, i.e., when probabilities are updated) was set to 1 day. The maximum delay τ max over which cross-correlations were calculated was set to 120s.…”

Section: B Validation: Pseudo-ground Truthmentioning

confidence: 99%

“…The aggregate rate of collected events is commonly high due to the very large number of monitored devices and services; a typical rate for a large-scale network deployment would be 10 6 events per second [2]. However, although the aggregate event rate is large, the rate at which individual devices emit events is extremely low such that correlating emitted events is inherently challenging; this becomes even more cumbersome in the presence of periodical informational events [3]. In addition, the vast majority of collected event data is noise and only a few of them may correlate with actionable incidents.…”

mentioning

confidence: 99%

“…Misconfiguration may result in fatal outages which could have been otherwise easily detected or predicted [8]. Kobayashi et al [3] recently proposed an This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/ inference algorithm for mining causality of network events that has been shown to have good performance.…”

mentioning

confidence: 99%

“…Our method does not rely on event pre-processing and de-duplication, therefore it is very efficient in terms of execution time and memory requirements. In contrast to [3], we take advantage of all events, including purely informational, periodical events, to infer functional connectivity between network nodes even in the absence of failures or service outages. Depending on the structure of the graph that is being produced (for example, if it consists of multiple connected components or features a strong modularity index), the output of our method can be interpreted in terms of one or many functional groups consisting of a 1 Services may be realised at different layers by in-network and end-host devices; e.g., a set of routers that form an OSPF area, a set of switches that are part of a spanning tree, or an application deployment that consists of application and database servers, load balancers and a firewall.…”

mentioning

confidence: 99%

See 3 more Smart Citations

Inferring Functional Connectivity From Time-Series of Events in Large Scale Network Deployments

Messager

Parisis

Kiss

et al. 2019

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

To respond rapidly and accurately to network and service outages, network operators must deal with a large number of events resulting from the interaction of various services operating on complex, heterogeneous and evolving networks. In this paper, we introduce the concept of functional connectivity as an alternative approach to monitoring those events. Commonly used in the study of brain dynamics, functional connectivity is defined in terms of the presence of statistical dependencies between nodes. Although a number of techniques exist to infer functional connectivity in brain networks, their straightforward application to commercial network deployments is severely challenged by: (a) non-stationarity of the functional connectivity, (b) sparsity of the time-series of events, and (c) absence of an explicit model describing how events propagate through the network or indeed whether they propagate. Thus, in this paper, we present a novel inference approach whereby two nodes are defined as forming a functional edge if they emit substantially more coincident or short-lagged events than would be expected if they were statistically independent. The output of the method is an undirected weighted graph, where the weight of an edge between two nodes denotes the strength of the statistical dependence between them. We develop a model of time-varying functional connectivity whose parameters are determined by maximising the model's predictive power from one time window to the next. We assess the accuracy, efficiency and scalability of our method on two real datasets of network events spanning multiple months and on synthetic data for which ground truth is available. We compare our method against both a general-purpose time-varying network inference method and network management specific causal inference technique and discuss its merits in terms of sensitivity, accuracy and, importantly, scalability.Index Terms-Network management, network events, functional connectivity inference.

show abstract

Section: B Validation: Pseudo-ground Truthmentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Inferring Functional Connectivity From Time-Series of Events in Large Scale Network Deployments

Messager

Parisis

Kiss

et al. 2019

IEEE Trans. Netw. Serv. Manage.

View full text Add to dashboard Cite

show abstract

“…When a system is monitored there are several sources of information one can use for cause analysis, spanning from network measures 12 down to daemon logs. 13 To perform such analysis there is a need for a large amount of data which in the past was approached in several ways, for instance, with local processing 14 or with distributed data collection. 15 Yet, cross-layer analysis is generally hard to carry-on especially in multidomain networks.…”

Section: State Of the Artmentioning

confidence: 99%

A Big Data and machine learning approach for network monitoring and security

Maccari

Passerini

2018

Security and Privacy

View full text Add to dashboard Cite

In the last decade the performances of 802.11 (Wi‐Fi) devices skyrocketed. Today it is possible to realize gigabit wireless links spanning across kilometers at a fraction of the cost of the wired equivalent. In the same period, mesh network evolved from being experimental tools confined into university labs, to systems running in several real world scenarios. Mesh networks can now provide city‐wide coverage and can compete on the market of Internet access. Yet, being wireless distributed networks, mesh networks are still hard to maintain and monitor. This paper explains how today we can perform monitoring, anomaly detection and root cause analysis in mesh networks using Big Data techniques. It first describes the architecture of a modern mesh network, it justifies the use of Big Data techniques and provides a design for the storage and analysis of Big Data produced by a large‐scale mesh network. While proposing a generic infrastructure, we focus on its application in the security domain.

show abstract

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

et al. 2021

Self Cite

View full text Add to dashboard Cite

Summary One of the ways to analyze unstructured log messages from large‐scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no common knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for comparing and combining diverse log template generation methods. Amulog consists of three key functions: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template‐matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time‐series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation‐wide academic network and demonstrate that it classifies the log data in a reasonable amount of time even with over 100,000 log template candidates. The template‐matching method in amulog also reduces 75% processing time for template generation and keeps the accuracy when combined with an existing structure‐based template generation method. In order to show the effectiveness of amulog in comparing log template generation methods, we demonstrate that the appropriate template generation methods and accuracy metrics largely depend on the purpose of further analysis by comparing the accuracy of six existing log template generation methods with 10 different accuracy metrics on amulog.

show abstract

Mining Causality of Network Events in Log Data

Cited by 46 publications

References 31 publications

Inferring Functional Connectivity From Time-Series of Events in Large Scale Network Deployments

Inferring Functional Connectivity From Time-Series of Events in Large Scale Network Deployments

A Big Data and machine learning approach for network monitoring and security

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

Contact Info

Product

Resources

About