Causal analysis of network logs with layered protocols and topology knowledge

Kobayashi, Satoru; Otomo, Kazuki; Fukuda, Kensuke

doi:10.23919/cnsm46954.2019.9012718

Cited by 9 publications

(6 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In amulog, the optional annotation information can be parsed with log2seq and used as annotation tags of log templates. The annotation tags can be used for search and classification of log time‐series in further analysis (e.g., for domain knowledge in causal analysis 14 ).…”

Section: Discussionmentioning

confidence: 99%

“…Log messages belonging to a log template usually contain information on a common system behavior, so template generation is an effective way to classify log messages with their behaviors. In contrast to full‐text search, 4‐6 which is another major approach used for keyword‐based analysis, the template‐based approach is suitable for aggregating and digesting the system behaviors, enabling time‐series‐based quantitative analysis such as anomaly detection 7‐11 and root cause analysis 12‐15 …”

Section: Introductionmentioning

confidence: 99%

“…In contrast to full-text search, [4][5][6] which is another major approach used for keyword-based analysis, the template-based approach is suitable for aggregating and digesting the system behaviors, enabling time-series-based quantitative analysis such as anomaly detection [7][8][9][10][11] and root cause analysis. [12][13][14][15] Many log template generation methods have been proposed in past literature. 16,17 These methods are based on diverse approaches such as source code analysis, [18][19][20] clustering, [21][22][23][24][25][26][27] prefix-tree approaches, 8,28,29 and machine learning.…”

mentioning

confidence: 99%

See 2 more Smart Citations

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

et al. 2021

Self Cite

View full text Add to dashboard Cite

Summary One of the ways to analyze unstructured log messages from large‐scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no common knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for comparing and combining diverse log template generation methods. Amulog consists of three key functions: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template‐matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time‐series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation‐wide academic network and demonstrate that it classifies the log data in a reasonable amount of time even with over 100,000 log template candidates. The template‐matching method in amulog also reduces 75% processing time for template generation and keeps the accuracy when combined with an existing structure‐based template generation method. In order to show the effectiveness of amulog in comparing log template generation methods, we demonstrate that the appropriate template generation methods and accuracy metrics largely depend on the purpose of further analysis by comparing the accuracy of six existing log template generation methods with 10 different accuracy metrics on amulog.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…Whilst there have been attempts to reduce the time complexity of such approaches through domain knowledgebased heuristics, e.g. [10], such approaches often rely on human-driven knowledge of the underlying system, which is impractical for modern infrastructure.…”

Section: Introductionmentioning

confidence: 99%

Exploiting Functional Connectivity Inference for Efficient Root Cause Analysis

Winchester

Parisis

Berthouze

2022

NOMS 2022-2022 IEEE/IFIP Network Operations and Management Symposium

View full text Add to dashboard Cite

A crucial step in remedying faults within network infrastructure is to determine their root cause. However, the large-scale, complex and dynamic nature of modern architecture makes root cause analysis challenging. Statistical approaches for causal inference are promising, however, their deployment has been historically limited due to their high time complexity. In this paper we propose a general framework for leveraging the concept of functional connectivity to reduce the computational overhead of causal inference algorithms. We demonstrate on synthetic data that our approach can achieve substantial speedups when combined with state-of-the-art causal discovery algorithms, with only a small cost in terms of loss of causal information in some cases.

show abstract

“…One approach in which scalability issues are mitigated in these techniques is as a by-product of the incorporation of costly domain knowledge, often utilised primarily to improve the accuracy of the approaches [13]- [15]. Although, in some cases such prior knowledge is incorporated to address scalability directly [16]. However, the cost associated with expert domain knowledge, both in required expertise and man-hours, is generally understudied and underappreciated.…”

Section: Introductionmentioning

confidence: 99%

Accelerating Causal Inference Based RCA Using Prior Knowledge From Functional Connectivity Inference

Winchester

Parisis

Harper

et al. 2022

2022 18th International Conference on Network and Service Management (CNSM)

View full text Add to dashboard Cite

A crucial step in remedying faults within network infrastructures is to determine their root cause. However, the large-scale, complex and dynamic nature of modern networks makes causal inference-based root cause analysis challenging in terms of scalability and knowledge drift over time. In this paper, we propose a framework that utilises the neuroscientific concept of functional connectivity -a graph representation of statistical dependencies between events -as a scalable approach to acquire and maintain prior knowledge for causal inferencebased RCA approaches in dynamic networks. We demonstrate on both synthetic and real-world data that our proposed approach can provide significant speedups to existing causal inference approaches without significant loss of accuracy. We show that, in some cases, such prior knowledge can even improve the accuracy of causal inference. Finally, we discuss the impact of the choice of user-defined parameters on causal inference accuracy and conclude that the framework can safely be deployed in the real world.

show abstract

Causal analysis of network logs with layered protocols and topology knowledge

Cited by 9 publications

References 20 publications

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

amulog: A general log analysis framework for comparison and combination of diverse template generation methods*

Exploiting Functional Connectivity Inference for Efficient Root Cause Analysis

Accelerating Causal Inference Based RCA Using Prior Knowledge From Functional Connectivity Inference

Contact Info

Product

Resources

About