Fast Dimensional Analysis for Root Cause Investigation in a Large-Scale Service Environment

Abstract

Root cause analysis in a large-scale production environment is challenging due to the complexity of the services running across global data centers. Due to the distributed nature of a large-scale system, the various hardware, software, and tooling logs are often maintained separately, making it difficult to review the logs jointly for understanding production issues. Another challenge in reviewing the logs for identifying issues is the scale - there could easily be millions of entities, each described by hundreds of features. In this paper we present a fast dimensional analysis framework that automates the root cause analysis on structured logs with improved scalability. We first explore item-sets, i.e. combinations of feature values, that could identify groups of samples with sufficient support for the target failures using the Apriori algorithm and a subsequent improvement, FP-Growth. These algorithms were designed for frequent item-set mining and association rule learning over transactional databases. After applying them on structured logs, we select the item-sets that are most unique to the target failures based on lift. We propose pre-processing steps with the use of a large-scale real-time database and post-processing techniques and parallelism to further speed up the analysis and improve interpretability, and demonstrate that such optimization is necessary for handling large-scale production datasets. We have successfully rolled out this approach for root cause investigation purposes within Facebook's infrastructure. We also present the setup and results from multiple production use cases in this paper.

Graph-Based Root Cause Localization in Microservice Systems with Protection Mechanisms

“…N, n the number of traces, the number of services m, c the type number of metrics, the collected number of each type metric Metric Anomaly Score: We use the mean µ ik and standard deviation σ ik of the service metrics to calculate service anomaly severity [20][21][22][23][24] . The µ ik is the expected normal value and the σ ik indicates that the metric deviates from the mean.…”

Incident Management for Explainable and Automated Root Cause Analysis in Cloud Data Centers    

“…One of the most important capabilities of the AI-powered data-analytics for IT operations (AIOps) is fully automated RCA ([Gartner Research, 2019]). Many AIOps platform vendors like IBM (see [IBM, 2021]), Facebook (see [Lin et al, 2020]), VMware (see , Marvasti et al, 2014a, Marvasti et al, 2014b, Marvasti et al, 2016, Harutyunyan et al, 2020b, Harutyunyan et al, 2020c), HPE (see [HPE, 2019]), BigPanda (see [BigPanda, 2020]), DataDog (see [Othmane A.-A., 2021]), Moogsoft (se [Sahil K., 2016]) and others ([Moogsoft, 2016]) have almost complete vision and solution for the domain-centric RCA described in Figure 1.…”

“…It has been in the focus of researchers for decades with diverse ideas including anomaly detection, event correlations, causal inference, correlation analysis, predictive models and many others (see [ABS Consulting et al, 2014, Zawawy et al, 2010, Chuah et al, 2010, Cai et al, 2019, Marvasti et al, 2014b with references therein). Ideally, RCA should analyze all acquired monitoring datasets including logs (see , Harutyunyan et al, 2018a, Mi et al, 2012, Kostroš et al, 2014, Tak et al, 2016, Chuah et al, 2010, Bird et al, 2015, Zawawy et al, 2010, Michalski, 1983), traces (see [Suriadi et al, 2013, Lin et al, 2020) and time series data (see [Jeyakumar et al, 2019, Pearl, 2009, Spirtes et al, 2000) with possible correlations among them. Distributed tracing is the classical approach to application monitoring and diagnostics (see [Opentracing, 2019]).…”

“…Rule learning systems have wide applications including analysis of log-data and trace-data. Papers ( [Suriadi et al, 2013, Lin et al, 2020) consider analysis of log-data. Paper [Suriadi et al, 2013] compares RIPPER to C4.5 ( [Quinlan, 2014]) for RCA on log data.…”

“…It uses implementation of both algorithms in WEKA ( [Witten et al, 2005]) known as JRip and J48 for RIPPER and C4.5, respectively. Paper ( [Lin et al, 2020]) considers RCA in a large-scale production environment based on structured logs. It explores application of the Apriori algorithm ( [Agrawal et al, 1993]) with subsequent improvement with FP-Growth ( [Han et al, 2004]).…”