Towards a Consistent Interpretation of AIOps Models

Abstract

Artificial Intelligence for IT Operations (AIOps) has been adopted in organizations in various tasks, including interpreting models to identify indicators of service failures. To avoid misleading practitioners, AIOps model interpretations should be consistent (i.e., different AIOps models on the same task agree with one another on feature importance). However, many AIOps studies violate established practices in the machine learning community when deriving interpretations, such as interpreting models with suboptimal performance, though the impact of such violations on the interpretation consistency has not been studied. In this article, we investigate the consistency of AIOps model interpretation along three dimensions: internal consistency, external consistency, and time consistency. We conduct a case study on two AIOps tasks: predicting Google cluster job failures and Backblaze hard drive failures. We find that the randomness from learners, hyperparameter tuning, and data sampling should be controlled to generate consistent interpretations. AIOps models with AUCs greater than 0.75 yield more consistent interpretation compared to low-performing models. Finally, AIOps models that are constructed with the Sliding Window or Full History approaches have the most consistent interpretation with the trends presented in the entire datasets. Our study provides valuable guidelines for practitioners to derive consistent AIOps model interpretation.

What makes Ethereum blockchain transactions be processed fast or slow? An empirical study

“…We choose a random forest model as it provides an optimal balance between predictive power and explainability. Such models are equipped to fit data better than simple regression models, potentially leading to more robust and reliable insights (Lyu et al 2021 ). Although they are not inherently explainable, several feature attribution methods can and have been used to derive explanations using these models (Breiman 2001 ; Lundberg et al 2018 ; Tantithamthavorn and Jiarpakdee 2021 ).…”

“…We thus define the training set as the first 28 days of data, the validation set as the 29 th day, and the test set as the final day. We split the data along its temporal nature to avoid data leakage (Lyu et al 2021 ). We then use the training set to generate 100 bootstrap samples (i.e., samples with replacement and of the same length as the training set).…”

What makes Ethereum blockchain transactions be processed fast or slow? An empirical study

“…We choose a random forest model as it provides an optimal balance between predictive power and explainability. Such models are equipped to fit data better than simple regression models, potentially leading to more robust and reliable insights [25]. Although they are not inherently explainable, several feature attribution methods can and have been used to derive explanations using these models [9,24,46].…”

“…We thus define the training set as the first 28 days of data, the validation set as the 29 th day, and the test set as the final day. We split the data along its temporal nature to avoid data leakage [25]. We then use the training set to generate 100 bootstrap samples (i.e., samples with replacement and of the same length as the training set).…”

Maintaining and Monitoring AIOps Models Against Concept Drift

“…One solution that ML practitioners adopt to handle the evolving character of data is retraining/updating ML models over time [24]. Periodical model retraining has also been studied for failure detection AIOps solutions [22], [21] and has proved that continuous model updates achieve better performance over time compared to non-updated models. However, the effects of continuously updated models have only been studied for failure prediction models [21], [22].…”