Are Concept Drift Detectors Reliable Alarming Systems? - A Comparative Study

Abstract

Due to the continuous change in operational data, AIOps solutions suffer from performance degradation over time. Although periodic retraining is the state-of-the-art technique to preserve the failure prediction AIOps models' performance over time, this technique requires a considerable amount of labeled data to retrain. In AIOps obtaining label data is expensive since it requires the availability of domain experts to intensively annotate it. In this paper, we present McUDI, a model-centric unsupervised degradation indicator that is capable of detecting the exact moment the AIOps model requires retraining as a result of changes in data. We further show how employing McUDI in the maintenance pipeline of AIOps solutions can reduce the number of samples that require annotations with 30k for job failure prediction and 260k for disk failure prediction while achieving similar performance with periodic retraining.

Maintaining and Monitoring AIOps Models Against Concept Drift

“…Operational data is constantly changing/evolving (concept drift occurs) due to uncontrollable factors such as user workloads or hardware/software upgrades. Concept drift on operational data 1 Replication Package: https://github.com/LorenaPoenaru/aiops failure prediction negatively impacts the failure prediction AIOps models performances, resulting in an increase in their error rates [Lyu et al, 2021a], [Lyu et al, 2021b], [Li et al, 2020].…”

“…As observed in other real-world ML applications, such as healthcare, manufacturing, finance, and agriculture [Vela et al, 2022], one of the biggest challenges in building AIOps models is the temporal quality degradation (AI aging) [Dang et al, 2019], [Lyu et al, 2021a], [Lyu et al, 2021b]. The model quality (performance/accuracy) degradation is a consequence of the evolving character of operational data, also known as concept drift.…”

“…In some situations, concept drift leads to severe model degradation [Bayram et al, 2022], which impacts the reliability and trustworthiness of AIOps models. Moreover, a considerable degradation in the quality of AIOps models generates service interruptions [Dang et al, 2019], which causes substantial monetary losses (between $300,000 and $400,000 according to [Lyu et al, 2021b]).…”

“…To overcome the challenge of model performance degradation, previous work proposes to periodically retrain (update) AIOps models [Lyu et al, 2021a], [Lyu et al, 2021b]. However, in a practical scenario, periodic retraining is sometimes unfeasible due to the hidden deployment costs [Haakman et al, 2021].…”

“…Previous work [Lyu et al, 2023] proposed a more systematic solution to periodic model retraining, namely retraining only when necessary. This solution involves updating an AIOps model when a model degradation indicator detects a significant drop in its performance.…”