Predicting Node Failures in an Ultra-Large-Scale Cloud Computing Platform

Li, Yangguang; Jiang, Zhen Ming; Li, Heng; Hassan, Ahmed E.; He, Cheng; Huang, Rongjin; Zeng, Zhengda; Wang, Mian; Chen, Pinan

doi:10.1145/3385187

Cited by 52 publications

(38 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In Table ??, a large number of studies adopted predictive models to predict bugs or faults in software systems. Failure prediction is critical to predictive maintenance since it has the ability to prevent maintenance costs and failure / bug occurrences [21,34,137,260,331]. Yilmaz and Porter [315] classify measured executions into successful and failed executions in order to apply the resulting models to systems with an unknown failure status.…”

Section: Sotware Testingmentioning

confidence: 99%

Predictive Models in Software Engineering: Challenges and Opportunities

Yang

Xia

et al. 2022

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Predictive models are one of the most important techniques that are widely applied in many areas of software engineering. There have been a large number of primary studies that apply predictive models and that present well-performed studies in various research domains, including software requirements, software design and development, testing and debugging and software maintenance. This paper is a first attempt to systematically organize knowledge in this area by surveying a body of 421 papers on predictive models published between 2009 and 2020. We describe the key models and approaches used, classify the different models, summarize the range of key application areas, and analyze research results. Based on our findings, we also propose a set of current challenges that still need to be addressed in future work and provide a proposed research road map for these opportunities.

show abstract

Section: Sotware Testingmentioning

confidence: 99%

Predictive Models in Software Engineering: Challenges and Opportunities

Yang

Xia

et al. 2022

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

show abstract

“…Besides, some black-box ML and DL models, like LSTM and MING-based models, is hard to explain the results. e lack of decision interpretability is not convenient for maintenance work of DevOps engineers [20]. erefore this paper calculate the monitoring time series anomaly score in real-time, which is automatic feature engineering to build a random forest-based prediction model in the second layer.…”

Section: Literature Reviewmentioning

confidence: 99%

A Two‐Layer Architecture for Failure Prediction Based on High‐Dimension Monitoring Sequences

Wang

Liu

Feng³

et al. 2021

Complexity

View full text Add to dashboard Cite

In recent years, the distributed architecture has been widely adopted by security companies with the rapid expansion of their business. A distributed system is comprised of many computing nodes of different components which are connected by high-speed communication networks. With the increasing functionality and complexity of the systems, failures of nodes are inevitable which may result in considerable loss. In order to identify anomalies of the possible failures and enable DevOps engineers to operate in advance, this paper proposes a two-layer prediction architecture based on the monitoring sequences of nodes status. Generally speaking, in the first layer, we make use of EXPoSE anomaly detection technique to derive anomaly scores in constant time which are then used as input data for ensemble learning in the second layer. Experiments are conducted on the data provided by one of the largest security companies, and the results demonstrate the predictability of the proposed approach.

show abstract

“…Cloud computing platforms generate a tremendous amount of data that is impossible to be analyzed manually. Recently, it has become increasingly common for organizations to use AIOps (Artificial Intelligence for IT Operations) to leverage such generated data to ensure the quality of service and high availability of cloud computing platforms [6,8,21,42,62,88]. AIOps leverages machine learning learners to construct machine learning models (hereafter AIOps models) with operations data collected from the cloud computing platforms (e.g., logs and alert signals) to enable quality assurance tasks such as predicting hard drive failures [42], job termination [22], service outages [88], and performance issues [44].…”

Section: Introductionmentioning

confidence: 99%

“…Recently, it has become increasingly common for organizations to use AIOps (Artificial Intelligence for IT Operations) to leverage such generated data to ensure the quality of service and high availability of cloud computing platforms [6,8,21,42,62,88]. AIOps leverages machine learning learners to construct machine learning models (hereafter AIOps models) with operations data collected from the cloud computing platforms (e.g., logs and alert signals) to enable quality assurance tasks such as predicting hard drive failures [42], job termination [22], service outages [88], and performance issues [44]. Note that we use the term "learner" to refer to a machine learning algorithm (e.g., Random Forest) and the term "model" to refer to a trained machine learning model (e.g., a Random Forest model trained on disk failure data).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Towards a Consistent Interpretation of AIOps Models

LyuYingzhe

Rajbahadur

Lin

et al. 2021

ACM Trans. Softw. Eng. Methodol.

Self Cite

View full text Add to dashboard Cite

Artificial Intelligence for IT Operations (AIOps) has been adopted in organizations in various tasks, including interpreting models to identify indicators of service failures. To avoid misleading practitioners, AIOps model interpretations should be consistent (i.e., different AIOps models on the same task agree with one another on feature importance). However, many AIOps studies violate established practices in the machine learning community when deriving interpretations, such as interpreting models with suboptimal performance, though the impact of such violations on the interpretation consistency has not been studied. In this article, we investigate the consistency of AIOps model interpretation along three dimensions: internal consistency, external consistency, and time consistency. We conduct a case study on two AIOps tasks: predicting Google cluster job failures and Backblaze hard drive failures. We find that the randomness from learners, hyperparameter tuning, and data sampling should be controlled to generate consistent interpretations. AIOps models with AUCs greater than 0.75 yield more consistent interpretation compared to low-performing models. Finally, AIOps models that are constructed with the Sliding Window or Full History approaches have the most consistent interpretation with the trends presented in the entire datasets. Our study provides valuable guidelines for practitioners to derive consistent AIOps model interpretation.

show abstract

Predicting Node Failures in an Ultra-Large-Scale Cloud Computing Platform

Cited by 52 publications

References 34 publications

Predictive Models in Software Engineering: Challenges and Opportunities

Predictive Models in Software Engineering: Challenges and Opportunities

A Two‐Layer Architecture for Failure Prediction Based on High‐Dimension Monitoring Sequences

Towards a Consistent Interpretation of AIOps Models

Contact Info

Product

Resources

About