The Challenging Landscape of Cloud Monitoring

Pourmajidi, William; Zhang, Lei; Miranskyy, Andriy; Steinbacher, John; Godwin, David; Erwin, Tony

doi:10.1201/9781003001188-12

Cited by 7 publications

(28 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, the advanced monitoring system has been deployed in data centers to observe the health of Cloud components. However, most current health monitoring systems still rely on statistics and heuristics based on resource utilization thresholds only from the observed health metrics [3,4]. The variety and the random nature of today's application requirements make such anomaly detection techniques less effective in a Cloud environment.…”

Section: Big Data Challengementioning

confidence: 99%

Anomaly Detection in Cloud Components

Islam

Miranskyy

2020

2020 IEEE 13th International Conference on Cloud Computing (CLOUD)

Self Cite

View full text Add to dashboard Cite

Cloud platforms, under the hood, consist of a complex inter-connected stack of hardware and software components. Each of these components can fail which may lead to an outage. Our goal is to improve the quality of Cloud services through early detection of such failures by analyzing resource utilization metrics. We tested Gated-Recurrent-Unit-based autoencoder with a likelihood function to detect anomalies in various multi-dimensional time series and achieved high performance.

show abstract

Section: Big Data Challengementioning

confidence: 99%

Anomaly Detection in Cloud Components

Islam

Miranskyy

2020

2020 IEEE 13th International Conference on Cloud Computing (CLOUD)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Thus, proper monitoring is essential to ensure service quality and keep up with the SLA. Though the current monitoring system encompasses the components to collect data, the statistics-based anomaly detectors generate too many alerts [3,4]. The flooding of alerts (many of which are false) makes it impractical to rely and work on those.…”

Section: Our Challengesmentioning

confidence: 99%

“…The logs' sources vary (e.g., hardware logs, operating system logs, and application logs), which reflect the heterogeneity of components. Increasing complexity, velocity, variety, and volume of logs generated by Cloud platforms components makes the analysis of such logs a Big Data challenge [2,3,4].…”

Section: Introductionmentioning

confidence: 99%

“…In recent years, advanced and automated monitoring systems have been implemented in data centres to monitor Cloud components' health. However, most existing monitoring systems still depend on statistics and heuristics from the observed health indicators centred on resource usage thresholds [3,4]. In a Cloud environment, such anomaly detection techniques are less effective due to the Cloud platforms' scale and the rapidly changing nature of the workloads executed on these platforms [3].…”

Section: Introductionmentioning

confidence: 99%

“…However, most existing monitoring systems still depend on statistics and heuristics from the observed health indicators centred on resource usage thresholds [3,4]. In a Cloud environment, such anomaly detection techniques are less effective due to the Cloud platforms' scale and the rapidly changing nature of the workloads executed on these platforms [3]. The reduction of efficiency is typically manifested by numerous false alarms that overwhelm the operational team [4].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Anomaly Detection in a Large-scale Cloud Platform

Islam,

Pourmajidi,

Zhang

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

Cloud computing is ubiquitous: more and more companies are moving the workloads into the Cloud. However, this rise in popularity challenges Cloud service providers, as they need to monitor the quality of their ever-growing offerings effectively. To address the challenge, we designed and implemented an automated monitoring system for the IBM Cloud Platform. This monitoring system utilizes deep learning neural networks to detect anomalies in near-real-time in multiple Platform components simultaneously.After running the system for a year, we observed that the proposed solution frees the DevOps team's time and human resources from manually monitoring thousands of Cloud components. Moreover, it increases customer satisfaction by reducing the risk of Cloud outages.In this paper, we share our solutions' architecture, implementation notes, and best practices that emerged while evolving the monitoring system. They can be leveraged by other researchers and practitioners to build anomaly detectors for complex systems.

show abstract

Anomaly Detection in Cloud Components

Islam¹

2023

Preprint

View full text Add to dashboard Cite

<p>Today’s smart connected world is all about devices, data and connectivity. Cloud computing has been introduced to support the rapidly growing technological demand by providing ﬂexibility, efﬁciency, mobility, robustness and disaster recovery mechanism to individuals and businesses. Cloud platforms, under the hood, consist of a complex inter-connected stack of hardware and software components. Each of these components can fail, which may lead to an outage. Therefore, anomaly detection accuracy plays a vital role in effective system operation management, which is the key to ensure stability and performance. Our goal is to improve the quality of Cloud services through early detection of such failures by analyzing resource utilization metrics. In this thesis, we constructed a Gated-Recurrent-Unit-based autoencoder to generate reconstruction errors and computed the likelihood of anomalies based on the distribution of reconstruction errors. The combined approach enables efﬁcient detection of abnormalities in multi-dimensional telemetry time-series gathered from a private Cloud.</p>

show abstract

The Challenging Landscape of Cloud Monitoring

Cited by 7 publications

References 0 publications

Anomaly Detection in Cloud Components

Anomaly Detection in Cloud Components

Anomaly Detection in a Large-scale Cloud Platform

Anomaly Detection in Cloud Components

Contact Info

Product

Resources

About