Anomaly detection and diagnosis for cloud services: Practical experiments and lessons learned

Sauvanaud, Carla; Kaniche, Mohamed; Kanoun, Karama; Lazri, Kahina; Silvestre, Guthemberg

doi:10.1016/j.jss.2018.01.039

Cited by 55 publications

(26 citation statements)

References 45 publications

(58 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This is mainly due to the versatility of this kind of methods and the increasing availability of data from which they can learn from, in a continuous manner [3]. Most of the approaches to anomaly detection address the associated challenges by feeding ML models with counters like CPU utilization, memory contention and network-related metrics [12,31,34]. Others include also system-level and/or application-level event logs in the analysis to increase the amount of features and facilitate the extraction of relevant patterns [8,35].…”

Section: Related Workmentioning

confidence: 99%

Using Self-Organizing Maps for the Behavioral Analysis of Virtualized Network Functions

Lanciano

Ritacco

Brau

et al. 2021

Communications in Computer and Information Science

View full text Add to dashboard Cite

Detecting anomalous behaviors in a network function virtualization infrastructure is of the utmost importance for network operators. In this paper, we propose a technique, based on Self-Organizing Maps, to address such problem by leveraging on the massive amount of historical system data that is typically available in these infrastructures. Indeed, our method consists of a joint analysis of system-level metrics, provided by the virtualized infrastructure monitoring system and referring to resource consumption patterns of the physical hosts and the virtual machines (or containers) that run on top of them, and application-level metrics, provided by the individual virtualized network functions monitoring subsystems and related to the performance levels of the individual applications. The implementation of our approach has been validated on real data coming from a subset of the Vodafone infrastructure for network function virtualization, where it is currently employed to support the decisions of data center operators. Experimental results show that our technique is capable of identifying specific points in space (i.e., components of the infrastructure) and time of the recent evolution of the monitored infrastructure that are worth to be investigated by human operators in order to keep the system running under expected conditions.

show abstract

Section: Related Workmentioning

confidence: 99%

Using Self-Organizing Maps for the Behavioral Analysis of Virtualized Network Functions

Lanciano

Ritacco

Brau

et al. 2021

Communications in Computer and Information Science

View full text Add to dashboard Cite

show abstract

“…In [2], the authors presented their anomaly detection approach for cloud services. They deployed a cloud application which consisted of several services on several VMs and each VM ran a specific service.…”

Section: Related Workmentioning

confidence: 99%

“…Data processing tasks Data processing helps us to detect and diagnose anomalies. Carla et al defined an anomaly as the part of the system state that may lead to an SLAV [2]. We use the same definition of anomaly as stated in Carla's work.…”

Section: Data Processingmentioning

confidence: 99%

See 1 more Smart Citation

Anomaly Detection and Diagnosis for Container-based Microservices with Performance Monitoring

Du¹,

Xie²,

He³

2018

EasyChair Preprints

View full text Add to dashboard Cite

With emerging container technologies, such as Docker, microservices-based applications can be developed and deployed in cloud environment much agiler. The dependability of these microservices becomes a major concern of application providers. Anomalous behaviors which may lead to unexpected failures can be detected with anomaly detection techniques. In this paper, an anomaly detection system(ADS) is designed to detect and diagnose the anomalies in microservices by monitoring and analyzing real-time performance data of them. The proposed ADS consists of a monitoring module that collects the performance data of containers, a data processing module based on machine learning models and a fault injection module integrated for training these models. The fault injection module is also used to assess the anomaly detection and diagnosis performance of our ADS. Clearwater, an open source virtual IP Multimedia Subsystem, is used for the validation of our ADS and experimental results show that the proposed ADS works well.

show abstract

“…Extensive studies have been done to resolve performance problems in distributed systems such as anomaly detection [6]- [8], root cause analysis [2], [9]- [11]. In these studies, the researchers need to deploy their own microservice benchmarks and inject some faults into these systems in order to get abnormal cases.…”

Section: Introductionmentioning

confidence: 99%

A Framework of Virtual War Room and Matrix Sketch-Based Streaming Anomaly Detection for Microservice Systems

Chen

2020

IEEE Access

View full text Add to dashboard Cite

Recently, microservice has been a popular architecture to construct cloud-native systems. This novel architecture brings agility and accelerates the software development process significantly. However, it is not easy to manage and operate microservice systems due to their scale and complexity. Many approaches are proposed to automatically operate microservice systems such as anomaly detection. Nevertheless, those methods cannot be sufficiently validated and compared due to a lack of real microservice systems, which leads to the slow process of intelligent operation. These challenges inspire us to build a system named ''VWR'', a framework of Virtual War Room for operating microservice applications which allows users to simulate their microservice architectures with low overhead and inject multiple types of faults into the microservice system with chaos engineering. VWR can mimic user requests and record the end-to-end tracing data (i.e., service call chains) for each request in a way consistent with OpenTracing. With easily designed tests and the produced streaming tracing data, the users can validate the performance of their intelligent operation algorithms and improve the algorithms as needed. Besides, based on the streaming tracing data generated by VWR, we introduce a novel unsupervised anomaly detection algorithm based on Matrix Sketch and set it as a default intelligent operation algorithm in VWR. This algorithm can detect anomalies by analyzing high-dimensional performance data collected from a microservice system in a streaming manner. The experimental result in VWR shows that the matrix sketch based method can precisely detect anomalies in microservice systems and outperform some widely used anomaly detection methods such as isolation forest in some scenario. We believe more approaches on the intelligent operation of microservice systems can be constructed based on VWR. INDEX TERMS Microservice, virtual war room, matrix sketch, anomaly detection, chaos engineering.

show abstract

Anomaly detection and diagnosis for cloud services: Practical experiments and lessons learned

Cited by 55 publications

References 45 publications

Using Self-Organizing Maps for the Behavioral Analysis of Virtualized Network Functions

Using Self-Organizing Maps for the Behavioral Analysis of Virtualized Network Functions

Anomaly Detection and Diagnosis for Container-based Microservices with Performance Monitoring

A Framework of Virtual War Room and Matrix Sketch-Based Streaming Anomaly Detection for Microservice Systems

Contact Info

Product

Resources

About