A Framework of Virtual War Room and Matrix Sketch-Based Streaming Anomaly Detection for Microservice Systems

Abstract

Recently, microservice has been a popular architecture to construct cloud-native systems. This novel architecture brings agility and accelerates the software development process significantly. However, it is not easy to manage and operate microservice systems due to their scale and complexity. Many approaches are proposed to automatically operate microservice systems such as anomaly detection. Nevertheless, those methods cannot be sufficiently validated and compared due to a lack of real microservice systems, which leads to the slow process of intelligent operation. These challenges inspire us to build a system named ''VWR'', a framework of Virtual War Room for operating microservice applications which allows users to simulate their microservice architectures with low overhead and inject multiple types of faults into the microservice system with chaos engineering. VWR can mimic user requests and record the end-to-end tracing data (i.e., service call chains) for each request in a way consistent with OpenTracing. With easily designed tests and the produced streaming tracing data, the users can validate the performance of their intelligent operation algorithms and improve the algorithms as needed. Besides, based on the streaming tracing data generated by VWR, we introduce a novel unsupervised anomaly detection algorithm based on Matrix Sketch and set it as a default intelligent operation algorithm in VWR. This algorithm can detect anomalies by analyzing high-dimensional performance data collected from a microservice system in a streaming manner. The experimental result in VWR shows that the matrix sketch based method can precisely detect anomalies in microservice systems and outperform some widely used anomaly detection methods such as isolation forest in some scenario. We believe more approaches on the intelligent operation of microservice systems can be constructed based on VWR. INDEX TERMS Microservice, virtual war room, matrix sketch, anomaly detection, chaos engineering.

EduChain: CIA-Compliant Blockchain for Intelligent Cyber Defense of Microservices in Education Industry 4.0

“…Mena et al [8] described the micro-services concept for the application of software architecture on IoT devices to acquire data and to maintain the module confguration. Chen et al [9] developed virtual war room model to handle the behavior of micro-service applications including data tracing, fault analysis. In software industry, Akbulut and Perros [10] executed the performance analysis of hardware behavior, time of query response and rate of packet loss for micro-services sketch patterns to reduce infrastructural-risks.…”

On Microservice Analysis and Architecture Evolution: A Systematic Mapping Study

“…The authors put the mechanism to the test on the SAVI testbed and observed that it could successfully detect anomalous application behavior [67]. Chen et al [38] leveraged resource metrics to find anomalies in the streaming of microservice systems.…”

Anomaly Detection and Failure Root Cause Analysis in (Micro)Service-Based Cloud Applications: A Survey

“…Trace comparison is another possible technique to enact online anomaly detection on multi-service applications instrumented to feature distributed tracing. This is the technique followed by Meng et al [54], Wang et al [88], and Chen et al [17], which all rely on collecting the traces that can possibly occur in a multi-service application and by then checking whether newly collected traces are similar to the collected ones.…”

“…Chen et al [17] instead uses matrix sketching to compare traces and detect anomalies. More precisely, Chen et al [17] adapt an existing matrix sketching algorithm [29] to detect response time anomalies in services in monitored traces.…”

“…Chen et al [17] instead uses matrix sketching to compare traces and detect anomalies. More precisely, Chen et al [17] adapt an existing matrix sketching algorithm [29] to detect response time anomalies in services in monitored traces. This is done by maintaining up-to-date a "sketch" of the high-dimensional data corresponding to the historically monitored response times of the services forming an application.…”