A comparison framework for runtime monitoring approaches

Abstract

The full behavior of complex software systems often only emerges during operation. They thus need to be monitored at run time to check that they adhere to their requirements. Diverse runtime monitoring approaches have been developed in various domains and for different purposes. Their sheer number and heterogeneity, however, make it hard to find the right approach for a specific application or purpose. The aim of our research therefore was to develop a comparison framework for runtime monitoring approaches. Our framework is based on an analysis of the literature and existing taxonomies for monitoring languages and patterns. We use examples from existing monitoring approaches to explain the framework. We demonstrate its usefulness by applying it to 32 existing approaches and by comparing 3 selected approaches in the light of different monitoring scenarios. We also discuss perspectives for researchers

How bad can a bug get? an empirical analysis of software failures in the OpenStack cloud computing platform

“…Moreover, residual faults should be detected and handled by means of run-time verification strategies, which perform redundant, endto-end checks after a service API call, to assert whether the virtual resources are in the expected state. For example, these checks can be specified using temporal logic and synthesized in a run-time monitor [8,13,64,79], e.g., a logical predicate for a traditional OS can assert that a thread suspended on a semaphore leads to the activation of another thread [2]. In the context of cloud management, the predicates should test at run-time the availability of virtual resources (e.g., volumes and connectivity), similarly to our assertion checks ( Table 1).…”

A comparison framework for runtime monitoring approaches (journal-first abstract)

“…Many of today's large-scale software systems are complex and heterogeneous, and need to be monitored since their full behavior often only emerges at runtime (e.g., when interacting with other systems or the environment). Diverse monitoring approaches for various kinds of systems and purposes have been proposed [1], e.g., requirements monitoring [2], monitoring of architectural properties [3], and runtime verification [4]. The desired runtime behavior is often formally expressed using temporal logic or through the use of domain-specific (constraint) languages.…”

“…for runtime monitoring approaches [1] based on the results of a systematic literature review [2], building on existing taxonomies for monitoring languages and patterns [5], [6], and taking inspiration from comparison frameworks from other domains such as software architecture or software product lines [7], [8].…”

“…Specifically, our paper [1] provides the following contributions: (i) a framework that supports analyzing and comparing runtime monitoring approaches using different dimensions and elements; (ii) an application of the framework to analyze and compare 32 existing monitoring approaches; and (iii) a discussion of perspectives and potential future applications of our framework, e.g., to support the selection of an approach for a particular monitoring problem or application context.…”

“…Approaches also differ regarding their expressiveness [5], e.g., the degree of support to check the occurrence and/or order of runtime events (temporal behavior), the interactions occurring between different (sub-)systems (structural behavior), and/or the properties held by certain runtime data (data checks).This diversity makes it difficult to analyze and compare existing approaches. We thus developed a comparison frameworkThe full version of this work was published as journal article [1].for runtime monitoring approaches [1] based on the results of a systematic literature review [2], building on existing taxonomies for monitoring languages and patterns [5], [6], and taking inspiration from comparison frameworks from other domains such as software architecture or software product lines [7], [8].Specifically, our paper [1] provides the following contributions: (i) a framework that supports analyzing and comparing runtime monitoring approaches using different dimensions and elements; (ii) an application of the framework to analyze and compare 32 existing monitoring approaches; and (iii) a discussion of perspectives and potential future applications of our framework, e.g., to support the selection of an approach for a particular monitoring problem or application context.…”