Adaptive Runtime Verification

Abstract

Abstract. We present Adaptive Runtime Verification (ARV), a new approach to runtime verification in which overhead control, runtime verification with state estimation, and predictive analysis are synergistically combined. Overhead control maintains the overhead of runtime verification at a specified target level, by enabling and disabling monitoring of events for each monitor instance as needed. In ARV, predictive analysis based on a probabilistic model of the monitored system is used to estimate how likely each monitor instance is to violate a given temporal property in the near future, and these criticality levels are fed to the overhead controllers, which allocate a larger fraction of the target overhead to monitor instances with higher criticality, thereby increasing the probability of violation detection. Since overhead control causes the monitor to miss events, we use Runtime Verification with State Estimation (RVSE) to estimate the probability that a property is satisfied by an incompletely monitored run. A key aspect of the ARV framework is a new algorithm for RVSE that performs the calculations in advance, dramatically reducing the runtime overhead of RVSE, at the cost of introducing some approximation error. We demonstrate the utility of ARV on a significant case study involving runtime monitoring of concurrency errors in the Linux kernel.

Runtime Verification with Particle Filtering

“…With larger numbers of gaps, the particles get more widely dispersed in the state space, and more particles are needed to cover all of the interesting states. To evaluate the performance and accuracy of RVPF, we implemented it along with our previous two algorithms in C and compared them through experiments based on the benchmarks used in [1]. Our results confirm RVPF's versatility.…”

“…The alphabet A is a subset of the observable actions of the HMM; actions not in A leave the DFA's state unchanged. 1 The goal is to compute P (φ | o 1:T ), that is, the probability that the system's behavior satisfies φ, given observation sequence o 1:T . This probability is computed from the probability distribution on composite states, where a composite state (x, s) is a pair containing an HMM state x and a DFA state s. Specifically,…”

“…In our measurements, this was often more than a factor of 10 larger than the overhead of monitoring the events themselves! To reduce the runtime overhead, we developed a version of the algorithm, which we call approximate precomputed RVSE (AP-RVSE), that precomputes the matrix calculations and stores the results in a table [1]. Essentially, AP-RVSE precomputes a potentially infinite graph unfolding, where nodes are labeled with state probability distributions, and edges are labeled with transitions.…”

“…In contrast to our previous work [10,1], we model the HMM, the DFA, and their composition in a more elegant and succinct way as a dynamic Bayesian network (DBN). This allows us to properly formalize a new kind of event, called peek events, which are inexpensive observations of part of the program state.…”

Distributed Finite-State Runtime Monitoring with Aggregated Events

“…Bartocci et al [2] extend the concept of probabilistically monitoring gaps in events, and introduce the notion of criticality levels, which vary based on the probability that a system reaches an error state. Criticality levels can then be used to determine the degree of instrumentation performed, with the system increasing sampling to determine the precise system state.…”