Formal Verification With Confidence Intervals to Establish Quality of Service Properties of Software Systems

Abstract

Formal verification is used to establish the compliance of software and hardware systems with important classes of requirements. System compliance with functional requirements is frequently analysed using techniques such as model checking, and theorem proving. In addition, a technique called quantitative verification supports the analysis of the reliability, performance, and other quality-of-service (QoS) properties of systems that exhibit stochastic behaviour. In this paper, we extend the applicability of quantitative verification to the common scenario when the probabilities of transition between some or all states of the Markov models analysed by the technique are unknown, but observations of these transitions are available. To this end, we introduce a theoretical framework, and a tool chain that establish confidence intervals for the QoS properties of a software system modelled as a Markov chain with uncertain transition probabilities. We use two case studies from different application domains to assess the effectiveness of the new quantitative verification technique. Our experiments show that disregarding the above source of uncertainty may significantly affect the accuracy of the verification results, leading to wrong decisions, and low-quality software systems.

Asymptotic Perturbation Bounds for Probabilistic Model Checking with Empirically Determined Probability Parameters

“…We also mention that parameters in a PMC are described with probability distributions in some papers [29], [30]. The authors employed statistical inference [29] or simulation [30] to deal with the verification problem of the resulted model.…”

“…The authors employed statistical inference [29] or simulation [30] to deal with the verification problem of the resulted model. By contrast, the reasoning techniques adopted by us and the aforementioned literature are analytical.…”

An Iterative Decision-Making Scheme for Markov Decision Processes and Its Application to Self-adaptive Systems

“…Several high-level frameworks and approaches based on probabilistic model checking have been proposed for self-adaptive systems recently, but with emphasis on different aspects of the adaptation, such as QoS management and optimization [4], adaptation decisions [20], verification with information of confidence intervals [3], runtime verification efficiency and sensitivity analysis [18], and proactive verification and adaptation latency [23]. None of those works addressed the problem of making a practical tradeoff similar to the one supported by IDMS.…”

“…The second contribution of this paper is an application of IDMS to selfadaptive systems. Several high-level frameworks and approaches based on probabilistic model checking have been proposed to aid the design of self-adaptive systems, but with emphasis on different aspects of the adaptation [3,4,18,20,23]. However none of these works address the problem of making the aforementioned tradeoff in the adaptation.…”

Designing Robust Software Systems through Parametric Markov Chain Synthesis

“…Delaying this evaluation until integration or system testing can greatly increase engineering costs, as defects identified late in the development lifecycle require much more effort to fix [3]. A common method to avoid this delay uses model-based simulation [4] or formal verification [5] to predict the quality attributes of alternative designs. Models that meet the quality requirements of the system under development are then used as a basis for its implementation.…”