Toward a Formalism for Conservative Claims about the Dependability of Software-Based Systems

Bishop, Peter; Bloomfield, Robin; Littlewood, Bev; Povyakalo, Andrey; Wright, David

doi:10.1109/tse.2010.67

Cited by 40 publications

(79 citation statements)

References 14 publications

(11 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In some earlier work [13] we considered a similar problem. There our interest was in a system's probability of failure on demand (rather than, as here, its probability of perfection, or non-perfection).…”

Section: Discussionmentioning

confidence: 99%

“…Informal introduction to our approach: the problem of the prior distribution Our approach to the problem is similar to that we introduced in [13]. However, in that work our interest centered upon the problem of obtaining conservative claims for system pfd; here our aim is to obtain conservative claims for probability of perfection (from which, of course, we can calculate a probability of non-perfection as required, for example, for the Littlewood/Rushby model).…”

Section: Claims For Probability Of Perfection Based On Failure-frmentioning

confidence: 99%

See 1 more Smart Citation

Conservative claims about the probability of perfection of software-based systems

Zhao

Littlewood

Povyakalo

et al. 2015

2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

Abstract-In recent years we have become interested in the problem of assessing the probability of perfection of softwarebased systems which are sufficiently simple that they are "possibly perfect". By "perfection" we mean that the software of interest will never fail in a specific operating environment. We can never be certain that it is perfect, so our interest lies in claims for its probability of perfection. Our approach is Bayesian: our aim is to model the changes to this probability of perfection as we see evidence of failure-free working. Much of the paper considers the difficult problem of expressing prior beliefs about the probability of failure on demand (pfd), and representing these mathematically. This requires the assessor to state his prior belief in perfection as a probability, and also to state what he believes are likely values of the pfd in the event that the system is not perfect. We take the view that it will be impractical for an assessor to express these beliefs as a complete distribution for pfd. Our approach to the problem has three threads. Firstly we assume that, although he cannot provide a full probabilistic description of his uncertainty in a single distribution, the assessor can express some precise but partial beliefs about the unknowns. Secondly, we assume that in the inevitable presence of such incompleteness, the Bayesian analysis needs to provide results that are guaranteed to be conservative (because the analyses we have in mind relate to critical systems). Finally, we seek to prune the set of prior distributions that the assessor finds acceptable in order that the conservatism of the results is no greater than it has to be, i.e. we propose, and eliminate, sets of priors that would appear generally unreasonable. We give some illustrative numerical examples of this approach, and note that the numerical values obtained for the posterior probability of perfection in this way seem potentially useful (although we make no claims for the practical realism of the numbers we use). We also note that the general approach here to the problem of expressing and using limited prior belief in a Bayesian analysis may have wider applicability than to the problem we have addressed.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Claims For Probability Of Perfection Based On Failure-frmentioning

confidence: 99%

Conservative claims about the probability of perfection of software-based systems

Zhao

Littlewood

Povyakalo

et al. 2015

2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The basic idea here is similar to that we proposed in (Bishop, Bloomfield et al 2011). In that paper it was shown how to obtain a conservative pfd for a single channel based on an assessor's limited prior belief, together with some failure-free operational testing.…”

mentioning

confidence: 96%

“…We begin, for simplicity, by assuming that the channel pfds are known with certainty: pfd A , pfd B . This assumption can later be relaxed by using the results of (Bishop, Bloomfield et al 2011) upon each channel. So the only uncertainty concerns the degree of association between the failures of channel A and channel B.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Conservative Bounds for the pfd of a 1-out-of-2 Software-Based System Based on an Assessor's Subjective Probability of "Not Worse Than Independence"

Littlewood

Povyakalo

2013

IIEEE Trans. Software Eng.

Self Cite

View full text Add to dashboard Cite

Citation: Littlewood, B. & Povyakalo, A. A. (2013). Conservative bounds for the pfd of a 1-out-of-2 software-based system based on an assessor's subjective probability of "not worse than independence". IEEE Transactions on Software Engineering, 39(12), pp. 1641 -1653 . doi: 10.1109 /TSE.2013 This is the accepted version of the paper.This version of the publication may differ from the final published version. Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to. City Research Online: http://openaccess.city.ac.uk/ publications@city.ac.uk Permanent repository link: City Research OnlineConservative bounds for the pfd of a 1-out-of-2 softwarebased system based on an assessor's subjective probability of "not worse than independence"Bev Littlewood, Andrey PovyakaloCentre for Software Reliability, City University, London EC1V 0HB AbstractWe consider the problem of assessing the reliability of a 1-out-of-2 software-based system, in which failures of the two channels cannot be assumed to be independent with certainty. An informal approach to this problem assesses the channel pfds (probabilities of failure on demand) conservatively and then multiplies these together in the hope that the conservatism will be sufficient to overcome any possible dependence between the channel failures. Our intention here is to place this kind of reasoning on a formal footing. We introduce a notion of "not worse than independence" and assume that an assessor has a prior belief about this, expressed as a probability. We obtain a conservative prior system pfd, and show how a conservative posterior system pfd can be obtained following the observation of a number of demands without system failure. We present some illustrative numerical examples, discuss some of the difficulties involved in this way of reasoning, and suggest some avenues of future research.KEY WORDS: System reliability; Software fault tolerance; 1-out-of-2 system; Dependent failures; Subjective probability. BackgroundWe consider the problem of assessing the reliability of a 1-out-of-2 system in which the two software-based channels are "diverse" as a result of having been developed independently of one another (indeed, their designs may have been forced to be diverse by imposing diverse development procedures upon their designers). Such design-diverse fault tolerant systems have been used successfully in some safety critical applications: see (Littlewood, Popov et al. 2002;Wood, Belles et al. 2010).Whilst there is some general evidence that this kind of design-diverse fault tolerance is a good way of achieving high reliability -for example from experiments -there are serious difficulties in assessing the reliability of a particular system. An important problem arises from the fact that we can never be certain that the channels in such a system will fail...

show abstract

Demonstrating software reliability using possibly correlated tests: Insights from a conservative Bayesian approach

Salako,

Zhao

2023

Quality & Reliability Eng

View full text Add to dashboard Cite

This paper presents Bayesian techniques for conservative claims about software reliability, particularly when evidence suggests the software's executions are not statistically independent. We formalise informal notions of “doubting” that the executions are independent, and incorporate such doubts into reliability assessments. We develop techniques that reveal the extent to which independence assumptions can undermine conservatism in assessments, and identify conditions under which this impact is not significant. These techniques – novel extensions of conservative Bayesian inference (CBI) approaches – give conservative confidence bounds on the software's failure probability per execution. With illustrations in two application areas – nuclear power‐plant safety and autonomous vehicle (AV) safety – our analyses reveals: (1) the confidence an assessor should possess before subjecting a system to operational testing. Otherwise, such testing is futile – favourable operational testing evidence will eventually decrease one's confidence in the system being sufficiently reliable; (2) the independence assumption supports conservative claims sometimes; (3) in some scenarios, observing a system operate without failure gives less confidence in the system than if some failures had been observed; (4) building confidence in a system is very sensitive to failures – each additional failure means significantly more operational testing is required, in order to support a reliability claim.

show abstract

Toward a Formalism for Conservative Claims about the Dependability of Software-Based Systems

Cited by 40 publications

References 14 publications

Conservative claims about the probability of perfection of software-based systems

Conservative claims about the probability of perfection of software-based systems

Conservative Bounds for the pfd of a 1-out-of-2 Software-Based System Based on an Assessor's Subjective Probability of "Not Worse Than Independence"

Demonstrating software reliability using possibly correlated tests: Insights from a conservative Bayesian approach

Contact Info

Product

Resources

About