Reliability assessment of service-based software under operational profile uncertainty

Abstract

We address the problem of operational reliability assessment through testing of software services delivered on-demand such as Web Services. Software reliability assessment is typically done for a specific operational profile: the profile is needed in testing to select or generate test cases (operational testing) in a way statistically similar to the anticipated use of software in operation; the observations of success/failure of test executions are used to predict software reliability in actual operation. It is well known that unless the profile is accurate, software reliability predictions obtained via operational testing cannot be trusted.We present a new way of dealing with the uncertainty in the operational profile adopting a two-stage Bayesian inference for reliability assessment. The technique relies on the availability of information about partitions of the input space. The approach is demonstrated on contrived examples and on a case study of real Web Services. We discuss the usefulness of the approach in dealing with two important practical problems: i) the true profile in operation differs from the one used in testing, ii) the profile in operation is changing continuously.

Understanding the Formation of Software as a Service (Saas) Commitment the Relational Value Creation Perspective

“…Studies on online relationship management and user behavior have used the dedication constraint framework to characterize post-adoption behavior. This framework is an appropriate theoretical base for explaining the formation of SaaS commitment because one's dynamic response to SaaS use for motivating commitment is captured by the dual consideration of benefits and investment [16]. However, we extend this framework by combining it with the relational view theory to provide a comprehensive explanation of commitment formation in the SaaS context.…”

Reliability Assessment and Safety Arguments for Machine Learning Components in System Assurance

“…It is therefore common to use a Bernoulli process as the mathematical abstraction of the failure process, which implies a Binomial likelihood. When used for traditional software that, upon establishing the likelihood, RAMs on estimating λ vary case by case-from the basic Maximum Likelihood Estimation (MLE) to Bayesian estimators tailored for certain scenarios when, e.g., seeing no failures [50,12], inferring ultra-high reliability [75], with certain forms of prior knowledge like perfectioness [62], with vague prior knowledge expressed in imprecise probabilities [65,73], with uncertain OPs [13,53], etc.…”

“…Principles and Main Steps of the RAM. Inspired by [53], our RAM first partitions the input domain into m small cells 10 , subject to the r-separation property. Then, for each cell c i (and its ground truth label y i ), we estimate:…”

“…OP-based software testing, also known as statistical/operational testing [61], is an established practice, which is supported by industry standards for conventional systems. There is a huge body of literature in the traditional software reliability community on OP-based testing and reliability modelling techniques, e.g., [9,13,53,75]. In contrast to this, OP-based software testing for ML components is still in its infancy: to the best of our knowledge, there are only two recent works that explicitly consider the OP in testing.…”

“…To the best of our knowledge, our RAM is the first to consider both, the OP and robustness evidence. It is inspired by partition-based testing [28,53], operational/statistical testing [61,75] and ML robustness evaluation [21,66]. Our RAM is model-agnostic and designed for pre-trained ML models, yielding estimates of, e.g., expected values or confidence bounds on the probability of misclassification per random input (pmi) 1 .…”