The Effectiveness of Statistical Testing when Applied to Logic Systems

Kuball, S.; Hughes, Gordon; May, John H R; Gallardo, Julio César Rubio; John, Andrew D.; Carter, Roy B.

doi:10.1007/3-540-45416-0_16

Cited by 3 publications

(6 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Cleanroom is an engineering and managerial process for the development of high-quality software with certified reliability [8]. Kuball et al [9] demonstrate the effectiveness of statistical testing for error detection by comparing the result of statistical testing with other testing methods. They also discuss the potential of statistical testing for reliability quantification.…”

Section: Statistical Testingmentioning

confidence: 99%

A Framework for Statistical Testing of Software Components

Shukla

Strooper

Carrington

2007

Int. J. Soft. Eng. Knowl. Eng.

View full text Add to dashboard Cite

Statistical testing involves the testing of software by selecting test cases from a probability distribution that is intended to represent the software's operational usage. In this paper, we describe and evaluate a framework for statistical testing of software components that incorporates test case execution and output evaluation. An operational profile and a test oracle are essential for the statistical testing of software components because they are used for test case generation and output evaluation respectively. An operational profile is a set of input events and their associated probabilities of occurrence expected in actual operation. A test oracle is a mechanism that is used to check the results of test cases. We present four types of operational profiles and three types of test oracles, and empirically evaluate them using the framework by applying them to two software components. The results show that while simple operational profiles may be effective for some components, more sophisticated profiles are needed for others. For the components that we tested, the fault-detecting effectiveness of the test oracles was similar.

show abstract

Section: Statistical Testingmentioning

confidence: 99%

A Framework for Statistical Testing of Software Components

Shukla

Strooper

Carrington

2007

Int. J. Soft. Eng. Knowl. Eng.

View full text Add to dashboard Cite

show abstract

“…An important goal is to make the generation of ST test cases more generic, and to this end studies of typical safety-related applications and of what statistical testing would mean for such applications need to be performed. Examples of papers discussing ST and the design and use of statistical test-cases are, for example references [2][3][4][5][6][7].…”

Section: Introductionmentioning

confidence: 99%

“…performance of statistical testing on the particular device code discussed, and analysis of the results, also forms part of the present authors' work, but remains outstanding. This second part has been performed on other examples of systems [7,8].…”

Section: Introductionmentioning

confidence: 99%

A discussion of statistical testing on a safety-related application

Kuball

May

2007

Proceedings of the Institution of Mechanical Engineers, Part O:

View full text Add to dashboard Cite

Safety-critical and safety-related systems are increasingly based, at least partially, on the use of software or logic components. High integrity claims are usually placed on these systems, which means that their probability of failure during operation should be below a specified level in order to ensure that risk of operation is sufficiently low. All this implies that some knowledge needs to be gained on the dependability of these systems or components in actual field use. Dependability assessment methods for software are not as well established as for hardware. Currently, formal proofs and statistical testing methods provide the only methods that have the potential to assess software dependability quantitatively. The present paper explores the applicability of statistical (software) testing (ST) to the example of a real safety-related software application. It discusses the key points arising in this task and highlights the unique and important role ST can play within the wider task of software verification.Software-or logic-based components increasingly play a role in the safety-critical industry, e.g. the nuclear industry, avionics, rail, automotive, and medical applications. For example, in nuclear safetyrelated systems the situation is encountered where obsolete hardwired components have to be replaced with programmable electronic systems (PESs). Also, smart sensors, which contain a software element, are increasingly employed here, and computer-based data processing systems are used to supply controlroom staff with essential information on the status of the plant. The software parts of these systems exhibit systematic failure and design faults rather than random hardware failure processes, and this creates important issues for safety assessment. Models to cope with random failure processes exist and have been widely used, but they do not apply to the presence of systematic failures. An additional problem arises in the assessment of smart devices. These devices are often commercial-off-the-shelf (COTS) components, and assessment techniques that would commonly be used are not available since it is sometimes not possible or permissible to access the code in order to perform tasks such as code inspection or static analysis.In order to address the issue of software failure, current industry standards [1] use clauses recommending system design techniques, which tools to use, how to manage the project team, and how to test the code (partition testing, coverage-based testing, etc.). Depending on which recommendations are followed, the users of a software-based system can claim that the dependability of their system falls within one of many safety integrity levels that represent target dependability or availability measures. Important as they are, these recommendations cannot actually measure or quantify whether target dependability has been achieved. Therefore a key requirement for any prospective software assurance method is increased objectivity through quantified dependability statements. There are two main metho...

show abstract

“…If the same amount of testing has been spread across different techniques, such as statement coverage tests, decision coverage tests, Boolean coverage tests etc, there are more types of faults that are potentially attacked, that would have been spotted with the prior test-set and that can thus be eliminated from our model of residual faults. In [20] for example it is shown that sets tests can have different errordetection effectiveness. Plant simulation tests for example consisted of re-running the same set of test-inputs with varying time-intervals.…”

Section: Influence Of Prior Testingmentioning

confidence: 99%

“…Targeted random tests would tackle faults associated with small deviations from the normal operating sequence, thus they would test e.g. the triggering and issuing of error messages and -as shown in [20] -the action carried out by the software in the presence of such messages. Thus faults associated with receiving conflicting signals such as "turret rotation demanded clockwise and counter-clockwise" would potentially have been tackled.…”

Section: Influence Of Prior Testingmentioning

confidence: 99%

Test-Adequacy and Statistical Testing: Combining Different Properties of a Test-Set

Kuball

May

15th International Symposium on Software Reliability Engineering

View full text Add to dashboard Cite

Dependability assessment of safety-critical or safetyrelated software components is an important issue for example within the nuclear industry, the avionics sector or the military. Statistical testing is one way of quantifying the dependability of a given software product. The use of sector-specific standards with their suggested test-criteria is another (non-quantitative) way of aiming at employing only components that are "dependable enough". Ideally, both, the acknowledged test criteria and statistical test methods should come into play when assessing software dependability. We want to -in the long-term -move towards this aim. Thus we investigate in this paper a model to combine the fault-detection power of a given testset (a test-adequacy criterion) with the statistical power of the test-set, i.e. the number of statistical tests within the test-set. With this model we aim at drawing out of any given test-set -whether devised by a plant engineer or a statistician -the overall contribution it can make to dependability assessment. Notation: STStatistical testing, Also known as: reliability engineered testing or operational-profile testing. pfdProbability of (software) failure on demand. OPOperational Profile. Pr(E) Probability of a random event E.

show abstract

The Effectiveness of Statistical Testing when Applied to Logic Systems

Cited by 3 publications

References 4 publications

A Framework for Statistical Testing of Software Components

A Framework for Statistical Testing of Software Components

A discussion of statistical testing on a safety-related application

Test-Adequacy and Statistical Testing: Combining Different Properties of a Test-Set

Contact Info

Product

Resources

About