On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software

Fifo, Miraldi; Enoiu, Eduard Paul; Afzal, Wasif

doi:10.1109/icstw.2019.00062

Cited by 6 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• An analysis of tests for Bombardier Transportation discovered that manually created tests by engineers did not achieve strong combinatorial coverage, with an average of 78.6% for 2-way interactions, 57% for 3-way interactions, 40.2% for 4-way interactions, 20.2% for 5-way interactions and 13% for 6-way interactions.. The measures were used to determine improvements to test sets for critical code [20].…”

Section: Applications Of Measurementmentioning

confidence: 99%

Combinatorial Coverage Difference Measurement

Kuhn¹,

Raunak²,

Kacker³

2021

View full text Add to dashboard Cite

Structural coverage criteria are widely used tools in software engineering, useful for measuring aspects of test execution thoroughness. However, in many cases structural coverage may not be applicable, either because source code is not available, or because processing is based on neural networks or other black-box components. Vulnerability and fault detection in such cases will typically rely on large volumes of tests, with the goal of discovering flaws that result in system failures or security weaknesses. This publication explains combinatorial coverage difference measures that have been applied to problems that include fault identification and autonomous systems validation, and documents functions of research tools for computing these measures. The metrics and tools described are introduced as research tools; later work will be useful in determining which are of value in assurance and testing or simulation.

show abstract

Section: Applications Of Measurementmentioning

confidence: 99%

Combinatorial Coverage Difference Measurement

Kuhn¹,

Raunak²,

Kacker³

2021

View full text Add to dashboard Cite

show abstract

“…Methods and tools for measuring combinatorial coverage were initially developed to analyze the degree to which test sets included t-way combinations of values (for some specified level of t) [1][2] [4] and have since been studied extensively in the realm of system and software testing [7][8] [9][10] [11]. Combinatorial coverage measures have been defined and applied to a wide range of problems, specifically for fault location and for evaluating the adequacy of test inputs and input space models.…”

Section: Introductionmentioning

confidence: 99%

Combination Frequency Differencing

Kuhn¹,

Raunak²,

Kacker³

2021

View full text Add to dashboard Cite

This paper introduces a new method related to combinatorial testing and measurement, combination frequency differencing (CFD), and illustrates the use of CFD in machine learning applications. Combinatorial coverage measures have been defined and applied to a wide range of problems, including fault location and for evaluating the adequacy of test inputs and input space models. More recently, methods applying coverage measures have been used in applications of artificial intelligence and machine learning, for explainability and for analyzing aspects of transfer learning. These methods have been developed using measures that depend on the inclusion or absence of t-tuples of values in inputs, training data, and test cases. In this paper, we extend these combinatorial coverage measures to include the frequency of occurrence of combinations. Combination frequency differencing is particularly suited to AI/ML applications, where training data sets used in learning systems are dependent on the prevalence of various attributes of elements of class and non-class sets. We illustrate the use of this method by applying it to analyzing physically unclonable functions (PUFs) for bit combinations that disproportionately influences PUF response values, and in turn provides indication of the PUF potentially being more vulnerable to model-building attacks. Additionally, it is shown that combination frequency differences provide a simple but effective algorithm for classification problems.

show abstract