Instance Space Analysis of Search-Based Software Testing

Abstract

Search-based software testing (SBST) is now a mature area, with numerous techniques developed to tackle the challenging task of software testing. SBST techniques have shown promising results and have been successfully applied in the industry to automatically generate test cases for large and complex software systems. Their effectiveness, however, has been shown to be problem dependent. In this paper, we revisit the problem of objective performance evaluation of SBST techniques in light of recent methodological advances -in the form of Instance Space Analysis (ISA) -enabling the strengths and weaknesses of SBST techniques to be visualised and assessed across the broadest possible space of problem instances (software classes) from common benchmark datasets. We identify features of SBST problems that explain why a particular instance is hard for an SBST technique, reveal areas of hard and easy problems in the instance space of existing benchmark datasets, and identify the strengths and weaknesses of state-of-the-art SBST techniques. In addition, we examine the diversity and quality of common benchmark datasets used in experimental evaluations.

Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles

“…Along with providing an objective measure of the adequacy of a test suite, these metrics offer valuable insights into the relationship between the structural properties of the test instances and their impact on test outcomes. To estimate TISA metrics, we employ a framework called Instance Space Analysis (ISA) [62] that projects test instances -characterised in terms of features -from 𝑛-dimensional feature space to a 2-dimensional space, called Instance Space (IS). The projections are performed in such a way that there is a clear distinction between failing and passing test scenarios, and the impact of each feature on the test outcome (fail or pass) can easily be mapped.…”

“…The initial stage of this process involves identifying a cluster of features that share similarities with each other. We use k-means clustering for this process as it is one of the simplest unsupervised machine learning algorithms and ISA is shown to perform well with this clustering technique in the previous studies [59][60][61][62]. The optimal number of clusters to be used by k-means is selected using silhouette analysis [8].…”

“…Given the significance of both diversity and coverage for blackbox testing of AVs, we propose a novel set of metrics, called Test suite Instance Space Adequacy (TISA) metrics, that provides an objective measure of the quality of testing in terms of both diversity and coverage. TISA metrics are computed by identifying the features of the test scenarios having maximum impact on test outcome (safe vs unsafe), and then projecting these test scenarios -defined in terms of the impactful features -on 2𝐷 space, using a framework called Instance Space Analysis (ISA) [62]. The representation of the test scenarios in a 2D space facilitates a clear visualisation of the test suite, resulting in an easy evaluation of the diversity and coverage of the whole test suite across each feature, along with the diversity of the detected bugs.…”