A Comprehensive Investigation of Modern Test Suite Optimization Trends, Tools and Techniques

Abstract

Software testing is an important but expensive activity of software development life cycle, as it accounts for more than 52% of entire development cost. Testing requires the execution of all possible test cases in order to find the defects in the software. Therefore, different test suite optimization approaches like the genetic algorithm and the greedy algorithm, etc., are widely used to select the representative test suite without compromising the effectiveness. Test suite optimization is frequently researched to enhance its competences but there is no study published until now that analyzes the latest developments from 2016 to 2019. Hence, in this article, we systematically examine the state-of-the-art optimizations' approaches, tools, and supporting platforms. Principally, we conducted a systematic literature review (SLR) to inspect and examine 58 selected studies that are published during 2016-2019. Subsequently, the selected researches are grouped into five main categories, i.e., greedy algorithm (seven studies), meta-heuristic (28 studies), hybrid (six studies), clustering (five studies), and general (12 studies). Finally, 32 leading tools have been presented, i.e., existing tools (25 tools) and proposed/developed tools (seven tools) along 14 platform supports. Furthermore, it is noted that several approaches aim at solving the single-objective optimization problem. Therefore, researchers should focus on dealing with the multi-objective problem, as multi-objective versions outperform the single-objective ones. Moreover, less attention has been given to clustering-based techniques. Thus, we recommend exploring the machine learning and artificial intelligencebased optimization approaches in the future. A broad exploration of tools and techniques, in this article, will help researchers, practitioners, and developers to opt for adequate techniques, tools, or platforms as per requirements.

Multiobjective Test Case Prioritization Using Test Case Effectiveness: Multicriteria Scoring Method

“…Regression testing also confirms that the previous functions of the software are working as per specifications Scientific Programming [2]. e main concept behind regression testing is retesting of SUT with the aim to expose the faults earlier [23].…”

Machine Learning for Software Engineering: A Tertiary Study

“…Here ML techniques are mainly used for test automation, specifically for test case generation, evaluation, and optimization. Genetic algorithms are frequently proposed [80,118]. In the test oracle problem, which is concerned with a software's output behavior based on a set of inputs, automation is achieved by training ML models to predict the outcome [48].…”

“…In mutation testing there is considerable room for expediting existing ML-based solutions for detecting possible equivalent mutants as well as automating more facets of the process, which are currently handled by humans [48]. In test suite optimization there are only few ML-based approaches [80].…”

Multi-Task Optimization-Based Test Data Generation for Mutation Testing via Relevance of Mutant Branch and Input Variable

“…To efficiently generate test data for killing the mutants, many scholars proposed various methods and developed a number of tools [17]- [19], to improve the efficiency of generating test data. Papadakis et al [5] summarized three main kinds of methods of generating test data for mutation testing, i.e., constraint-based test generation [20], dynamic symbolic execution [8], [21], and search-based test generation [22]- [24].…”