A Survey of Constrained Combinatorial Testing

Abstract

Combinatorial Testing (CT) is a potentially powerful testing technique, whereas its failure revealing ability might be dramatically reduced if it fails to handle constraints in an adequate and efficient manner. To ensure the wider applicability of CT in the presence of constrained problem domains, large and diverse efforts have been invested towards the techniques and applications of constrained combinatorial testing. In this paper, we provide a comprehensive survey of representations, influences, and techniques that pertain to constraints in CT, covering 129 papers published between 1987 and 2018. This survey not only categorises the various constraint handling techniques, but also reviews comparatively less well-studied, yet potentially important, constraint identification and maintenance techniques. Since real-world programs are usually constrained, this survey can be of interest to researchers and practitioners who are looking to use and study constrained combinatorial testing techniques.

A synergic quantum particle swarm optimisation for constrained combinatorial test generation

“…For the CCAG problem, there are two main categories of constraints handling strategies suit for search-based algorithms [7]. One is avoiding strategy and three representative methods are verifying-based methods, solver-based methods and weight-based methods.…”

“…Therefore, the method needs post-processing checked. Another constraints handling strategy is the post-process strategy which focusses on handling the constraint after test suite generation, and the main approach is the replacing method [7,35]. The replacing methods [28,43] construct CA ignoring constraints at first.…”

“…As an extension of CAG, the CCAG is the most useful technique to generate test suites meeting constraints for real application testing. Many research studies about CAG and CCAG have been proposed in the past decades, and these studies can be found in surveys summarised by Nie and Wu et al [7,20]. Different from mainstream algorithms including greedy algorithms and heuristic search methods, some special technologies based on machine learning [13,21] and a GPU-enabled parallel method [22] have recently appeared.…”

“…The invalid test cases could cause a waste of test resources and lost coverage of the test suite. Although many studies have researched the subject of constraints, effectively handling constraints remains a challenge in CT [7]. Therefore, the constrained CAG (CCAG) is a critical and practical problem in CT.…”

Constrained detecting arrays for fault localization in combinatorial testing

“…In the practice of CIT, constraint handling has always been a vital issue. Surveys about constraint handling for CIT include [13,14,15].…”