Covering arrays for efficient fault characterization in complex configuration spaces

Abstract

Abstract-Many modern software systems are designed to be highly configurable so they can run on and be optimized for a wide variety of platforms and usage scenarios. Testing such systems is difficult because, in effect, you are testing a multitude of systems, not just one. Moreover, bugs can and do appear in some configurations, but not in others. Our research focuses on a subset of these bugs that are "option-related"-those that manifest with high probability only when specific configuration options take on specific settings. Our goal is not only to detect these bugs, but also to automatically characterize the configuration subspaces (i.e., the options and their settings) in which they manifest. To improve efficiency, our process tests only a sample of the configuration space, which we obtain from mathematical objects called covering arrays. This paper compares two different kinds of covering arrays for this purpose and assesses the effect of sampling strategy on fault characterization accuracy. Our results strongly suggest that sampling via covering arrays allows us to characterize option-related failures nearly as well as if we had tested exhaustively, but at a much lower cost. We also provide guidelines for using our approach in practice.

Test Algebra for Concurrent Combinatorial Testing

“…This classification analysis is then used either to permit a system developer to focus on a small collection of possible faults, or to design additional tests to further restrict the set of possible faults. In Yilmaz et al (2004), empirical results demonstrate the effectiveness of this strategy at limiting the possible faulty interactions to a manageable number.…”

“…At the same time they often detect more failures earlier and with fewer test cases. According to Cohen et al (1997); Dalal et al (1999); Yilmaz et al (2004), combinatorial interaction testing yields small test suites with high code coverage and good fault detection ability. In CIT, construction of the best test suite Grindal et al (2005b); Nie and Leung (2011) can be costly; even a solution with a small number of tests that guarantees complete coverage of t-way interactions may be difficult to produce.…”

Community-based, collaborative testing and analysis

“…The dotted line surrounds a neighborhood of ACE+TAO configurations that failed for the same underlying reason. In another implementation [8,4], we developed techniques for systematically sampling the design space, using mathematical covering arrays [2]. A covering array induces a configuration sample in which all t-way interactions between options are observed at least once.…”

Interaction Coverage Meets Path Coverage by SMT Constraint Solving

“…There has been a large body of work on generating test suites and configuration samples using CIT [1,3,15,22,18]. Applications of CIT include functional input testing, [3], configuration sampling [22], regression testing [18] and more recently event sequence testing [23].…”