Covering arrays for efficient fault characterization in complex configuration spaces

Abstract

Many modern software systems are designed to be highly configurable so they can run on and be optimized for a wide variety of platforms and usage scenarios. Testing such systems is difficult because, in effect, you are testing a multitude of systems, not just one. Moreover, bugs can and do appear in some configurations, but not in others. Our research focuses on a subset of these bugs that are "option-related"-those that manifest with high probability only when specific configuration options take on specific settings. Our goal is not only to detect these bugs, but also to automatically characterize the configuration subspaces (i.e., the options and their settings) in which they manifest. To improve efficiency, our process tests only a sample of the configuration space, which we obtain from mathematical objects called covering arrays. This paper compares two different kinds of covering arrays for this purpose and assesses the effect of sampling strategy on fault characterization accuracy. Our results strongly suggest that sampling via covering arrays allows us to characterize option-related failures nearly as well as if we had tested exhaustively, but at a much lower cost. We also provide guidelines for using our approach in practice.

A Review of Covering Arrays and Their Application to Software Testing

“…For the first time, the research reports the application of VSCA to tests the effects of using it practically. The use of VSCA allows testing stronger interactions for subspaces where they are needed (i.e., in high-risk subspaces) and keeping a lower strength of interaction across the entire space (Yilmaz et al, 2006). The research reports the same finding for the CA as in the case of the original research.…”

“…The research concludes that the fault characterization could be improved in term of accuracy with low cost if the low strength CA be used. Yilmaz et al (2006) extend the above research to include more empirical studies. For the first time, the research reports the application of VSCA to tests the effects of using it practically.…”

Constructing heterogeneous hash families by puncturing linear transversal designs

“…Applications to interaction testing, in particular to testing component-based software, have driven much recent research; see [10][11][12][13]54]. In applications in testing, columns of the array correspond to experimental factors, and the symbols in the column form values or levels for the factor.…”

Constructing Covering Arrays using Parallel Computing and Grid Computing

“…Colbourn et al (2005) applied combinatorial testing to progressive ranking and composition of Web services. Yilmaz et al (2006) applied combinatorial testing to an open-source CORBA middleware implementation ACE+TAO. Kuhn et al (2010) presented the use of combinatorial testing in a smart phone application.…”

“…Cawse (2003) used covering arrays for design of materials, Hedayat et al (1999) used them in medicine and agriculture; in biology and industrial processes have also been used by Shasha et al (2001) and Phadke (1995). Covering arrays have been used in hardware testing Vadde and Syrotiuk (2004) but significantly the area with the major application of these objects is in software testing Burr and Young (1998);Yilmaz et al (2006).…”