SPLat: lightweight dynamic analysis for reducing combinatorics in testing configurable systems

Abstract

Many programs can be configured through dynamic and/or static selection of configuration variables. A software product line (SPL), for example, specifies a family of programs where each program is defined by a unique combination of features. Systematically testing SPL programs is expensive as it can require running each test against a combinatorial number of configurations. Fortunately, a test is often independent of many configuration variables and need not be run against every combination. Configurations that are not required for a test can be pruned from execution. This paper presents SPLat, a new way to dynamically prune irrelevant configurations: the configurations to run for a test can be determined during test execution by monitoring accesses to configuration variables. SPLat achieves an optimal reduction in the number of configurations and is lightweight compared to prior work that used static analysis and heavyweight dynamic execution. Experimental results on 10 SPLs written in Java show that SPLat substantially reduces the total test execution time in many cases. Moreover, we demonstrate the scalability of SPLat by applying it to a large industrial code base written in Ruby on Rails.

Exploring variability-aware execution for testing plugin-based web applications

“…Kim et al and Shi et al have explored static and dynamic analyses to avoid reexecutions of configurations that have exactly the same execution path [36,38,50]. They demonstrated in only one large industrial application and mostly small examples for unit tests with few configuration options.…”

Family-Based Model Checking with mCRL2

“…Then we define a partitioning procedure for μL f that allows us to apply our results from [20]. Next, we evaluate our approach by verifying a number of representative properties over an mCRL2 specification of the minepump SPL benchmark model [8,13,[23][24][25]. We verify typical lineartime and branching-time properties.…”

Using machine learning to infer constraints for product lines

“…Another research direction focuses on the identification of relevant products to test [10,20,21,31,32]. Kim et al [20] applied static program analysis techniques to find irrelevant features for a test.…”

“…Kim et al [20] applied static program analysis techniques to find irrelevant features for a test. SPLat is a dynamic analysis technique for pruning irrelevant configurations [21]. The goal is to reduce the combinatorial number of variants (e.g., Java programs) to examine.…”