Test them all, is it worth it? Assessing configuration sampling on the JHipster Web development stack

Abstract

Many approaches for testing configurable software systems start from the same assumption: it is impossible to test all configurations. This motivated the definition of variability-aware abstractions and sampling techniques to cope with large configuration spaces. Yet, there is no theoretical barrier that prevents the exhaustive testing of all configurations by simply enumerating them if the effort required to do so remains acceptable. Not only this: we believe there is a lot to be learned by systematically and exhaustively testing a configurable system. In this case study, we report on the first ever endeavour to test all possible configurations of the industry-strength, open source configurable software system: JHipster, a popular code generator for web applications. We built a testing scaffold for the 26,000+ configurations of JHipster using a cluster of 80 machines during 4 nights for a total of 4,376 hours (182 days) CPU time. We find that 35.70% configurations fail and we identify the feature interactions that cause the errors. We show that sampling strategies (like A. Halin, G. Perrouin (FNRS research associate) PReCISE, dissimilarity and 2-wise): (1) are more effective to find faults than the 12 default configurations used in the JHipster continuous integration; (2) can be too costly and exceed the available testing budget. We cross this quantitative analysis with the qualitative assessment of JHipster's lead developers.

Uniform Sampling of SAT Solutions for Configurable Systems: Are We There Yet?

“…In their experiments, Medeiros et al relied on the first configuration returned by the SAT solver [24]. However, Halin et al showed that taking the first configuration did not reveal faults in their studied system, while an approach relying on random sampling was effective [25]. In the absence of constraints between the options, Arcuri et al theoretically demonstrate that a uniform random sampling strategy may outperform t-wise sampling [26].…”

“…• RQ3 (relevance for testing): How does QuickSampler's sacrifices on uniformity impact its bug-finding ability in JHipster? By relating sampled frequencies of features with their associated bugs on the JHipster case [25], we perform an early exploration of how these techniques behave with respect to bug distribution [26].…”

“…Second, we include 10 additional feature models used in [8] and not in [30], [31]; they also contain a large number of features (e.g., more than 6,000). Third, we also add the JHipster feature model [25], [32] to the study, a realistic but relatively smaller feature model (45 variables, 26,000+ configurations). We later refer to these benchmarks as the feature model benchmarks.…”

Empirical Assessment of Multimorphic Testing

“…For instance, prior work has considered the problems of optimizing the execution of tests for a set of related products [38], [39], [40], sampling the configurations to test [41], [42], [43], [44], [9], [24], [45], [46], or identifying relevant products to test [37], [47]. In this paper, we do not aim to verify an existing product line; we synthetically create a family of variants with the objective of assessing the quality of a test suite.…”

Towards Quality Assurance of Software Product Lines with Adversarial Configurations

“…Testing and learning SPLs. Testing all configurations of an SPL is most of time challenging and sometimes impossible, due to the exponential number of configurations [31,37,39,[58][59][60]. ML techniques have been developed to reduce cost, time and energy of deriving and testing new configurations using inference mechanisms.…”