Supporting the Statistical Analysis of Variability Models

Abstract

Variability models are broadly used to specify the configurable features of highly customizable software. In practice, they can be large, defining thousands of features with their dependencies and conflicts. In such cases, visualization techniques and automated analysis support are crucial for understanding the models. This paper contributes to this line of research by presenting a novel, probabilistic foundation for statistical reasoning about variability models. Our approach not only provides a new way to visualize, describe and interpret variability models, but it also supports the improvement of additional state-of-the-art methods for software product lines; for instance, providing exact computations where only approximations were available before, and increasing the sensitivity of existing analysis operations for variability models. We demonstrate the benefits of our approach using real case studies with up to 17,365 features, and written in two different languages (KConfig and feature models).

Uniform and scalable sampling of highly configurable systems

“…This paper extends our paper in SPLC'20 (Heradio et al 2020), where (i) a statistical test is formulated to reduce the sample size required for assessing a samplers' uniformity, and (ii) population parameters are computed with scalable algorithms we proposed in Heradio et al (2019). The additional contributions of this present paper are:…”

“…Therefore, d = 100 ⋅ |0.01−0.015| 0.01 = 50 , and thus the sampler uniformity would be rejected. The chances that these types of wrong diagnoses happen increases with the number of low-probability variables, and it is worth noting that real models with numerous low-probability variables are not "corner cases"; for example, in three out of the seven configuration models analyzed in Heradio et al (2019), more than 46% of their variables have p ≤ 0.05 : the open-source project Fiasco v2014092821, the Dell laptop configurator, and the Automotive 02 system.…”

“…The distribution of the whole population of SAT-solutions of a model can be computed with the Product Distribution (PD) 13 algorithm we proposed in Heradio et al (2019). PD takes the BDD encoding of a model as input, and as explained in Section 3.D of Heradio et al (2019), its time complexity is O(nv 2 ) , where n is the number of BDD nodes and v the number of model variables. Accordingly, PD scales for large models.…”

“…As we mentioned in Section 5.4, before the publication of FP in Heradio et al (2019), the literature relied on limited tests that only could handle the simplest models with a few hundred variables. As more complex are considered, the chances to detect samplers' additional flaws increases.…”

“…Verifying that these algorithms and their tools indeed generate genuine uniform samples is a challenge by itself, because it requires examining the consistency between sample statistics and their corresponding population parameters (e.g., how frequently a feature appears in a sample compared to its probability of being included in every valid configuration (Heradio et al 2019)). As configuration spaces can be colossal, current procedures that certify a sampler's uniformity has the severe shortcoming of requiring gigantic sample sizes to estimate reliable statistics (Dutra et al 2018;Achlioptas et al 2018;Chakraborty and Meel 2019).…”