Variability-aware performance prediction: A statistical learning approach

Abstract

Configurable software systems allow stakeholders to derive program variants by selecting features. Understanding the correlation between feature selections and performance is important for stakeholders to be able to derive a program variant that meets their requirements. A major challenge in practice is to accurately predict performance based on a small sample of measured variants, especially when features interact. We propose a variability-aware approach to performance prediction via statistical learning. The approach works progressively with random samples, without additional effort to detect feature interactions. Empirical results on six real-world case studies demonstrate an average of 94 % prediction accuracy based on small random samples. Furthermore, we investigate why the approach works by a comparative analysis of performance distributions. Finally, we compare our approach to an existing technique and guide users to choose one or the other in practice.

Performance-influence models for highly configurable systems

“…Our approach aims at determining the individual influences of configuration options and their interactions, which has several use cases, such as performance-bug detection or configuration optimization. There are many successful approaches that aim at finding optimal configurations without pinpointing the influence of configuration options explicitly [7,12,13]. More closely related to our work are standard machine-learning techniques, such as supportvector machines, Bayesian nets, and evolutionary algorithms.…”

Learning Contextual-Variability Models

“…We have considered 10 publicly available configurable systems (see Table 1) for which we have reused performance measurements (execution time, footprint, etc.) of configurations using benchmarks [4,7]. We used our learning method to synthesize constraints in such a way we only retain software configurations meeting a certain performance objective.…”

Using machine learning to infer constraints for product lines

“…The prediction of the performance of individual variants is subject to intensive research. Approaches usually handle a small sample of measured variants and seek to understand the correlation between configurations and performance [15,27,30,34,37]. The e↵ectiveness of statistical learning techniques and regression methods have been empirically studied.…”