Empirical comparison of regression methods for variability-aware performance prediction

Valov, Pavel; Guo, Jianmei; Czarnecki, Krzysztof

doi:10.1145/2791060.2791069

Cited by 37 publications

(52 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) Common changes developers choose to alter in the software system. In practice, it is these factors that affect the performance of systems the most [20], [24], [25], [32].…”

Section: Definitions and Problem Statementmentioning

confidence: 99%

“…As evidenced by our results that are remarkably stable over 30 repeated runs. • Learner Bias: There are various models used in performance optimization such as Gaussian Process [25], Regression Trees [7], [26], [27], and Bagging, Random Forest, and Support Vector Machines (SVMs) [5]. It is possible that changing the learner used may change our findings.…”

Section: Threats To Validitymentioning

confidence: 99%

“…Given the large number of configurations, it becomes increasingly difficult to assess the impact of the configuration options on the system's performance. To address this issue, a common practice is to employ performance prediction models to estimate the performance of the system under these configurations [26], [32], [54]- [57]. To leverage the full benefit of a software system and its features, researchers augment performance prediction models to enable performance optimization [8], [12].…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Krishna

Nair

Jamshidi

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

As software systems grow in complexity and the space of possible configurations increases exponentially, finding the near-optimal configuration of a software system becomes challenging. Recent approaches address this challenge by learning performance models based on a sample set of configurations. However, collecting enough sample configurations can be very expensive since each such sample requires configuring, compiling, and executing the entire system using a complex test suite. When learning on new data is too expensive, it is possible to use Transfer Learning to "transfer" old lessons to the new context. Traditional transfer learning has a number of challenges, specifically, (a) learning from excessive data takes excessive time, and (b) the performance of the models built via transfer can deteriorate as a result of learning from a poor source. To resolve these problems, we propose a novel transfer learning framework called BEETLE, which is a "bellwether"-based transfer learner that focuses on identifying and learning from the most relevant source from amongst the old data. This paper evaluates BEETLE with 57 different software configuration problems based on five software systems (a video encoder, an SAT solver, a SQL database, a high-performance C-compiler, and a streaming data analytics tool). In each of these cases, BEETLE found configurations that are as good as or better than those found by other state-of-the-art transfer learners while requiring only a fraction ( 1 7 th) of the measurements needed by those other methods. Based on these results, we say that BEETLE is a new high-water mark in optimally configuring software.

show abstract

“…2) Common changes developers choose to alter in the software system. In practice, it is these factors that affect the performance of systems the most [20], [24], [25], [32].…”

Section: Definitions and Problem Statementmentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Krishna

Nair

Jamshidi

2021

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…There are numerous works for predicting the performance of configurations (e.g., [5,12,13,17,19]). We share the need to learn over a sample of measured configurations.…”

Section: Related Workmentioning

confidence: 99%

VaryLATEX

Acher

Temple

Jézéquel

et al. 2018

Proceedings of the 12th International Workshop on Variability Modelling of Software-Intensive Systems

View full text Add to dashboard Cite

How to submit a research paper, a technical report, a grant proposal, or a curriculum vitae that respect imposed constraints such as formatting instructions and page limits? It is a challenging task, especially when coping with time pressure. In this work, we present VaryL A T E X, a solution based on variability, constraint programming, and machine learning techniques for documents written in L A T E X to meet constraints and deliver on time. Users simply have to annotate L A T E X source files with variability information, e.g., (de)activating portions of text, tuning figures' sizes, or tweaking line spacing. Then, a fully automated procedure learns constraints among Boolean and numerical values for avoiding non-acceptable paper variants, and finally, users can further configure their papers (e.g., aesthetic considerations) or pick a (random) paper variant that meets constraints, e.g., page limits. We describe our implementation and report the results of two experiences with VaryL A T E X.

show abstract

“…The prediction of the performance of individual variants is subject to intensive research. Approaches usually handle a small sample of measured variants and seek to understand the correlation between configurations and performance [15,27,30,34,37]. The e↵ectiveness of statistical learning techniques and regression methods have been empirically studied.…”

Section: Related Workmentioning

confidence: 99%

Using machine learning to infer constraints for product lines

Temple

Galindo

Acher

et al. 2016

Proceedings of the 20th International Systems and Software Product Line Conference

View full text Add to dashboard Cite

Variability intensive systems may include several thousand features allowing for an enormous number of possible configurations, including wrong ones (e.g. the derived product does not compile). For years, engineers have been using constraints to a priori restrict the space of possible configurations, i.e. to exclude configurations that would violate these constraints. The challenge is to find the set of constraints that would be both precise (allow all correct configurations) and complete (never allow a wrong configuration with respect to some oracle). In this paper, we propose the use of a machine learning approach to infer such product-line constraints from an oracle that is able to assess whether a given product is correct. We propose to randomly generate products from the product line, keeping for each of them its resolution model. Then we classify these products accord-ing to the oracle, and use their resolution models to infer crosstree constraints over the product-line. We validate our approach on a product-line video generator, using a simple computer vision algorithm as an oracle. We show that an interesting set of cross-tree constraint can be generated, with reasonable precision and recall.

show abstract

Empirical comparison of regression methods for variability-aware performance prediction

Cited by 37 publications

References 13 publications

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

Whence to Learn? Transferring Knowledge in Configurable Systems Using BEETLE

VaryLATEX

Using machine learning to infer constraints for product lines

Contact Info

Product

Resources

About