The interplay of compile-time and run-time options for performance prediction

Abstract

Many software projects are configurable through compile-time options (e.g., using ./configure) and also through run-time options (e.g., command-line parameters, fed to the software at execution time). Several works have shown how to predict the effect of run-time options on performance. However it is yet to be studied how these prediction models behave when the software is built with different compile-time options. For instance, is the best run-time configuration always the best w.r.t. the chosen compilation options? In this paper, we investigate the effect of compile-time options on the performance distributions of 4 software systems. There are cases where the compiler layer effect is linear which is an opportunity to generalize performance models or to tune and measure runtime performance at lower cost. We also prove there can exist an interplay by exhibiting a case where compile-time options significantly alter the performance distributions of a configurable system. CCS CONCEPTS• Software and its engineering → Software product lines; Software product lines; Software performance; • Computing methodologies → Machine learning.

Artifact for paper "Lifelong Dynamic Optimization for Self-Adaptive Systems: Fact or Fiction?" published at IEEE SANER 2022

“…As a concrete example in Figure 2, STORM can handle data streaming process under various environments on the incoming batch of jobs, e.g., ROLLINGCOUNT and WORDCOUNT. We see that (1) although the default plan leads to different results across the environments, it performs fairly poor overallcompared with the global optimum, it is 6.4× worse on ROLLINGCOUNT and 11.5× worse on WORDCOUNT; (2) the global optimum can differ for different environments. All above motivate the need of self-adaptation for STORM, where the aim is to achieve the best possible throughput by searching the right adaptation plan, e.g., settings for num_counters and num_splitters, over changing environments.…”

“…Many software systems are highly-configurable, thereby making them flexible to different needs. When operating under dynamic and uncertain environment, those systems are capable of changing their own configuration at runtime with an aim to achieve the best of their performance objective, e.g., higher throughput or smaller latency [1], [2] -a typical type of self-adaptive systems (SASs) that we consider in this work. For example, APACHE STORM, a stream processing framework, can change some adaptation options (e.g., num_counters and num_splitters) at runtime to react to the changing batch of the jobs with different types and workloads [3], [4].…”

Reproducible Science and Deep Software Variability

“…I report on preliminary evidence of complex interactions between variability layers. For instance, run-time options (e.g., command line parameters) interact with compile-time options (e.g., using ./configure) with different effects of non-functional properties of a software [10]. There are various consequences w.r.t.…”

Towards Incremental Build of Software Configurations

“…Configurations are used to build or execute variants and are subject to intensive research. For instance, building variants is a necessary step before deriving performance prediction models [4,14,19,20,28]. Formal methods and program analysis can identify some classes of configuration defects [7,40], leading to variability-aware testing approaches (e.g., [12, 21-24, 29, 35, 36, 38, 42]).…”