Mastering uncertainty in performance estimations of configurable software systems

Dorn, Johannes; Apel, Sven; Siegmund, Norbert

doi:10.1145/3324884.3416620

Cited by 19 publications

(9 citation statements)

References 46 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The results are also evaluated validated Scott-Knott test [38]. We also set a data samples of 5, 000, which tends to be reasonable as this is what has been commonly used in prior work [19,22,34,47]. Indeed, using other metrics or different sample size may offer new insights, which we plan to do in future work.…”

Section: Threats To Validitymentioning

confidence: 99%

“…Further, using the full samples for some systems with a large configuration space can easily lead to unrealistic training time for certain models, e.g., with Neural Network, it took several days to complete only one run under our learning pipeline on the full datasets of Trimesh. Therefore in this work, for each system, we randomly sample 5,000 configurations from the dataset as our experiment data, which tends to be reasonable and is also a commonly used setting in previous work [19,22,34,47].…”

Section: System and Data Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

Gong,

Chen

2022

Preprint

View full text Add to dashboard Cite

Learning and predicting the performance of a configurable software system helps to provide better quality assurance. One important engineering decision therein is how to encode the configuration into the model built. Despite the presence of different encoding schemes, there is still little understanding of which is better and under what circumstances, as the community often relies on some general beliefs that inform the decision in an ad-hoc manner. To bridge this gap, in this paper, we empirically compared the widely used encoding schemes for software performance learning, namely label, scaled label, and one-hot encoding. The study covers five systems, seven models, and three encoding schemes, leading to 105 cases of investigation.Our key findings reveal that: (1) conducting trial-and-error to find the best encoding scheme in a case by case manner can be rather expensive, requiring up to 400+ hours on some models and systems;(2) the one-hot encoding often leads to the most accurate results while the scaled label encoding is generally weak on accuracy over different models; (3) conversely, the scaled label encoding tends to result in the fastest training time across the models/systems while the one-hot encoding is the slowest; (4) for all models studied, label and scaled label encoding often lead to relatively less biased outcomes between accuracy and training time, but the paired model varies according to the system.We discuss the actionable suggestions derived from our findings, hoping to provide a better understanding of this topic for the community. To promote open science, the data and code of this work can be publicly accessed at https://github.com/ideas-labo/MSR2022encoding-study. CCS Concepts• Software and its engineering → Software performance.

show abstract

Section: Threats To Validitymentioning

confidence: 99%

Section: System and Data Selectionmentioning

confidence: 99%

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

Gong,

Chen

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In this way, users and developers can better understand how different interactions influence the system performance and make rational configuration decisions to maximize software efficacy. Besides, there is also some work on increasing explainability of the performance model by measuring the uncertainty in performance estimations with Bayesian method [5]. We will also consider the uncertainty measurement of our method in the future work.…”

Section: Related Workmentioning

confidence: 99%

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

Cheng

Gao

Zheng

2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Modern software systems are usually highly configurable, providing users with customized functionality through various configuration options. Understanding how system performance varies with different option combinations is important to determine optimal configurations that meet specific requirements. Due to the complex interactions among multiple options and the high cost of performance measurement under a huge configuration space, it is challenging to study how different configurations influence the system performance. To address these challenges, we propose HINNPerf , a novel hierarchical interaction neural network for performance prediction of configurable systems. HINNPerf employs the embedding method and hierarchic network blocks to model the complicated interplay between configuration options, which improves the prediction accuracy of the method. Besides, we devise a hierarchical regularization strategy to enhance the model robustness. Empirical results on 10 real-world configurable systems show that our method statistically significantly outperforms state-of-the-art approaches by achieving average 22.67% improvement in prediction accuracy. In addition, combined with the Integrated Gradients method, the designed hierarchical architecture provides some insights about the interaction complexity and the significance of configuration options, which might help users and developers better understand how the configurable system works and efficiently identify significant options affecting the performance.

show abstract

“…To obtain 𝜌 for real-world systems, Siegmund et al [76,77] presented a black-box method to generate linear-equation models for performance measures by multivariable linear regression on sampled configurations. Other black-box approaches rely on regression trees [43], Fourier learning [95], or probabilistic programming [27]. Related white-box approaches use insights of local measurements and taint analysis information [87] or profiling information [90].…”

Section: Effect Properties and Setsmentioning

confidence: 99%

Causality in Configurable Software Systems

Dubslaff,

Weis,

Baier

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Detecting and understanding reasons for defects and inadvertent behavior in software is challenging due to their increasing complexity. In configurable software systems, the combinatorics that arises from the multitude of features a user might select from adds a further layer of complexity. We introduce the notion of feature causality, which is based on counterfactual reasoning and inspired by the seminal definition of actual causality by Halpern and Pearl. Feature causality operates at the level of system configurations and is capable of identifying features and their interactions that are the reason for emerging functional and non-functional properties. We present various methods to explicate these reasons, in particular well-established notions of responsibility and blame that we extend to the feature-oriented setting. Establishing a close connection of feature causality to prime implicants, we provide algorithms to effectively compute feature causes and causal explications. By means of an evaluation on a wide range of configurable software systems, including community benchmarks and real-world systems, we demonstrate the feasibility of our approach: We illustrate how our notion of causality facilitates to identify root causes, estimate the effects of features, and detect feature interactions.

show abstract

Mastering uncertainty in performance estimations of configurable software systems

Cited by 19 publications

References 46 publications

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

Causality in Configurable Software Systems

Contact Info

Product

Resources

About