Predicting Performance of Software Configurations: There is no Silver Bullet

Grebhahn, Alexander; Siegmund, Norbert; Apel, Sven

doi:10.48550/arxiv.1911.12643

Cited by 5 publications

(7 citation statements)

References 57 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The software engineering community has also invested significant research into finding techniques for optimal application-level configurations. Among the most recent work is an excellent empirical study by Grebhahn et al [27]. The authors evaluated several popular techniques in Software Engineering, including k-Nearest Neighbours, Support Vector Regression and Random Forests for creating surrogate performance models.…”

Section: E Search-based Software Engineeringmentioning

confidence: 99%

See 1 more Smart Citation

Analyzing Search Techniques for Autotuning Image-based GPU Kernels: The Impact of Sample Sizes

Torring¹,

Elster²

2022

Preprint

View full text Add to dashboard Cite

Modern computing systems are increasingly more complex, with their multicore CPUs and GPUs accelerators changing yearly, if not more often. It thus has become very challenging to write programs that efficiently use the associated complex memory systems and take advantage of the available parallelism. Autotuning addresses this by optimizing parameterized code to the targeted hardware by searching for the optimal set of parameters. Empirical autotuning has therefore gained interest during the past decades. While new autotuning algorithms are regularly presented and published, we will show why comparing these autotuning algorithms is a deceptively difficult task.In this paper, we describe our empirical study of state-ofthe-art search techniques for autotuning by comparing them on a range of sample sizes, benchmarks and architectures. We optimize 6 tunable parameters with a search-space size of over 2 million. The algorithms studied include Random Search (RS), Random Forest Regression (RF), Genetic Algorithms (GA), Bayesian Optimization with Gaussian Processes (BO GP) and Bayesian Optimization with Tree-Parzen Estimators (BO TPE).Our results on the ImageCL benchmark suite suggest that the ideal autotuning algorithm heavily depends on the sample size. In our study, BO GP and BO TPE outperform the other algorithms in most scenarios with sample sizes from 25 to 100. However, GA usually outperforms the others for sample sizes 200 and beyond. We generally see the most speedup to be gained over RS in the lower range of sample sizes (25-100). However, the algorithms more consistently outperform RS for higher sample sizes (200-400). Hence, no single state-of-theart algorithm outperforms the rest for all sample sizes. Some suggestions for future work are also included.

show abstract

Section: E Search-based Software Engineeringmentioning

confidence: 99%

“…We propose to use the Mann-Whitney U(MWU) test or the Wilcoxon rank-sum test. The choice of this test is motivated by the wide-spread support of the Wilcoxon rank-sum test for these types of studies [30], [27]. For our study, we chose the significance threshold α = 0.01.…”

Section: A Distribution Of Samples and Significance Testsmentioning

confidence: 99%

Analyzing Search Techniques for Autotuning Image-based GPU Kernels: The Impact of Sample Sizes

Torring¹,

Elster²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…1 Global performance-influence models are typically built by measuring execution time under different configurations [64]. The models can be built using white-box techniques [71,72,76], machinelearning approaches [20,[22][23][24]35], or a brute-force approach. Details on these techniques are beyond the scope of this paper.…”

Section: Identifying Influencing Optionsmentioning

confidence: 99%

On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support

Vélez¹,

Jamshidi²,

Siegmund³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Determining whether a configurable software system has a performance bug or it was misconfigured is often challenging. While there are numerous debugging techniques that can support developers in this task, there is limited empirical evidence of how useful the techniques are to address the actual needs that developers have when debugging the performance of configurable software systems; most techniques are often evaluated in terms of technical accuracy instead of their usability. In this paper, we take a human-centered approach to identify, design, implement, and evaluate a solution to support developers in the process of debugging the performance of configurable software systems. We first conduct an exploratory study with 19 developers to identify the information needs that developers have during this process. Subsequently, we design and implement a tailored tool, adapting techniques from prior work, to support those needs. Two user studies, with a total of 20 developers, validate and confirm that the information that we provide helps developers debug the performance of configurable software systems.

show abstract

“…On the other hand, to mitigate the limitation of the small sample, both SPLConqueror [42] and DECART [12] incorporate several sampling heuristics to select a set of representative configurations for training. However, it takes extra time and effort to determine the appropriate sampling strategy for each system since there is no universal optimal sampling heuristics [10]. Instead, PerLasso [14] and DeepPerf [13] focus on restricting the performance model to be sparse with 𝐿 1 regularization.…”

Section: Introductionmentioning

confidence: 99%

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

Cheng

Gao

Zheng

2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Modern software systems are usually highly configurable, providing users with customized functionality through various configuration options. Understanding how system performance varies with different option combinations is important to determine optimal configurations that meet specific requirements. Due to the complex interactions among multiple options and the high cost of performance measurement under a huge configuration space, it is challenging to study how different configurations influence the system performance. To address these challenges, we propose HINNPerf , a novel hierarchical interaction neural network for performance prediction of configurable systems. HINNPerf employs the embedding method and hierarchic network blocks to model the complicated interplay between configuration options, which improves the prediction accuracy of the method. Besides, we devise a hierarchical regularization strategy to enhance the model robustness. Empirical results on 10 real-world configurable systems show that our method statistically significantly outperforms state-of-the-art approaches by achieving average 22.67% improvement in prediction accuracy. In addition, combined with the Integrated Gradients method, the designed hierarchical architecture provides some insights about the interaction complexity and the significance of configuration options, which might help users and developers better understand how the configurable system works and efficiently identify significant options affecting the performance.

show abstract

Predicting Performance of Software Configurations: There is no Silver Bullet

Cited by 5 publications

References 57 publications

Analyzing Search Techniques for Autotuning Image-based GPU Kernels: The Impact of Sample Sizes

Analyzing Search Techniques for Autotuning Image-based GPU Kernels: The Impact of Sample Sizes

On Debugging the Performance of Configurable Software Systems: Developer Needs and Tailored Tool Support

HINNPerf: Hierarchical Interaction Neural Network for Performance Prediction of Configurable Systems

Contact Info

Product

Resources

About