Incremental Calibration of Architectural Performance Models with Parametric Dependencies

Abstract

Architecture-based Performance Prediction (AbPP) allows evaluation of the performance of systems and to answer what-if questions without measurements for all alternatives. A difficulty when creating models is that Performance Model Parameters (PMPs, such as resource demands, loop iteration numbers and branch probabilities) depend on various influencing factors like input data, used hardware and the applied workload. To enable a broad range of what-if questions, Performance Models (PMs) need to have predictive power beyond what has been measured to calibrate the models. Thus, PMPs need to be parametrized over the influencing factors that may vary.Existing approaches allow for the estimation of parametrized PMPs by measuring the complete system. Thus, they are too costly to be applied frequently, up to after each code change. They do not keep also manual changes to the model when recalibrating.In this work, we present the Continuous Integration of Performance Models (CIPM), which incrementally extracts and calibrates the performance model, including parametric dependencies. CIPM responds to source code changes by updating the PM and adaptively instrumenting the changed parts. To allow AbPP, CIPM estimates the parametrized PMPs using the measurements (generated by performance tests or executing the system in production) and statistical analysis, e.g., regression analysis and decision trees. Additionally, our approach responds to production changes (e.g., load or deployment changes) and calibrates the usage and deployment parts of PMs accordingly.For the evaluation, we used two case studies. Evaluation results show that we were able to calibrate the PM incrementally and accurately.

Data Stream Operations as First-Class Entities in Component-Based Performance Models

“…We, furthermore, will investigate the applicability of our approach to other types of systems, particularly self-adaptive systems. Another interesting direction of research is how to extract the type of model presented in this article from code or other artifacts and how to integrate the prediction using these models in an agile software engineering process, as proposed by Mazkatli et al [14].…”

A Model-driven Approach for Continuous Performance Engineering in Microservice-based Systems

“…Mazkatli et al [36] have presented an approach for continuous integration of performance models that considers parametric dependencies after analyzing the source code changes. The approach builds upon the Palladio approach and the goal is to automatically keep the performance models up-to-date to allow architecture-based performance prediction.…”

Performance-detective

“…Furthermore, Performance-Detective can remove configurations aimed at finding interactions between 𝑝𝑟𝑜𝑐𝑠 and 𝑑𝑖𝑟𝑠𝑒𝑡𝑠 as they influence distinct sets of functions (Insight 2). Thus, Performance-Detective varies 𝑝𝑟𝑜𝑐𝑠 independently of 𝑑𝑖𝑟𝑠𝑒𝑡𝑠, resulting in 5 measuring points with (p, dirsets) = (8, 8), (16,16), (32,24), (64, 32), (128, 64).…”

“…Parametric Performance Models. The difficulty of modeling modern software systems with many parameters has been addressed by parametric software performance models [5,28,32,35] and Performance-Influence Models (PIMs) [39].…”

“…The parametric software performance modelling approach Palladio [5,35] allows the modelling of parametric dependencies. However, automated extraction of these models [28,32] assumes that the system consists of software components with well-defined interfaces, which is usually not the case in HPC applications.…”