Validating Model-Driven Performance Predictions on Random Software Systems

Abstract

Abstract. Software performance prediction methods are typically validated by taking an appropriate software system, performing both performance predictions and performance measurements for that system, and comparing the results. The validation includes manual actions, which makes it feasible only for a small number of systems. To significantly increase the number of systems on which software performance prediction methods can be validated, and thus improve the validation, we propose an approach where the systems are generated together with their models and the validation runs without manual intervention. The approach is described in detail and initial results demonstrating both its benefits and its issues are presented.Key words: performance modeling, performance validation, MDD MotivationState of the art in model-driven software performance prediction builds on three related factors: the availability of architectural and behavioral software models, the ability to solve performance models, and the ability to transform the former models into the latter. This is illustrated for example by the survey of modeldriven software performance prediction [3], which points out that the typical approach is to use UML diagrams for specifying both the architecture and the behavior of the software system, and to transform these diagrams into performance models based on queueing networks.Both the models and the methods involved in the prediction process necessarily include simplifying assumptions that help abstract away from some of the complexities of the modeled system, e.g., approximating real operation times with probability distributions or assuming independence of operation times. These simplifications are necessary to make the entire prediction process tractable, but the complexity of the modeled system usually makes it impossible to say how the simplifications influence the prediction precision.Self-archived copy. The original publication is available at www.springerlink.com,

Can Linear Approximation Improve Performance Prediction ?

“…To overcome this obstacle, we use our tool for generating synthetic software applications [7]. The tool assembles modules taken from industry standard benchmarks [22] and other sources [6] into applications whose architecture is random and whose workload exhibits the effects of resource sharing.…”

“…The original publication is available at www.springerlink.com, http://www.springerlink.com/content/50868p3861927512/ Using a wide range of synthetic software applications makes our results more general in that our observations are not collected on only a few systems, where inadvertent bias, experiment tuning or even plain luck can distort the conclusions significantly. For a more involved discussion of the representativeness of the applications, we refer the reader to [7] -here, we limit ourselves to asserting that our conclusions should be reasonably valid for concurrent systems running industry-standard processor-intensive workloads.…”

“…For this paper, we use the Queueing Petri Nets as implemented by the SimQPN tool [16]. The details of the mapping from the architectural and behavioral model into the performance model are given in [7,3]. Briefly, the mapping uses queueing places to represent processors and clients.…”

“…Helper places are used to represent thread pools, thread synchronization, branching and looping. Service demands use normal distribution, which we found to be more appropriate than the more common exponential distribution [7], loop counts are approximated with geometric distribution.…”

“…Obviously, software performance models that do not themselves capture the resource sharing effects will achieve better accuracy with service demands measured under resource sharing [7]. This, however, is not always realistic -as described, measurement under resource sharing requires having the modeled application, which is not the case in predictive performance modeling.…”