Decision support via automated metric comparison for the palladio-based performance blame analysis

Abstract

When developing component-based systems, we incorporate third-party black-box components. For each component, performance contracts have been specified by their developers. If errors occur when testing the system built from these components, it is very important to find out whether components violate their performance contracts or whether the composition itself is faulty. This task is called performance blame analysis. In our previous work we presented a performance blame analysis approach that blames components based on a comparison of response time values from the failed test case to expected values derived from the performance contract. In that approach, the system architect needs to manually assess if the test data series shows faster or slower response times than the data derived from the contract. This is laborious as the system architect has to do this for each component operation. In this paper we present an automated comparison of each pair of data series as decision support. In contrast to our work, other approaches do not achieve fully automated decision support, because they do not incorporate sophisticated contracts. We exemplify our performance blame analysis including the automated decision support using the "Common Component Modeling Example" (CoCoME) benchmark.

PBlaman: performance blame analysis based on Palladio contracts

“…In two case studies, we demonstrate the applicability of PBlaman as well as the consistency of the blame graph with the performance report. In addition to our case study from [13] that considers the layered system CoCoME [14], here we also show the application of PBlaman to a pipesand-filters style system based on Apache UIMA, that is, a framework for extracting information from unstructured texts [15]. Both case studies suggest that our approach is generalizable to systems with an according architectural style.…”

“…In this paper, we present the complete performance blame analysis approach PBlaman, which combines our basic concepts for blame analysis [12] with the decision support introduced in [13]. PBlaman builds on the model-based development process [2] and relies on the PCM [4].…”

“…We now present our PBlaman approach, which brings together the basic idea [12] with the decision support [13]. The following subsections elaborate on the three steps of its underlying process that are Here, the response time estimation of the component (0.64 ms) is multiplied with the hardware correction factor.…”

“…Two provided artifacts, called blame graph and performance report, serve as decision support and are part of our previous work [13]. They enable a system architect to easily interpret the performance blame analysis results (step 3) as well as to decide whether there are component operations to be blamed.…”

“…In our previous work, we first [12] summarized the basic idea of PBlaman, that is, using PCM instances as test oracles in order to judge component performance. Then, we detailed the decision support [13] and demonstrated PBlaman's usefulness for CoCoME. The case study of the UIMA-based system is a new contribution of this paper, providing further insights into the usefulness of our performance blame results.…”