An Extensible Monitoring Framework for Measuring and Evaluating Tool Performance in a Service-Oriented Architecture

Abstract

Abstract. The lack of QoS attributes and their values is still one of the fundamental drawbacks of web service technology. Most approaches for modelling and monitoring QoS and web service performance focus either on client-side measurement and feedback of QoS attributes, or on ranking and discovery, developing extensions of the standard web service discovery models. However, in many cases, provider-side measurement can be of great additional value to aid the evaluation and selection of services and underlying implementations.We present a generic architecture and reference implementation for non-invasive provider-side instrumentation of data-processing tools exposed as QoS-aware web services, where real-time quality information is obtained through an extensible monitoring framework. In this architecture, dynamically configurable execution engines measure QoS attributes and instrument the corresponding web services on the provider side. We demonstrate the application of this framework to the task of performance monitoring of a variety of applications on different platforms, thus enriching the services with real-time QoS information, which is accumulated in an experience base.

Improving component selection and monitoring with controlled experimentation and automated measurements

“…Both general Quality-of-Service (QoS) attributes such as performance, throughput, resource usage, etc., as well as specific QoS criteria such as accuracy need to be continuously monitored during operation to ensure that the used component actually keeps fulfilling the requirements as expected [9]. Any deviation in QoS of the level measured during the experiments is an indication of either an incomplete evaluation procedure, or a change in the environment that needs to be addressed, such as a sudden increase in data volume.…”

“…The candidate tools are applied to a previously defined set of sample objects, and the outcome is evaluated against the requirements, relying on automated measurements. That means that all considered components are executed in their respective deployment location inside a controlled environment that monitors their behaviour and collects data about their resource utilisation [9]. This meta-information on the service execution is delivered along with the component output and mapped semi-automatically to the specified requirements.…”

“…Process-and performance-related characteristics can be measured automatically during tool execution in the experiments stage. The tools are invoked through a monitoring framework which is able to measure general process-related characteristics such as performance [9]. 3.…”

Generalized aggregate Quality of Service computation for composite services

“…The QoS values for each service and the branching probabilities of gateways in the BPMN model are defined in separate (text) files. The tool is distributed as an extension of the BPStruct tool 4 and is available at: http://sep.cs.ut.ee/Main/Bpstruct. Below we present an evaluation of the scalability of the QoS aggregation method using the implemented tool.…”

“…In particular, providers of composite services need to assess the expected quality of these services and to detect and act upon unexpected QoS variations [2,3,4].…”