K-WfGrid Distributed Monitoring and Performance Analysis Services for Workflows in the Grid

Abstract

Grid workflows for e-science are complex and prone to failures. However, there is a lack of performance monitoring and analysis tools for supporting the user as well as workflow middleware to monitor and understand the performance of complex interactions among Grid applications, middleware and resources involved in workflow executions. In this paper, we present a novel integrated environment which supports online performance monitoring and analysis of service-oriented workflows. Performance monitoring and analysis of Grid workflows and infrastructure is conducted through a Web portal. Performance overheads of Grid workflows are analyzed in a systematic way, and performance problems can be detected during runtime. Moreover, we present several languages that alleviate the interaction among performance monitoring and analysis services and their clients. Our system has been integrated into the K-WfGrid knowledge-based workflow system. It plays a key role in supporting the user and developer to analyze their workflows and in providing performance knowledge for constructing and executing workflows.

DIPAS: A distributed performance analysis service for grid service-based workflows

“…This paper refines and extends a previous paper published in [10]. We are grateful to the support of our colleagues in the K-WfGrid consortium.…”

“…We also illustrate DIPAS through the performance evaluation of real-world workflows. This paper substantially refines and extends our previous conference paper published in [10], focusing on the workflow performance evaluation and providing an up-to-date status of the development of DIPAS.…”

“…During runtime, using execution tracing tool, if the user detects some strange performance behavior, then the user can suspend the workflow, examine the execution status, and resume or terminate the execution of the workflow. Moreover, by using the detailed information about machines used in workflows, further infrastructure monitoring and analysis could be conducted through the integration with an infrastructure monitoring service [10]. Figure 12 shows an example of performance overhead analysis.…”

Performance metrics and ontologies for Grid workflows

“…Fig. 7(a) and (b) show metrics associated with activity instance second 16 0, provided by ASKALON workflow performance tool [37] and with activity instance computeSSSP, provided by the K-WfGrid performance tool [38], respectively. Fig.…”

Workflow adaptation as an autonomic computing problem

“…In a scientific workflow of the type defined in Section 3, there is a set of generic data that can be monitored that will be of use in determining the state of the system, and some work has been undertaken to monitor workflow progress [17]. These monitoring data cab be characterised as relating to task state, execution environment state, service availability, etc.…”