Self-Healing of Operational Workflow Incidents on Distributed Computing Infrastructures

Abstract

Abstract:Distributed computing infrastructures are commonly used through scientific gateways, but operating these gateways requires important human intervention to handle operational incidents. This report presents a self-healing process that quantifies incident degrees of workflow activities from metrics measuring long-tail effect, application efficiency, data transfer issues, and site-specific problems. These metrics are simple enough to be computed online and they make little assumptions on the application or resource characteristics. From their degree, incidents are classified in levels and associated to sets of healing actions that are selected based on association rules modeling correlations between incident levels. We specifically study the long-tail effect issue, and propose a new algorithm to control task replication. The healing process is parametrized on real application traces acquired in production on the European Grid Infrastructure. Experimental results obtained in the Virtual Imaging Platform show that the proposed method speeds up execution up to a factor of 4, consumes up to 26% less resource time than a control execution and properly detects unrecoverable errors.

Workflow Fairness Control on Online and Non-clairvoyant Distributed Computing Platforms

“…The work presented here is a step in our attempt to control computing platforms where very little is known about applications and resources, and where situations change over time. Our works in [12,20] consider similar platform conditions but they target completely different problems, namely fault-tolerance and granularity control. We believe that results of this paper are the first ones presented to control fairness in such conditions which are often met in production platforms.…”

“…Grid conditions vary among repetitions because computing, storage and network resources are shared with other users . We use MOTEUR 0.9.21, configured to resubmit failed tasks up to 5 times, and with the task replication mechanism described in [12] activated. We use the DIRAC v6r5p1 instance provided by France-Grilles 4 , with a first-come, first-served policy imposed by submitting workflows with decreasing priority values.…”

“…1. A new instantiation of our control loop [12] to handle unfairness, consisting of (i) an online, non-clairvoyant fairness metric, and (ii) a task prioritization algorithm.…”

Online Task Resource Consumption Prediction for Scientific Workflows

“…Since the outputs of each task in a workflow become inputs to subsequent tasks, and we use input size to estimate all the target parameters, poor output data size estimates for tasks at higher levels of the workflow may lead to a chain of increasing estimation errors for tasks at subsequent levels. Therefore, in addition to the offline estimation process, we also propose an online estimation process based on the MAPE-K loop (Monitoring, Analysis, Planning, Execution, and Knowledge), where task executions are constantly monitored [41,42]. Upon task completion, estimated values for the task are updated with the real values, and, based on these values, a new prediction is generated (using the regression tree of Fig.…”

Self-Protection against Insider Threats in DBMS through Policies Implementation

“…Instead of directly user input in the system, User defines general procedures and policies that guide the self-management process. IBM defines four main self-* components [7] [41] [42] [43] [44] [45].…”