Integrating Abstractions to Enhance the Execution of Distributed Applications

Abstract

One of the factors that limits the scale, performance, and sophistication of distributed applications is the difficulty of concurrently executing them on multiple distributed computing resources. In part, this is due to a poor understanding of the general properties and performance of the coupling between applications and dynamic resources. This paper addresses this issue by integrating abstractions representing distributed applications, resources, and execution processes into a pilot-based middleware. The middleware provides a platform that can specify distributed applications, execute them on multiple resource and for different configurations, and is instrumented to support investigative analysis. We analyzed the execution of distributed applications using experiments that measure the benefits of using multiple resources, the late-binding of scheduling decisions, and the use of backfill scheduling.

Evaluating Distributed Execution of Workloads

“…The execution strategy of Experiment 3 is consistent with the insight gained from Experiment 1 and 2 and with previous experimental results [1]. As each pilot can execute all the tasks of the given BoT, T T C should not depend on the last pilot becoming active, as in Experiments 1 and 2, but on the first one (and on how many pilots become active after the first pilot).…”

“…5 and 6 show that we experienced very short T w compared to the historical annual average queue time recorded by XDMoD [19]. In presence of longer T w , Experiment 3's execution strategy could perform worse than Experiment 4's due to the reduced interplay between the queue times of just 2 resources, which would be consistent with previous data [1].…”

“…Our contribution is to advance and extend the concept of execution strategy [1]. We integrated Swift and AIMES into a system specifically designed to execute, characterize, and measure alternative execution strategies ( §V).…”

“…We address these limitations by devising abstractions and integrating existing middleware. Previously, we used sequences of decisions to model the coupling between workload requirements and resource capabilities [1]. We called these sequences Katz moved from the CI to U. Illinois during this work.…”

Application Skeleton: Generating Synthetic Applications for Infrastructure Research

“…The generated applications are compatible with workflow system such as Swift (Zhao et al 2007, Wilde et al (2009), Wilde et al (2011) and Pegasus (Ewa Deelman et al 2004, E. Deelman et al (2005), as well as the ubiquitous UNIX shell. The application can also be created as a generic JSON object that can be used by other systems such as the AIMES (Turilli et al 2015) middleware. …”