DCI Bridge: Executing WS-PGRADE Workflows in Distributed Computing Infrastructures

Abstract

Abstract. Solving distributed computing infrastructure (DCI) incompatibility issues in a generic way is a challenging and complex task. Gateways and workflow management systems are often tightly bound to some limited number of supported DCIs. To enable gateways access to many different DCIs and to solve DCI compatibility among the very different workflow management systems, we have developed a generic solution, the DCI Bridge. In this chapter we describe its internal architecture, provide usage scenario and show how the DCI Bridge resolves interoperability issues between various middleware-based DCIs. We also provide insight about the capabilities of the realized system. The generic DCI Bridge service seamlessly enables the execution of workflows (and jobs) on major DCI platforms such as ARC, Globus, gLite, UNICORE, SGE, PBS, as well as web services or clouds. IntroductionIn most cases, gateways and workflow management systems (and therefore their workflows) are tightly bound to some small number of specific distributed computing infrastructures (DCIs), and effort is required to enable additional DCI support. As a result, solving workflow management systems' DCI incompatibility, or their workflow interoperability [Krefting/2011] issues are very challenging and complex tasks. In this chapter we show a concept of how to enable generic DCI compatibility, which is feasible for many major gateways and grid workflow management systems (such as ASKALON [Duan/2005], MOTEUR [Glatard/2008], WS-PGRADE/gUSE [Kacsuk/2012], etc.) on workflow level (and also on the job level). To enable DCI compatibility among the different gateways and workflow management systems, we have developed the DCI Bridge, which is also one of the main components of the so-called fine-grained interoperability approach (FGI) developed by the SHIWA (SHaring Interoperable Workflows for large-scale scientific simulations on Available DCIs) project [Plankensteiner/2013]. In this chapter we target the generic DCI Bridge service component and describe its internal architecture, provide usage scenarios, and show how the DCI Bridge can resolve

Large-scale climate simulations harnessing clusters, grid and cloud infrastructures

“…For these situations DIRAC provides a matchmaking scheduling. The metabroker concept posed by gUSE/WS-PGRADE aims at supporting interoperability among resources at workflow level [33]. Thus each component of the workflow can be run on a different resource.…”

Performance Analysis of Open Source Machine Learning Frameworks for Various Parameters in Single-Threaded and Multi-threaded Modes

“…The additional important aspects are the low entrance barrier and fast learning curve. They usually are based on [9], KNIME [10], etc.) and applications in physics [11], chemistry [12], astronomy [13], brain-computing [14], eHealth [15] can be found elsewhere.…”

Infrastructure Aware Scientific Workflows and Infrastructure Aware Workflow Managers in Science Gateways

“…In our experimental system the WFM is WS-PGRADE/gUSE and the BES-compliant job submission service is DCI Bridge [28]. …”