Critical perspectives on large-scale distributed applications and production Grids

Abstract

Abstract-It is generally accepted that the ability to develop large-scale distributed applications that are extensible and independent of infrastructure details has lagged seriously behind other developments in cyberinfrastructure. As the sophistication and scale of distributed infrastructure increases, the complexity of successfully developing and deploying distributed applications increases both quantitatively and in qualitatively newer ways. In this manuscript, we trace the evolution of a representative set of "state-of-the-art" distributed applications and production infrastructure; in doing so we aim to provide insight into the evolving sophistication of distributed applications -from simple generalizations of legacy static high-performance to applications composed of multiple loosely-coupled and dynamic components. The ultimate aim of this work is to highlight that even accounting for the fact that developing applications for distributed infrastructure is a difficult undertaking, there are suspiciously few novel and interesting distributed applications that utilize production Grid infrastructure. Along the way, we aim to provide an appreciation for the fact that developing distributed applications and the theory and practise of production Grid infrastructure have often not progressed in phase. Progress in the next phase and generation of distributed applications will require stronger coupling between the design and implementation of production infrastructure and the theory of distributed applications, including but not limited to explicit support for distributed application usage modes and advances that enable distributed applications to scale-out.

Distributed computing practice for large‐scale science and engineering applications

“…There remains, however, scope for improvement in both the range and number of supported abstractions and the type of infrastructure over which such abstractions are made available. In fact, not only do distributed applications need abstractions to support the requirements through the individual stages of development, deployment and execution, support for applications through these stages needs to be a coherent integrated activity, as opposed to a disjointed effort . This will then enable methods for developing distributed applications that, for example, take into account the status of the infrastructure (deployment) as well as the execution modes of the application.…”

Case Studies in Designing Elastic Applications

“…The authors in [21] provide a characterization of scientific workflows in terms of their size, complexity, composition, and computation requirements. The authors in [22] study the evolution of distributed infrastructure such as grids, and profile a set of applications deployed on such infrastructure. They use this study to offer critical insights on the challenges facing the developers of distributed computing systems (such as grids) and the distributed applications deployed on these systems.…”

Multi-domain grid/cloud computing through a hierarchical component-based middleware

“…Both enterprises and the HPC community are looking into the aggregation of existing resources across multi-domain environments or into outsourcing computation to Cloud providers as possible ways to gather more resources, reducing costs and improving application distribution [11]. In [13], authors acknowledge that supporting systems for multi-domain (Grid and Cloud) computing have been developed to hide from heterogeneity, dynamism and distributed nature of resources, instead of embracing such characteristics. Regardless of these efforts, heterogeneity and dynamism have been the major barrier in the uptake of distributed systems and distributed applications.…”