Jungle Computing: Distributed Supercomputing Beyond Clusters, Grids, and Clouds

Abstract

In recent years, the application of high-performance and distributed computing in scientific practice has become increasingly wide-spread. Among the most widely available platforms to scientists are clusters, grids, and cloud systems. Such infrastructures currently are undergoing revolutionary change due to the integration of many-core technologies, providing orders-of-magnitude speed improvements for selected compute kernels. With high-performance and distributed computing systems thus becoming more heterogeneous and hierarchical, programming complexity is vastly increased. Further complexities arise because urgent desire for scalability, and issues including data distribution, software heterogeneity, and ad-hoc hardware availability, commonly force scientists into simultaneous use of multiple platforms (e.g. clusters, grids, and clouds used concurrently). A true computing jungle. In this chapter we explore the possibilities of enabling efficient and transparent use of Jungle Computing Systems in every-day scientific practice. To this end, we discuss the fundamental methodologies required for defining programming models that are tailored to the specific needs of scientific researchers. Importantly, we claim that many of these fundamental methodologies already exist today, as integrated in our Ibis high-performance distributed programming system. We also make a case for the urgent need for easy and efficient Jungle Computing in scientific practice, by exploring a set of state-of-the-art application domains. For one of these domains we present results obtained with Ibis on a real-world Jungle Computing System. The chapter concludes by exploring fundamental research questions to be investigated in the years to come.

The Astrophysical Multipurpose Software Environment

“…Current computing resources available to researchers are more varied than simple workstations: clusters, clouds, grids, desktop grids, supercomputers and mobile devices complement standalone workstations, and in practice one may want to take advantage of this ecosystem, which has been termed Jungle computing (Seinstra et al 2011), to quickly scale up calculations beyond local computing resources.…”

HARD: Hybrid Adaptive Resource Discovery for Jungle Computing

“…Resource discovery requirements for such future systems are beyond the techniques used in today's Grids, Clouds and HPC clusters. We were inspired by the concept of "Jungle Computing", introduced in the recent years [1][2][3][4][5][6][7][8], and accordingly, we can envision that most of requirements for Jungle can be applied for future computing systems. In fact, two important features of such future environments are: first, heterogeneity of resources; and, second, very large number of resources.…”

“…Moreover, data distribution, hardware availability, software heterogeneity, and also the sheer size of scientific problems, commonly force scientists to resort to Jungle Computing instead of traditional supercomputers and clusters. Jungle Computing (i.e., leveraging multiple computing platforms simultaneously) is a recent distributed computing paradigm based on concurrent combination of various hierarchical and distributed computing environments with large number of heterogeneous resources [1][2][3][4][5][6][7][8]) .…”

Evolution of the Distributed Computing Paradigms: a Brief Road Map

“…[55]. It refers to the use of diverse, distributed and highly non-uniform high performance computer systems to achieve peak performance [1].…”