Multimethod communication for high-performance metacomputing applications

Abstract

Metacomputing systems use high-speed networks to connect supercomputers, mass storage systems, scientific instruments, and display devices with the objective of enabling parallel applications to access geographically distributed computing resources. However, experience shows that high performance often can be achieved only if applications carn integrate diverse communication substrates, transport mechanisms, and protocols, chosen according to where communication is directed, what is communicated, or when communication is performed. In this article, we describe a software architecture that addresses this requirement. This architecture allows multiple communication methods to be supported transparently in a single application, with either automatic or user-specified selection criteria guiding the methods used for each communication. We describe an implementation of this architecture, based on the Nexus communication library, and use this implementation to evaluate performance issues. The implementation supported a wide variety of applications in the I-WAY metacomputing experiment at Supercomputing 95; we use one of these applications to provide a quantitative demonstration of the advantages of multimethod communication in a heterogeneous networked environment. I IntroductionFuture networked computing systems will be increasingly heterogeneous in terms of both the types of networked devices and the capabilities of the networks used to connect these devices. At the same time, the applications that run on these networks are becoming more sophisticated in terms of the computations they perform and the types of data that they communicate [6].

A secure communications infrastructure for high-performance distributed computing

“…Nexus incorporates automatic configuration mechanisms that allow it to use configuration information provided by a Metacomputing Directory Service (MDS) [7] to determine which startup mechanisms, network interfaces, and communication methods to use in different situations [8]. These mechanisms allow Nexus programs to execute unchanged in different environments, with communication methods selected according to default rules, depending on the source and destination of the message being sent.…”

“…An advantage of the startpoint construct in a distributed computing environment is that the startpoint can be used to encapsulate not only information about where a remote object is located, but also how to communicate with that remote object. This feature has been exploited to manage the use of multiple communication methods [8].…”

A capabilities based communication model for high-performance distributed applications: The Open HPC++ approach

“…Modern communication libraries like Nexus [2] provide adaptivity features such as multi-method communication, automatic and user controlled protocol selection, and custom protocols. But these libraries provide these features at a very low-level and hence they are not directly usable by application programmers.…”

A network status predictor to support dynamic scheduling in network-based computing systems

“…AppLeS agents utilize a network weather service (NWS) [1,11] namically forecast their expected future values using several numerical models. The Legion [6] and Globus/Nexus [4,5] metacomputing systems also employ NWS for network scheduling.…”

“…While substantial research efforts have focused on improving various aspects of the communication performance of networkbased systems [2,3,5,7,8,10], network resources typically have not been actively scheduled. However, understanding dynamic variations in both network bandwidth and latency is critical to maximize an application's overall performance and resource utilization.…”