Cluster communication protocols for parallel-programming systems

Abstract

Clusters of workstations are a popular platform for high-performance computing. For many parallel applications, efficient use of a fast interconnection network is essential for good performance. Several modern System Area Networks include programmable network interfaces that can be tailored to perform protocol tasks that otherwise would need to be done by the host processors. Finding the right trade-off between protocol processing at the host and the network interface is difficult in general. In this work, we systematically evaluate the performance of different implementations of a single, user-level communication interface. The implementations make different architectural assumptions about the reliability of the network and the capabilities of the network interface. The implementations differ accordingly in their division of protocol tasks between host software, network-interface firmware, and network hardware. Also, we investigate the effects of alternative data-transfer methods and multicast implementations, and we evaluate the influence of packet size. Using microbenchmarks, parallel-programming systems, and parallel applications, we assess the performance of the different implementations at multiple levels. We use two hardware platforms with different performance characteristics to validate our conclusions. We show how moving protocol tasks to a relatively slow network interface can yield both performance advantages and disadvantages, depending on specific characteristics of the application and the underlying parallelprogramming system.

Experiences with Fine-Grained Distributed Supercomputing on a 10G Testbed

“…Often, an implicit assumption is made that a parallel application will perform well on a "low latency" and/or "high throughput" network, while in reality the host-level overhead of sending the data across the network is frequently a much more determining factor [15]. As shown in Table 1, the send, receive and polling overhead (i.e., the number of compute cycles spent in the host-level network protocol layers) differ significantly for TCP/IP and Myri-10G's native MX (for Cluster/TCP and Grid/TCP they are almost identical).…”

Efficient Java Communication Protocols on High-speed Cluster Interconnects

“…This growing need of efficient communication middleware has led to devote important efforts on this subject, although almost exclusively on native protocols. A thorough work focused on native protocols is that of Verstoep et al [26], where several implementation issues are studied in order to obtain an efficient use of Myrinet. In that work, a non standard user level communication interface is implemented varying reliability protocols, the Maximum Transfer Unit (MTU), multicast protocols and studying Serial Direct Memory Access (SDMA)-based versus Processor Input/Output (PIO)-based message passing and remotememory copy.…”

“…In that work, a non standard user level communication interface is implemented varying reliability protocols, the Maximum Transfer Unit (MTU), multicast protocols and studying Serial Direct Memory Access (SDMA)-based versus Processor Input/Output (PIO)-based message passing and remotememory copy. Our proposed approach inherits some optimizations from [26]. Most of these topics are also covered in the related literature, where works are focused on evaluating multicast performance of user level multicast libraries on SCI and Myrinet [16], on obtaining lightweight transport protocols (RMPP [19]) and also on RDMA-based efficient message passing implementations [10].…”

Non-blocking Java Communications Support on Clusters

“…A thorough work focused on native protocols is that of Verstoep et al [1], where several implementation issues are studied in order to obtain an efficient use of Myrinet. In this study, a non standard user level communication interface is implemented varying reliability protocols, maximum transfer unit, multicast protocols and studying Serial Direct Memory Access (SDMA)-based versus Processor Input/Output (PIO)-based message passing and remotememory copy.…”

“…In this study, a non standard user level communication interface is implemented varying reliability protocols, maximum transfer unit, multicast protocols and studying Serial Direct Memory Access (SDMA)-based versus Processor Input/Output (PIO)-based message passing and remotememory copy. The proposed approach inherits some optimizations from [1].…”