Fast messages: efficient, portable communication for workstation clusters and MPPs

Abstract

P rognostications of the National Information Infrastructure's future structure typically include high-performance servers of information, computation, and other specialized services embedded in a high-speed network fabric with hundreds of millions of other hosts. Two system architectures are likely candidates for these servers: massively parallel processors and networks of workstations. Both are attractive because of their ability to scale. However, both critically depend on internal communication performance to be effective servers, and any NII server depends on excellent external networking to support NII service. Interestingly, in recent years, MPP and NOW hardware have become increasingly similar, as both are driven by the significant cost advantages of high-volume products. Hence, many issues involved in delivering communication performance in such systems have converged as well.The Illinois Fast Messages project intends to exploit this convergence to develop communications technology that spans both MPPs and NOWs, supporting both intracluster communication and high-speed external networking. FM is a portable, low-level messaging interface that can deliver high network data rates, even for small messages. It delivers the lowlatency communication that is essential for efficient coordination and data movement on large-scale parallel systems. FM implementations not only deliver high performance but also provide the high-level guarantees that enable streamlined implementations of higher-level protocols atop FM. FM delivers performance to the higher-level layers, not just to applications written directly to the messaging layer.We've implemented FM 1.1 on the Cray T3D MPP and on Myrinetbased workstation clusters. Both implementations perform substantially better than the vendor-supplied messaging layers. We've also implemented two widely accepted standard interfaces-Unix sockets and the Message 60

Final Report for the 10 to 100 Gigabit/Second Networking Laboratory Directed Research and Development Project

“…An implementation of FM for the Cray T3D is described in detail in [53]. This implementation provides two flavors of FM -Push FM and Pull FM.…”

“…However, the FM API imposes more stringent requirements for the underlying network and NIC, including the implementation of flow control, in-order delivery, and buffering. In [53], performance results are provided for an implementation of FM over Myrinet, using native FM APIs, sockets (TCP/IP), and MPI APIs.…”

“…Fast Messages (FM) [53] is an OS bypass mechanism that was developed by Andrew Chien and his colleagues at the University of Illinois. FM is based largely on the Active Messages (AM) work performed at U.C.…”

User-level communication in a system with gang scheduling

“…The FM system was developed at the University of Illinois as part of the High Performance Virtual Machine (HPVM) project [10]. The system offers high speed communication over Myricom's Myrinet SAN (System Area Network).…”

“…The Myrinet network interface cards have a LANai 4.3 processor and 512 KB RAM. Data communications use a modified version of the FM 2.0 library from the University of Illinois [10] (more in Section 3).…”